Tokenomics Calculator

How the number is built

1. The model's KV cache size per token comes from its published architecture, counting only layers whose cache grows with context. Most current models compress the cache (MLA, or DeepSeek's compressed attention) or use full attention on only some layers, so their KV cache per token is far smaller than older models such as Llama. All model figures are editable.
2. Each GPU serves a number of active sessions, and keeps idle sessions that may resume. Their combined KV cache is the KV state to hold.
3. That state fills tiers in order: GPU HBM left after model weights, host memory, G3.5, then G4. G3.5 is pod-scoped and not persistent, so a share of resumed sessions (another pod, or after a restart) can't use it. With G4 those turns load from G4; without it they recompute.
4. On shared GPUs, only part of GPU memory, host memory and G3.5 capacity is available to this model's KV cache. GPU and host memory are also volatile: rescheduling, autoscaling, model swaps and restarts clear them before some sessions resume. That context then comes from G3.5 if it's still reachable, from G4 if present, or is recomputed. G4 is the only tier that persists.
5. On each turn the prior context is either already in HBM, reloaded from a tier, or recomputed. Reload time depends on bytes, per-GPU network bandwidth and the connection's efficiency. Part of each reload hides behind compute, but never more than the time the turn's new input takes to prefill: a reload far longer than that compute stays mostly visible. If reloading would take longer than recomputing, the calculator assumes the system recomputes.
6. Prefill is compute-bound (FLOPs at the stated utilization, including attention over the context). Decode is memory-bandwidth-bound. Their GPU time per turn sets tokens served per hour.
7. If flash reads would exceed the storage network, throughput drops to what the network can carry.
8. Cost per hour is GPU rental plus flash capacity and provisioned throughput. Per GB and per GB/s prices are the same for every tier and connection; a less efficient connection needs more provisioned throughput to deliver the same bytes.
9. Cost per million output tokens is cost per hour divided by output tokens generated per hour. The GPU time spent on prefill and on reloading or recomputing context is carried by the output tokens.

Sources

• GPU prices: median on-demand price per GPU-hour, AMCompute GPU cloud price comparison, updated 25 Oct 2026. The MI355X price comes from a single provider and the GB200 and GB300 medians from six and three providers.
• GPU memory, memory bandwidth and dense FP8 throughput: NVIDIA and AMD product specifications.
• NVIDIA Rubin (R200, called R100 in early supply-chain reports; the same GPU): 288 GB HBM4 and 22 TB/s per GPU from NVIDIA's announcements; about 16 PFLOPS dense FP8 is an estimate (Spheron Rubin guide). No cloud provider had published a Rubin hourly rate as of October 2026, and a 30 to 50% premium over Blackwell NVL72 is expected (Spheron), so the default price is the GB300 NVL72 median plus 40%. Rubin Ultra (2027 or later) isn't included because its per-GPU bandwidth isn't published.
• Model architectures: each model's published configuration on Hugging Face, checked 2 Nov 2026. DeepSeek V4 and V4.1 Flash KV sizes are DeepSeek's published bytes per token. Active parameters and memory size for DeepSeek V4.1 Flash, and memory size for Kimi K3, are estimates. Model list: most used open-weight models by weekly tokens on OpenRouter, 24 to 30 Oct 2026, as reported by TeamDay. Closed models such as GPT and Claude don't publish their architecture, so they can't be modelled.
• G3.5 tier: NVIDIA Technical Blog on the BlueField-4 context memory (G3.5) tier, which describes an Ethernet-attached flash tier for KV cache with petabytes of shared capacity per GPU pod. NVIDIA gives no per-GPU figure, so the default per GPU is a Silk modelling assumption.
• Transfer granularity matters as much as bandwidth: Kanichai, De Matteis and Trivedi, "Building py-kvcache", VU Amsterdam and IBM Research Zurich, arXiv 2609.11744, September 2026. The study compares offload paths to NVMe SSD; it doesn't compare block, file and object protocols.
• G4 connection defaults are Silk modelling assumptions, anchored where public data exists. NVMe-oF over RDMA adds about 12 microseconds per 4 KB read over local NVMe (Guz et al., SYSTOR 2017), so its fixed delay per reload is 0.1 ms. Object over TCP: a fast all-flash object store showed 3 to 5 ms median first-byte latency under load (MinIO AIStor, 400GbE), and AWS quotes single-digit milliseconds for S3 Express One Zone, so the default is 4 ms. Object over RDMA: no per-request latency has been published; RDMA removes the HTTP data path but keeps the S3 request and authentication step, and vendors claim sub-millisecond latency, so the default is 1 ms. Object efficiency (0.50 with RDMA, 0.35 without) reflects measured RDMA gains of 1.3 to 1.5× in Dell’s KV cache offload tests and 2.2 to 5.2× throughput in SeaweedFS’s S3 over RDMA tests. End-to-end tests that add tens of milliseconds at short contexts (Dell; ObjectCache, arXiv 2605.22850) include KV cache software every connection pays, so they aren’t charged to object storage here. No comparable public measurements were found for NFS over RDMA for KV cache, so NFS sits between block and object.
• Layer-by-layer streaming (as in LMCache's layer-wise mode and the ObjectCache paper) splits each chunk into one request per layer that holds KV cache, and hides more of the reload behind compute: 80% by default, against 50% chunk by chunk (Silk modelling assumptions).
• Requests per reload: object stores keep one object per KV chunk (256 tokens by default, as in LMCache), so each chunk is one request; block and file read in I/Os of up to 1 MB. Requests in flight per GPU default to 128 for NVMe-oF (deep hardware queues), 64 for NFS and 32 for object storage (parallel HTTP connections). Silk modelling assumptions; all are editable.
• Utilization, overlap, storage prices, session counts, the memory share on shared GPUs and the share of resumed sessions with cleared memory: Silk modelling assumptions. Dedicated preset: 100% of memory, 2% cleared. Shared AI cloud preset: 50% of memory, 20% cleared.

Limits

This is a planning model, not a benchmark. It ignores batching dynamics, scheduling, networking topology inside a pod and software overheads, and treats reuse as uniform across sessions. Use it to compare designs on equal terms, then validate with a test on your own workload.