What one H200 actually holds: KV-cache capacity for Qwen 3.6 27B FP8 on vLLM

Ask how many users a GPU can serve and every answer you get is denominated in tokens per second. Benchmarks are tok/s. The vLLM startup log greets you with throughput numbers. That’s the first mistake, and it hides a second one: expecting the answer to be a number at all.

I spent a few evenings measuring it on a single H200 running Qwen 3.6 27B FP8 on vLLM. The right first question is: how many users’ sessions fit warm in KV cache? The answer is a property of your users, not of the GPU.

What the GPU is doing between your requests

An inference server does two very different jobs. Prefill reads every token of your prompt in parallel. It’s compute-bound and happens before you see any output. Decode writes the answer, one token at a time, each step reading the model weights plus everything cached so far. Decode is what you see: tokens streaming, tool calls firing. Prefill is invisible. Nothing moves, and a long one is indistinguishable from a hang.

The KV cache is the server’s memory of text it has already read: every prefilled token leaves a small block of attention state behind, and keeping that around means never processing the same token twice. Agent traffic is shaped to exploit this. Every request your coding agent sends is the previous request plus the model’s answer plus a small new tail, so a byte-identical prefix grows with every request: same system prompt, same conversation history, same tool outputs. If that prefix is still resident in cache, the server prefills only the tail. If it was evicted, it re-processes all of it.

The effect is not subtle. There’s a public Claude Code session breakdown where a two-minute burst bills as 3M input tokens but only 138k of them were actually processed. The other 95% hit cache. This is inherent to agentic coding, where an LLM conversation is composed of many short request/response pairs, tool calls and tool results, each one resending the whole history.

Warm session = comfortable user

The proxy I dimension around: a user is served comfortably when their session stays warm in cache, warm meaning their prefix is still resident, nothing has evicted it. A warm user’s next request prefills a few thousand new tokens, which is seconds, and the GPU’s work stays mostly decode. Lose warmth and users wait through 100k+ token re-prefills that also steal decode bandwidth from everyone still warm.

Every API provider prices these same mechanics: writing to cache costs a premium over plain input, but reading from it costs a fraction. Anthropic bills a cached read at a tenth of a fresh input token. So the bill for agent traffic is mostly storage and reads rather than compute.

How much of that discount you collect is decided by the harness, the client program that assembles each request (Claude Code, opencode, and friends). That’s because the server reuses a prefix only if it comes back byte-identical: ideally append-only history, stable tool-output serialization, no timestamp or request id near the top of the prompt. vLLM can reuse part of a cached prefix, but a one-token change invalidates all of the cache that comes after it.

When people claimed opencode doesn’t cache tokens, its founder dax posted a week of cache hit rates by user agent from their inference service: 96.2% for the best client down to 67.8% for the worst, on the same backend. At a 10× cached-read discount that spread is worth roughly 3× on the input bill, and it is decided entirely client-side.

The hit rate is engineered on both sides of the API.

The working-set repo makes “comfortable” operational: a returning user’s request hits the warm prefix, which keeps time-to-first-token (TTFT) to a few seconds at most, and decode stays above 40 tok/s. 40 is the hard floor, 50 is comfortable. I wanted to know which of the two breaks first, and to answer that you need to know how much cache you actually have.

vLLM told me 352k tokens. It was off by nearly 4×

vLLM 0.19.0 prints GPU KV cache size: 352,000 tokens at startup for this model, and Maximum concurrency 5.1x at 262k max sequence length. Those two numbers contradict each other: 352k divided by 262k is 1.34, not 5.1. The size report turns out to be a known issue on 0.19.0.

So I measured it. The method: harvest real prompts from my own coding-agent logs, inject a UID at the start of each system prompt so no prompt can reuse another’s cache, send n of them in parallel for three rounds at 140k tokens each, and flag a request as warm when its TTFT comes in under 0.4× the cold time. Sweep n from 7 to 10 and watch where warmth collapses.

The true pool lands in [1139k, 1399k] tokens, nearly 4× what the log claims: enough for the full KV of 4 to 5 max-length sequences. And 5.1 × 262144 = 1337k sits inside the measured interval, so I deduced the concurrency multiplier was right all along. Only the token count printed next to it was wrong.

The collapse at n = 10 is the failure mode you’ll actually hit. The cache evicts least-recently-used (LRU) entries: to make room it drops whatever was touched longest ago. Ten prompts cycling in order through a cache that holds nine is the textbook worst case for that policy, because each prompt gets evicted just before its turn comes around again, so the hit rate goes to zero. One user too many doesn’t cost you one user’s worth of comfort.

One measurement is not a capacity plan

So the repo builds a transparent model on top of it: calibrate the memory budget so it reproduces the measured pool, then fill that pool Monte-Carlo style, meaning thousands of random draws of simulated sessions so you get a spread of outcomes rather than a single estimate. Each resident session is charged its fixed per-session state, and decode speed is priced against memory bandwidth, since every decode step has to stream the weights and the cache out of GPU memory. The sessions are drawn from a log-normal distribution fitted to 1,850 of my own coding-agent requests, median 31k tokens. Log-normal because session lengths are inherently skewed: context accretes multiplicatively as an agent works, so most sessions sit near the median and a few grow enormous. Every number below regenerates from tables.py.

For the 27B model with FP8 KV cache on one H200, the reference workload holds 86 warm users on the median draw (p50), and 69 on the p5 draw, the pessimistic one that 95% of draws beat. The finding I didn’t expect: in every configuration the repo studies, one or two GPUs, dense or mixture-of-experts, the cache binds before the bandwidth does. Even with every warm session decoding simultaneously, per-user speed stays at or above 41 tok/s; the speed floor alone would allow 118 concurrent decoders, but the cache runs out at 86 users. That ordering is bought by MTP, a speculative-decoding feature covered with the knobs below: every decode figure here includes its 1.7× speedup, and turning it off admits only 60 decoders at the floor, fewer than the 86 warm users, so bandwidth binds first. With MTP, capacity planning for agentic coding is cache planning.

Why does memory run out before bandwidth? Because memory is consumed by presence, whereas bandwidth is consumed by activity. A warm session occupies its slice of the pool around the clock, about a gigabyte for a median session (31k tokens × 32 KiB per token of FP8 KV on this model), whether the user is typing or reading a diff. Bandwidth, by contrast, is drawn only during decode, and the weight read, the big fixed cost of every step, is amortized across everyone decoding at once.

Agentic coding traffic is long-context and low-duty-cycle: everyone is resident, few are decoding at any instant, so memory fills long before bandwidth saturates. Flip the workload to short-context chat with every user streaming and bandwidth binds first on the exact same GPU.

The H200’s 141 GB of memory and 4.8 TB/s of bandwidth don’t pick the bottleneck. The shape of your traffic does.

There is no single capacity number

The reference distribution has a 31k median prompt. Make the sessions heavier and the same GPU serves a different number of people:

Plan on the p5 row. Halfway to doubling the median already costs a third of the capacity, and harnesses that stuff 30k tokens of system prompt, skills, and rules before the user says a word are moving themselves left-to-right across this table. So the only honest answer to “how many users fit on an H200” is a function, and I put the function online: an interactive explorer with the full model behind sliders for the workload, LLM, KV dtype, and topology. Put your own median in.

The knobs, in order of leverage

If you serve this yourself, these are the levers, biggest first.

FP8 KV cache (--kv-cache-dtype fp8_e4m3) stores the cache in 8-bit floats instead of 16, which doubles the pool. Under FP16 the same GPU holds 45 warm users, and per-user decode at 64 concurrent sequences drops to 39 tok/s, under the hard floor. One flag, minimal quality impact, the difference between the table above and half of it.

A shared, byte-stable prefix. Counterintuitive: a bigger shared system prompt raises warm capacity, because it’s stored once and shared by every session. Sweeping the user prefix 3k → 15k → 30k moves warm capacity 209 → 280 → 399 in the repo’s MoE configuration (Qwen 3.6 35B-A3B). But the win is fragile in both directions: a request that can’t match the prefix re-prefills the whole thing, and per-user drift in a 30k prefix turns the sharing into 30k × N of duplicated cache. Whether the prefix stays byte-stable is, again, harness engineering. And users pay for it in context window: a 30k system prompt they can’t trim spends 30k of their window before they’ve said a word.

CPU KV offload. Memory is scarce because it’s consumed by idle-but-resident sessions, so move precisely those to system RAM. An offloaded session keeps its warmth as storage while GPU memory serves the sessions actively prefilling and decoding. The repo’s scenario study puts 600 GiB of RAM at ~2,400 warm sessions for the MoE sibling on one H200, an order of magnitude past what the GPU holds alone. Caveat: the model prices offload as storage only; it doesn’t price the PCIe-bandwidth-bound restore on a session’s next turn, 0.1 to 0.3 seconds per 100k tokens plus contention. CPU-offloaded sessions are “less warm” than GPU-resident ones.

max_seq_len at 180k instead of the model’s 262k maximum. Caps the worst-case cold prefill and stops one runaway session from hogging the pool.

Tensor-parallel over data-parallel for the second GPU. Data-parallel (DP2) runs two complete, mutually unaware engines behind a load balancer: each replica has its own weight copy, its own scheduler, and its own private KV pool. Your session’s cache is a data structure inside whichever engine served you. Land the next request on the other replica and you re-prefill 100k tokens cold while a perfect copy sits idle on the card next door, and the session now eats pool in both replicas. So DP’s warm capacity is 2 × 86 only if the router pins every user to a home replica, and that router is yours to build. vLLM’s own docs are explicit about what the built-in balancer looks at:

Currently, the internal DP load balancing is done within the API server process(es) and is based on the running and waiting queues in each of the engines. This could be made more sophisticated in future by incorporating KV cache aware logic.

Queue depth, then. The balancer does not know where your cache lives.

Tensor-parallel (TP2) instead runs one engine with the weights sharded across both GPUs: one pool, one hash table, no routing condition. The second weight copy DP pays for becomes cache instead, so the pool grows 2.3–2.4×, not 2×, and holds 202 warm users unconditionally.

MTP speculative decoding. Every tok/s figure in this post assumes it: the tested config decodes two tokens per forward pass, priced as a 1.7× speedup, roughly 47% per-draft acceptance; real coding traffic runs higher. Not every model has this. Qwen 3.6 ships a trained MTP (multi-token prediction) head, a small module that drafts tokens for the main model to verify in the same forward pass. Turn it off and every decode number above divides by 1.7, and bandwidth binds before cache size.

The Qwen 3.6 MTP head is half of why this model serves coding agents so well. Light KV is the other half. Most of its layers replace standard attention with a linear-attention variant (Gated DeltaNet) that keeps one small constant-size state per session instead of state that grows with every token. As a result, only the minority of standard-attention layers pay per-token KV at all: 32 KiB per token, where a standard-attention stack this size pays several times more. A few months ago you could not hold this many sessions on one GPU with a model this size, and decode was slower too, because every step re-read a heavier cache.

And one non-knob: oversubscription past warm capacity degrades into cold-TTFT churn, not slow decode. Users don’t feel it as “the model got slower,” they feel it as “the tool randomly hangs for a minute.” That symptom is LRU eviction.

What this model doesn’t know

This is a calculation with one measured anchor. The repo says so precisely: the FP8 pool is projected from the measured FP16 pool, prefill/decode interference isn’t priced, and “warm” is a cache-residency claim, not a latency SLA. Each is one measurement run away from being checked, and a re-measured anchor recalibrates everything downstream of it. That’s the standard I’d hold any capacity calculator to: not that it’s right, but that it states exactly what would prove it wrong.

The model tells you which knob moves the number. It can’t tell you the number. Calculation narrows the search, but load testing decides.