GPU scaling — interactive scenario explorer

A transparent memory model, calibrated to a 1×H200 + 27B FP8 pool of 2.77M KV tokens — a figure projected from the baseline's measured FP16 pool (~1.337M tokens, ×2 for FP8 plus freed activation memory), not itself a direct measurement. The per-GPU activation reserve is solved from that anchor, so every other configuration falls out of the same arithmetic rather than being guessed.

• Goal & primary metric: we size deployments by warm sessions, planning on p5. Working hypothesis: a warm hit prefills only the new turn — served well and comfortably; a cold request re-prefills its whole context, briefly stealing bandwidth from every active user (thrash). Cold behaviour is deliberately NOT priced beyond this.
• TP (tensor-parallel, any N) shares one weight copy across all GPUs, so the pooled KV cache grows faster than N×; effective decode bandwidth scales by N × 0.90^log₂N (the baseline's 0.90 haircut applied per doubling — beyond 2 GPUs this is a projection, not a measurement).
• DP (data-parallel, any N) keeps N per-replica caches. Aggregate throughput scales ~N×, but a returning user is only warm on their home replica (sticky routing), and a shared CPU-offload buffer splits N ways.
• Monte-Carlo jitter: every render redraws the workload, so repeated runs (and neighbouring slider positions) can differ by a few sessions — that is sampling noise, not a real effect. The Python sweeps in tables.py use a fixed seed instead and are smooth. Real non-linearities (the expert-union kink at n=32, the cap's tail-truncation, the prefix floor) are explained in docs/scenarios.md.
• MoE decode reads only the active experts: weight bytes per step grow with the batch's expert union, from the shared block up to all routed experts. The explorer uses the conservative no-overlap union bound (docs also show the optimistic expected-union bracket).
• Structural assumptions & MTP knob: "Conservative" applies the stacked-adverse case from the docs (low anchor, fp32 state, +15% weights — add 10% invalidation manually for the full 403-session TP2 case); the MTP slider sets the speculative-decode speedup (1.0× = off) and shows the implied per-draft acceptance α it is computed from (speedup = 1 + α + α²; 1.7× ⇔ α ≈ 47%, the measured fit). The purchasing base excludes MTP, CPU offload and N>2 TP; these controls exist so that headroom stays visible, not hidden.
• KV dtype: all constants assume the FP8 KV cache (--kv-cache-dtype fp8_e4m3), as tested in the baseline. The FP16 switch doubles KV bytes/token (half the pool, heavier decode reads); weights and the DeltaNet state are unaffected, and the activation-reserve calibration stays pinned to the FP8 anchor. Sanity: FP16 + 27B + 1×H200 gives a 1.39M-token pool, inside the baseline's measured [1.14M, 1.40M].
• Measured cross-check (2026-07-22): a real 27B + FP16-KV + TP2 bring-up reported 3,233,564 KV tokens at startup (110.59 GiB per worker; 17.54× concurrency at 184,320 tokens) — within 0.3% of this model's 3.24M prediction, with nothing about TP2 or FP16 fitted. First non-circular validation of the reserve + TP arithmetic + FP16 doubling; see docs/scenarios.md § Measured cross-check.
• CPU offload is storage only. A session offloaded to host RAM still counts as warm capacity, but only sessions whose KV is resident in GPU HBM can decode — an offloaded one must be restored over PCIe first (a latency the model does not price). Every decode-side readout (the per-user/aggregate stress tiles, chart C's capacity zone) is therefore computed over HBM-resident warm sessions only, so moving the offload slider never changes per-user decode speed.
• Cold / invalidating requests occupy KV but never count as warm — they cap the reusable-hit ceiling at (1−f).
• Capacity is Monte-Carlo over a user+subagent length mixture; every resident session is also charged its constant DeltaNet recurrent state. Decode speed is per-step bandwidth ÷ (weights + live KV + per-sequence state read/write), sped up by MTP.
• GPU choice (H200 / B300): the H200 is the calibrated baseline. The B300 (Blackwell Ultra: 288 GB HBM3e, 8 TB/s, native FP4) reuses the H200-solved ~18 GiB per-GPU reserve, plus a measured +9.75 GB/GPU correction: the H200 actually delivers ~150.75 GB usable against its 141 GB vendor figure (the calibration silently absorbs that margin), while a real B300 nvidia-smi dump shows 275,040 MiB = 288.4 GB — nominal bytes, no Hopper-style over-provision — so the transferred reserve must add the hidden margin back (research/gpu_b300.md; formerly a sensitivity, measured 2026-07-27).
• NVFP4 weights (B300-only): swaps in real NVFP4-checkpoint byte counts (4.5 bits/param on quantized tensors; what stays high-precision is recipe-specific — see research/nvfp4.md — e.g. RedHatAI's 35B-A3B keeps the DeltaNet blocks BF16 while NVIDIA's GLM recipe quantizes only the routed experts). Gated to native-FP4 GPUs: vLLM's Hopper fallback is weight-only Marlin with a known correctness bug (open as of 2026-07-27), so H-generation NVFP4 is deliberately not modelled. The KV cache is never 4-bit here — an owner policy: vLLM's nvfp4 KV shipped 2026-05 (Blackwell-datacenter-only, values dequantized to FP8 before attention) but is not modelled in this study (research/nvfp4.md). Non-obvious: on the MoE models NVFP4 makes the fixed per-step read heavier (BF16-kept blocks) while expert reads shrink 1.78× — low-concurrency decode slows, high-concurrency speeds up.
• Mistral-Medium-3.5-128B (open weights, 2026-04): dense, 88 uniform GQA layers → 176 KiB/token FP8 KV, no recurrent state, no MTP module (speculative default 1.0×; external EAGLE draft unmeasured). research/model_mistral_medium35.md.
• GLM-5.2 (open weights, 2026-06): 744B-A40B MLA + DeepSeek Sparse Attention; stores 47.3 KiB/token (MLA latent + indexer keys) but decode reads only the top-2048 tokens per layer plus an indexer scan (2,772 B per context token) — priced via kv_decode_bpt/const. vLLM requires FP8 KV on this path, so the FP16 toggle is disabled. FP8 weights (~756 GB) fit from 7×H200 / 4×B300 per the vLLM recipe's 893 GB floor (3×B300 = 864 GB passes the pool arithmetic only); NVFP4 (~465 GB) from 2×B300. research/model_glm52.md.
• Note: the 35B-A3B constants come from the published Qwen3.6-35B-A3B config (40 layers: 30 DeltaNet + 10 full-attention, 2 KV heads × 256, 256 experts / 8 routed) — derivations in research/model_35ba3b.md. Edit the CONFIG block to revise.
• Provenance: the B300 / NVFP4 / Mistral-3.5 / GLM-5.2 constants were researched 2026-07-27 with HuggingFace and NVIDIA domains proxy-blocked — chains run through first-party GitHub repos, three-way config mirrors, and cross-checked snippets, with an assumptions / re-verification ledger in each of the four research notes and per-claim confidence tiers in nvfp4.md and gpu_b300.md. Strongest cross-check: the derived GLM-5.2 NVFP4 resident matches the vLLM recipe's ~465 GB within 0.05%.