The Lab Notes

vLLM

vLLM trades up to 18% slower output for a bigger KV cache

vLLM now sizes its KV cache from free GPU memory by default, removes Model Runner V1 and raises the default block size from 16 to 64.

A tall stack of flat white square modules with green and black edges and small red tabs sits on an orange background patterned with blue and yellow rectangles.

vLLM, the open-source server for running large language models, now sizes its KV cache from the GPU memory that is actually free, which fits more tokens in the cache but slows output on some models. The release also removes Model Runner V1, so servers that still select it will not start.

The vLLM project published v0.34.0 as the latest stable release on GitHub, and its release notes are dated Nov. 16, 2026. They list 1,061 commits from 412 contributors, 120 of them new.

The KV Cache Grows and Output Slows

The extensible KV cache is now on by default on CUDA and ROCm, the software stacks for Nvidia and AMD GPUs. The KV cache is the memory that holds attention data for tokens a model has already processed, so it does not have to recompute them. The release says the cache is now “sized from memory actually free after CUDA graph capture and warmup” instead of from an estimate.

The release’s example is MiniMax-M3 on an H200 GPU, where the cache grows from 433k to 672k tokens. The release also reports that time per output token, or TPOT, rises 6% to 18% on some H200 models at concurrency 128.

With the extensible cache on, gpu_memory_utilization now defaults to 1.0 when unset. Operators who want the old sizing can pass --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache.

Model Runner V1 Is Gone

Model Runner V2 is now the only runner in vLLM. Setting the VLLM_USE_V2_MODEL_RUNNER environment variable to anything but 1 “fails at startup,” so operators have to remove VLLM_USE_V2_MODEL_RUNNER=0 from their configs. Logits processors written for V1 have to be ported to vllm.v1.worker.gpu.sample.logits_processor.

Other Upgrade Checks

The default --block-size changed from 16 to 64. Passing --block-size 16 keeps the old behavior. Backends with a preferred block size still override the default, and ROCm paged attention still uses 16.

The project no longer publishes ROCm 7.2 wheels or -rocm72 images. Release images and wheels are ROCm 10.0 only.

New minimum versions are huggingface_hub 2.0.0, tokenizers 0.23.3, timm 1.0.30 and mistral-common 1.13.0.

The release removes QuTLASS and fp_quant, and fp_quant checkpoints no longer load. It also removes Marlin W4A8 and VLLM_MARLIN_INPUT_DTYPE, and says to switch to --quantization humming with VLLM_HUMMING_INPUT_QUANT_CONFIG. The Keye-VL and Keye-VL-1.5 models are gone too.

Best-effort NUMA binding is now on by default, and --no-numa-bind turns it off.

The release counts 10 new models, among them Mistral Large 4, MiniMax-M3.1-Flash and Nemotron-3.5-Transcribe.

Our View

We think the memory default is the setting to look at first, because one number now drives both the gain and the cost. With the extensible cache on, gpu_memory_utilization defaults to 1.0, so the server sizes its cache from the memory left free after warmup. For an operator, that means more tokens held in the cache on the same card, which the MiniMax-M3 example puts at 672k instead of 433k. It also means the 6% to 18% rise in time per output token that the release reports on some H200 models.

An operator whose users wait on each token will feel the slowdown first, and the release gives two ways back to the old sizing. An operator who has been running out of cache room gets the larger cache without touching a flag. The release reports the slowdown on some H200 models at concurrency 128, so the gain holds for an operator if a test on their own model and load shows the added cache room is worth more than the slower tokens.