vLLM Reaches 25K Total TPS/GPU on Qwen3.5

Qwen3.5, released in early 2026, remains a workhorse model in many production systems. But the recent vLLM blog post reveals a staggering performance milestone: on GB200 NVL72 systems, vLLM achieves 25,000 total tokens per second per GPU. This isn't just a vanity metric — it means serving more users with fewer GPUs, lower latency, and drastically shorter batch processing times.

More interesting than the number itself is the engineering story behind it. Qwen3.5 uses a hybrid attention architecture: some layers are standard full-attention, while others employ Gated Delta Network (GDN), a variant of state-space models. This creates two major challenges: accelerating GDN computation on modern hardware, and correctly transferring two different kinds of state between prefill and decode workers in a disaggregated serving setup.

First Hurdle: Squeezing GDN Performance on Blackwell GPUs

The original GDN implementation relied on Triton for portability, but it couldn't exploit Blackwell's new features. The vLLM community contributed a Blackwell-optimized GDN prefill kernel using FlashInfer. In microbenchmarks, it delivered speedups ranging from 1.02x to 5.78x depending on model size, parallelism, and sequence length. Integrated into vLLM, this kernel improved end-to-end prefill throughput by 13% and reduced time-to-first-token by 12%. For large models like Qwen3.5-397B, where GDN layers dominate, such gains are transformative.

Second Hurdle: Moving Two Kinds of State Across Nodes

Disaggregated serving splits prefill and decode across different machines, necessitating state transfer. Pure Transformer models only need KV caches; Qwen3.5 also requires GDN's convolutional state. vLLM implemented dual descriptor views and homogeneous tensor-parallel support, allowing NIXL to simultaneously transfer KV cache and SSM state. It also correctly maps logical blocks to physical memory for each layer type, slashing descriptor count from 4,284 to 1,650 and improving throughput by roughly 7% even in a small H100 test. A dedicated GDN support patch closed the loop for Qwen3.5.

Third Hurdle: The Misunderstood Async Scheduler

Async KV block transfer was supposed to overlap communication and computation for higher throughput, but enabling it previously caused total accuracy collapse. The vLLM team finally found and fixed two race conditions where block transfer completion signals clashed with scheduler state updates. After the fix, async scheduling not only became stable but unlocked an additional 4.5% throughput gain. Now enabled by default, it's a free performance boost for anyone upgrading vLLM.

What This Reveals About the Inference Landscape

25K TPS/GPU isn't a one-off achievement — it signals a shift from general-purpose inference engines to model-optimized vertical stacks. Just as compilers generate vectorized code for specific CPUs, inference frameworks will increasingly tune for a handful of dominant models. For developers, the takeaways are clear:

1. Hardware-aware optimization matters. The 5.92x kernel speedup shows that next-gen GPUs hide latent performance that generic code can't reach. Staying current with inference engine releases avoids leaving money on the table.
2. Hybrid architectures are the new normal. With Qwen3.5 and successors, expect more models mixing attention and SSM layers. Serving them efficiently will need native support — don't assume vanilla Transformer pipelines will suffice.
3. Disaggregated serving is not optional. Long-context prompting makes prefill/decree separation imperative for cost control. vLLM's Qwen3.5 foundation can be reused for any similar model.

Ultimately, 25K TPS/GPU is the product of months of community effort across kernels, state transfer, and scheduling. It's a reminder that in the AI engineering era, making a model fast requires sustained, collaborative grinding — and your competitors might already be using these same tools to gain an edge.