vLLM introduces adaptive verification that dynamically adjusts the verification budget based on token confidence, making speculative decoding effective even at high concurrency and reducing tuning overhead.
vLLM achieves 25K tokens/sec/GPU on Qwen3.5 through Blackwell-optimized GDN kernels, hybrid cache/state transfer, and race-free async scheduling — the optimization story goes beyond raw numbers.
vLLM open-sources support for parallel speculative decoding algorithms like P-EAGLE, breaking the autoregressive drafting bottleneck for higher acceptance rates and simpler tuning.
EAGLE-3 speculative decoding, deployed on AMD GPUs via vLLM, losslessly accelerates inference for large models like Kimi-K2.5, highlighting a shift toward algorithm-hardware co-design for efficient AI serving.
Tencent Hunyuan contributed production-grade high-performance Attention and MoE kernels to vLLM, achieving up to 2.95x attention speedup for mixed-length decode and 1.59x MoE speedup on H20 GPUs, reducing end-to-end latency by up to 24%.
ServiceNow AI discovered that subtle differences in vLLM V1's inference engine could crash RL training, and restored stability by fixing four critical backend issues.
A comprehensive benchmark by the vLLM team reveals that TurboQuant generally underperforms FP8 quantization and is only potentially viable for extreme memory-constrained edge deployments.
vLLM introduces Elastic Expert Parallelism (Elastic EP), enabling runtime scaling of MoE inference deployments by adding or removing GPU workers without restarts, adapting to demand fluctuations and laying the groundwork for fault-tolerant serving.
The Speculators v0.5.0 release introduces the DFlash algorithm for speculative decoding, which generates draft tokens in a single forward pass, significantly reducing inference latency, and unifies online and offline training workflows.
vLLM uses FP8 quantization for KV cache to halve memory usage and double throughput for long-context inference while maintaining accuracy, though specific performance pitfalls need attention.