Poolside's 33B-parameter agentic coding model, Laguna XS.2, achieves 2-3x inference speedup without quality loss through native vLLM integration, DFlash speculative decoding, and LLM Compressor quantization.
The EAGLE team, in collaboration with vLLM and TorchSpec, releases EAGLE 3.1, which significantly improves speculative decoding robustness and acceptance length in long-context and varied chat scenarios by addressing the 'attention drift' problem.
NVIDIA's new diffusion language models generate tokens in parallel and refine them iteratively, potentially breaking the latency limits of traditional autoregressive models and enabling self-correction.
vLLM and Novita AI collaborate on PegaFlow, externalizing the KV cache into a standalone service with a three-level cache hierarchy, achieving doubled startup speed and significantly higher throughput.
Hugging Face reveals the bottleneck of alternating CPU/GPU waits in continuous batching, and shows how asynchronizing their workloads can yield a free 24% throughput boost.
A large-scale benchmark by the vLLM team reveals that while TurboQuant's extreme low-bit compression saves memory, it significantly degrades inference speed and accuracy, making FP8 quantization the current best balance.
The open-source inference engine vLLM outperforms all proprietary competitors in multiple frontier model inference benchmarks, thanks to deep kernel fusion optimizations tailored to each model's specific bottlenecks.
DeepSeek-V4 makes million-token context windows practically usable for long-running AI agents by dramatically cutting inference costs and memory usage through its novel hybrid attention architecture.
vLLM announces support for DeepSeek V4 models, featuring a novel attention mechanism that tackles the core challenges of memory and computational cost in million-token long-context inference.
vLLM's comprehensive testing reveals that FP8 KV-cache quantization can significantly reduce memory usage and decoding costs under specific conditions, but introduces critical accuracy and performance pitfalls in certain models and scenarios, requiring careful adoption.