vLLM achieved efficient day-0 support for the trillion-parameter MoE model Kimi K3, paving the way for ultra-large model deployment through key optimizations like hybrid caching and speculative decoding.
vLLM provides day-0 support for TML Inkling, achieving 380 tok/s on 4 GB200 GPUs with full feature parity, 1M context, and multimodal input.
Inkling, a 1T-parameter open model with native multimodal understanding and 1M context, redefines open-source AI with architecture innovations that enable efficient inference and agentic applications.
Tencent Hunyuan contributed production-grade high-performance Attention and MoE kernels to vLLM, achieving up to 2.95x attention speedup for mixed-length decode and 1.59x MoE speedup on H20 GPUs, reducing end-to-end latency by up to 24%.
AI2 releases EMO, a new MoE model pretrained to enable emergent modularity, allowing users to selectively use just 12.5% of experts for a task while maintaining near full-model performance.