vLLM achieved efficient day-0 support for the trillion-parameter MoE model Kimi K3, paving the way for ultra-large model deployment through key optimizations like hybrid caching and speculative decoding.
vLLM, together with Moonshot AI, NVIDIA, and AMD, is racing to bring day-0 serving support for Kimi K3 by reinventing prefix caching for KDA attention and fusing kernels across hardware stacks.
ServiceNow AI discovered that subtle differences in vLLM V1's inference engine could crash RL training, and restored stability by fixing four critical backend issues.