vLLM and Novita AI collaborate on PegaFlow, externalizing the KV cache into a standalone service with a three-level cache hierarchy, achieving doubled startup speed and significantly higher throughput.
ServiceNow AI discovered that subtle differences in vLLM V1's inference engine could crash RL training, and restored stability by fixing four critical backend issues.
vLLM's comprehensive testing reveals that FP8 KV-cache quantization can significantly reduce memory usage and decoding costs under specific conditions, but introduces critical accuracy and performance pitfalls in certain models and scenarios, requiring careful adoption.