Tag: 大模型部署 (2 articles)

vLLM and Novita AI collaborate on PegaFlow, externalizing the KV cache into a standalone service with a three-level cache hierarchy, achieving doubled startup speed and significantly higher throughput.

The open-source inference engine vLLM outperforms all proprietary competitors in multiple frontier model inference benchmarks, thanks to deep kernel fusion optimizations tailored to each model's specific bottlenecks.