Tag: AI推理 (2 articles)

By integrating Mooncake's distributed KV cache store, vLLM overcomes the efficiency bottleneck of recomputing long-context prefixes in AI Agent workloads, achieving a 3.8x throughput increase and 46x lower time-to-first-token.

Hugging Face integrates the cost-effective inference platform DeepInfra into its Inference Providers ecosystem, offering developers more model choices, flexible billing, and a unified API.