The State of FP8 KV-Cache and Attention Quantization in vLLM

Why FP8 Quantization Matters Now When context lengths exceed 128k tokens, KV cache memory usage starts to dominate GPU memory. This means each decoding step requires reading massive amounts of cached data, shifting inference systems from compute-bound to memory-bound. The vLLM team discovered that storing KV cache in FP8 format could theoretically halve memory usage—provided accuracy doesn’t degrade significantly. This article answers the critical question: Is FP8 quantization reliable in production?

Technical Breakthroughs and Pitfalls of FP8 Quantization The --kv-cache-dtype fp8 feature in vLLM isn’t new, but stress tests revealed two key issues. First, the accuracy pitfall: On Hopper GPUs, when context length reaches 128k, FP8 attention computation suffers from precision loss in Tensor Core accumulation, causing accuracy on the "needle-in-a-haystack" task to plummet from 91% to 13%. This is fundamentally a hardware-level issue—when the contraction dimension exceeds 100k, FP32 register accumulation lacks sufficient precision. The solution involves a two-level accumulation strategy that writes partial results to actual FP32 registers, restoring accuracy to 89% at the cost of increased register pressure.

Second, the performance pitfall: For models with sliding-window attention (e.g., gpt-oss-20b), FP8 quantization provides minimal decoding speedup (only 4% faster than BF16) because memory savings primarily occur in global attention layers, while sliding-window layers themselves have low memory footprint. vLLM addresses this by allowing users to skip these layers via --kv-cache-dtype-skip-layers sliding_window.

Trend Insight: Quantization Becomes Standard for Inference Systems This article reveals a deeper trend: Inference optimization is shifting from "brute-force hardware scaling" to "fine-grained memory management." FP8 quantization isn’t simply about reducing numerical precision—it requires coordination between hardware characteristics, kernel optimizations, and model architecture. For example, FP8 still causes performance regression during prefill for large-head-dimension models (head_dim=256), but accelerates both prefill and decoding for head_dim=64/128. This shows there’s no one-size-fits-all optimization; strategies must be tailored to model architecture.

Practical Value: How Should Developers Use This? For most developers using mainstream models like Llama, enabling --kv-cache-dtype fp8 yields significant benefits—halving memory usage while improving decoding speed up to 54% of BF16 cost per token. However, for hybrid attention models, always skip sliding-window layers. Additionally, the vLLM team has tested the FlashInfer backend on Blackwell GPUs (B200), where FP8 quantization performs even better. Notably, when models require high-precision inference (e.g., complex logical tasks), consider running calibration tests or temporarily reverting to BF16.

Counterintuitive Insights: Quantization Isn’t a "Free Lunch" Many assume quantization is simply about reducing numerical precision, but FP8 exposes hardware-level flaws in long-context scenarios. For instance, Hopper GPU Tensor Cores have known precision issues during FP8 accumulation, which even affected DeepSeek-V3 training. vLLM’s two-level accumulation scheme essentially uses software to compensate for hardware limitations. Another surprise is the minimal quantization benefit for sliding-window layers—this reminds us that model architecture details significantly impact optimization effectiveness, and generic solutions shouldn’t be applied blindly.