vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan

Why It Matters: Real Traffic Is Messy and Mixed-Length

vLLM has become the de facto serving engine for large language models, but its default Attention and MoE backends were originally tuned for relatively uniform, single-length batches. In the real world, requests are dynamic and wildly mixed-length — some prompts are thousands of tokens long while others are single sentences. When you pack them into a batch, a fixed split-KV schedule forces every short request to wait on the longest one, leaving GPU compute units idle. MoE models compound the problem: without careful operator fusion, the overhead of launching many small expert GEMMs and moving intermediate results to and from HBM can dominate the actual compute time. Tencent Hunyuan’s infrastructure team experienced this messiness every day at production scale. They responded by building HPC-Ops, a hardened operator library, and they have now upstreamed its Attention and MoE kernels directly into vLLM’s main branch.

Breaking Down the Kernels

• Attention Backend: A Dynamic, Load-Balanced Scheduler During decode, the traditional approach statically partitions KV-cache chunks among compute units. HPC-Ops replaces this with a per-step, load-balanced scheduler: it evaluates progress dynamically and prioritizes nearly-finished short requests, while more evenly distributing longer ones across the grid. It also fuses RoPE, query-key normalization, and KV-write into a tight prologue, slashing kernel launch overhead and HBM traffic. On H20 GPUs, this dynamic scheduling yields up to a 2.95× speedup over static KV splitting for mixed-length decode, and on average it outperforms FlashInfer and FlashAttention by 2.25×.

• MoE Backend: A Fully Fused FP8 Pipeline The MoE bottleneck isn’t the large matmul; it’s the token dispatch to dozens of experts, which results in many small GEMMs and intermediate data movement. The HPC-Ops MoE backend fuses token rearrangement, expert GEMMs (in FP8), activation functions, and result gathering into a single low-latency kernel pipeline, practically eliminating intermediate HBM reads/writes and launch costs. Under both tensor-parallel (TP8) and expert-parallel (EP8) configurations, it delivers average speedups of 1.59× and 1.21× over Triton and CUTLASS baselines, with strictly matched output quality.

The Bigger Trend: The Bottleneck Moves from FLOPs to Scheduling and Data Flow

This work highlights a profound shift: as models go MoE, context windows extend, and agentic workloads multiply, latency is no longer bounded by raw FLOPs. Much like training where communication optimization eventually outweighs pure compute, in inference, how kernels schedule work across the GPU and fuse operations to reduce memory traffic is becoming the decisive factor for user experience. HPC-Ops is not just a set of performant operators; it represents a paradigm of “designing computation for real traffic patterns” — a philosophy that will increasingly dominate serving framework evolution.

Practical Value: vLLM Users Can Benefit Right Now

The most immediate takeaway: you don’t need to modify a single line of model code or maintain a long-lived fork. Once your vLLM install includes the relevant PRs, you can select the HPC-Ops Attention or MoE backend with simple configuration flags. If you’re serving MoE models like DeepSeek, Qwen, or Mixtral on H20 or other Hopper GPUs, it’s highly recommended to try these new backends. The end-to-end latency reduction directly improves user experience and system throughput.

A Counterintuitive Truth: 2–3× Gains on the Same Hardware

Many assume that once you have the latest GPU (e.g., H100, H20), there’s little performance left on the table. Reality paints a different picture: when a batch mixes short and long requests, a naive scheduler can drop GPU utilization below 50%. The HPC-Ops results show that smarter computation organization alone can squeeze 2–3× more performance out of the same silicon. It’s not magic; it’s solid engineering — and it’s becoming available out of the box.