TML Inkling on vLLM: Day-0 Support with Optimized Performance
vLLM provides day-0 support for TML Inkling, achieving 380 tok/s on 4 GB200 GPUs with full feature parity, 1M context, and multimodal input.
- vLLM provides day-0 support for TML Inkling with NVFP4 and BF16 variants
- 1T-parameter multimodal model supporting text, image, and audio with 1M context window
- Novel architecture with relative attention, short convolutions, and expert sinks for efficiency
- Achieves 380 tok/s (with MTP) and 140 tok/s (without) on 4 GB200 GPUs
- Full feature parity including LoRA, parallelism strategies, prefix caching, and disaggregated serving
Why does this matter right now?
In the AI inference infrastructure space, vLLM has become the de facto standard framework. When TML (Thinking Machines Lab) released Inkling, a 1-trillion-parameter multimodal model, the vLLM team didn't wait. They delivered Day-0 support, meaning developers could run the model on release day, not just barely working, but with full optimizations and feature parity. This speed reflects a broader shift: AI infrastructure is moving from reactive adaptation to synchronized co-evolution.
What makes Inkling different?
Inkling isn't just another parameter-heavy model. Its architecture introduces several clever design choices. First, the 1-million-token context window doesn't rely on brute-force compute. Instead, it combines sliding-window attention with relative positional encoding, making long-context processing genuinely efficient. Second, it aggressively uses short convolutions (sconv) with a window size of 4. Each layer contains four sconv modules that act like tiny local attention mechanisms, adding minimal compute and memory overhead. Most notably, Inkling introduces an expert sink mechanism: two shared experts participate in routing score computation but are excluded from top-6 selection. This acts like a probability absorber, stabilizing expert routing in the mixture-of-experts setup.
What do the performance numbers tell us?
On 4 GB200 GPUs, the model achieves 380 tokens per second per user with multi-token prediction (MTP) and 140 tokens per second without it. For a 1T-parameter model, this is highly competitive. More importantly, vLLM didn't just make it run. The team implemented Sconv-aware tensor parallelism sharding, low-latency fused collectives, kernel fusion, and multi-stream execution. This shows that inference framework competitiveness has shifted from making models run to making them run fast and reliably.
What does this mean for developers?
If you're building multimodal applications, Inkling natively accepts text, image, and audio inputs with a 1-million-token context. You can feed entire project documentation, conversation history, or even audio meeting transcripts in one go. vLLM also provides enterprise-grade features like LoRA, prefix caching, and disaggregated serving. Combined with the NVFP4 quantized variant, you get substantial memory savings without sacrificing accuracy. For teams wanting to quickly prototype long-context multimodal workflows, this is a ready-to-use stack.
An overlooked trend
Most people focus on parameter counts and benchmark scores, but what truly determines deployment speed is Day-0 infrastructure support. This time, vLLM didn't just load the model. They verified tool-call parsing, function calling accuracy, and ran evaluations across MMAU, MMMU-Pro, and other benchmarks. This reveals a deeper trend: model releases and inference deployment are shifting from sequential steps to parallel engineering. Going forward, whoever can turn new models into stable, production-ready services first will control the pace of real-world AI adoption.
Analysis by BitByAI · Read original