Inkling, a 1T-parameter open model with native multimodal understanding and 1M context, redefines open-source AI with architecture innovations that enable efficient inference and agentic applications.
EAGLE-3 speculative decoding, deployed on AMD GPUs via vLLM, losslessly accelerates inference for large models like Kimi-K2.5, highlighting a shift toward algorithm-hardware co-design for efficient AI serving.
TTS inference is a heterogeneous pipeline combining latency-bound and throughput-bound stages, making traditional LLM optimization strategies ineffective and requiring architecture-aware scheduling.
Google open-sources DiffusionGemma, applying diffusion architecture to text generation for the first time, achieving over 500 tokens/sec and offering a new paradigm for high-throughput scenarios.
vLLM natively supports a discrete diffusion language model that replaces sequential generation with parallel block denoising, trading compute for bandwidth to significantly reduce latency.
EAGLE 3.1 addresses the performance degradation of speculative decoding in long-context and varied chat templates by introducing FC normalization and post-norm design, doubling acceptance length in long-context scenarios and significantly improving the robustness and practicality of inference acceleration.