TTS inference is a heterogeneous pipeline combining latency-bound and throughput-bound stages, making traditional LLM optimization strategies ineffective and requiring architecture-aware scheduling.
Google open-sources DiffusionGemma, applying diffusion architecture to text generation for the first time, achieving over 500 tokens/sec and offering a new paradigm for high-throughput scenarios.
vLLM natively supports a discrete diffusion language model that replaces sequential generation with parallel block denoising, trading compute for bandwidth to significantly reduce latency.
EAGLE 3.1 addresses the performance degradation of speculative decoding in long-context and varied chat templates by introducing FC normalization and post-norm design, doubling acceptance length in long-context scenarios and significantly improving the robustness and practicality of inference acceleration.