Announcing VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models

The Context: Why is VeRL-Omni Needed Now? Over the past year, the Reinforcement Learning (RL) training stack for Large Language Models (LLMs), such as RLHF and GRPO, has evolved rapidly and become a core method for aligning models with human preferences and downstream task rewards. However, when this mature methodology is extended to the field of multimodal generation, it faces significant adaptation challenges. Generative models for images, videos, and audio—especially those based on Diffusion Transformers and omni-modality architectures—are fundamentally different from autoregressive text generation. Their "generation" process involves multi-step denoising in a continuous latent space, rather than predicting discrete token sequences one by one. This leads to three major engineering hurdles: first, how to efficiently schedule complex generation pipelines composed of heterogeneous components like text encoders, DiTs, and VAEs; second, how to manage the extremely high memory peaks brought by multimodal generation (especially video); and third, how to design reward functions capable of handling multimodal inputs (e.g., using a VLM to judge the OCR accuracy in a generated image). Existing LLM RL frameworks cannot directly address these challenges. Therefore, VeRL-Omni emerges precisely to fill this critical gap, endowing the next generation of multimodal generative models with the powerful alignment capabilities of RL.

Breakdown: What Problems Does It Actually Solve? VeRL-Omni is not built from scratch; rather, it is constructed on top of the mature verl (an LLM RL framework) and vLLM-Omni (a multimodal inference engine). Its core value lies in a set of meticulously designed engineering solutions, which can be loosely understood as installing an "intelligent scheduling center" and an "efficient pipeline" for multimodal RL training. First, by integrating vLLM-Omni, it achieves efficient multimodal rollout generation. The high-throughput asynchronous serving of vLLM-Omni, combined with techniques like step-wise continuous batching and embedding caching, ensures speed and accuracy in the generation phase. Second, it provides a flexible reward engine that can utilize simple rule-based rewards or integrate models like VLMs as "judges" for complex evaluations (e.g., determining if the text in a generated image is correct). Crucially, it overlaps reward computation with generation and training processes to minimize waiting time. Finally, it offers modular training backends supporting various parallelism strategies like FSDP, USP, and TP, with built-in optimizations for diffusion and omni-modal model characteristics, while also being compatible with both NVIDIA and Ascend hardware. These features collectively make it feasible and efficient to train advanced RL algorithms like FlowGRPO and MixGRPO on models such as Qwen-Image (text-to-image) and BAGEL (unified understanding and generation).

Trend Insight: RL is Becoming a "Standard Feature" for Multimodal Models The release of VeRL-Omni reveals a deeper trend: Reinforcement Learning is evolving from a "post-training alignment tool" for LLMs into a "standard capability enhancer" for all generative foundation models. Just as RLHF fundamentally transformed the conversational quality of ChatGPT, RL training based on human preferences or task rewards is becoming a key component for improving the controllability of image generation, the coherence of video generation, and the naturalness of audio generation. This signifies that the focus of multimodal AI development is shifting from "whether it can generate" to "how well and how intentionally it can generate." Furthermore, the framework's support for heterogeneous hardware indicates that multimodal RL training is becoming more engineered and platform-based, lowering the barrier for developers outside of top research labs to apply this technology.

Practical Value: What Does This Mean for Developers? For AI practitioners, especially those focused on multimodal generation, VeRL-Omni's value lies in providing a reproducible, high-performance pathway. If you are fine-tuning a text-to-image or text-to-video model and want it to generate images with fewer artifacts, more accurate text, or videos with more coherent motion, you can now try using VeRL-Omni, combined with a VLM as a reward model, to perform RL post-training on your diffusion model. The framework provides ready-to-use examples (like the OCR reward task for Qwen-Image) and performance benchmarks, significantly bridging the gap between research papers and engineering implementation. It makes complex multimodal RL training, previously accessible only to large corporate research departments, attainable for the broader developer community.

Counterintuitive/Overlooked Angle: A Perspective Easily Missed A potentially overlooked point is that VeRL-Omni supports not only generation tasks but also multimodal understanding tasks (such as text/image/video/audio understanding with Qwen3-Omni-Thinker). This means RL training can optimize not just "generation" but also "understanding"—for example, making a model more accurate and logically consistent when answering questions about an image. This suggests that the application of RL in the multimodal domain is broader than initially imagined, and it might become a universal training paradigm for uniformly enhancing a model's "perception" and "creation" capabilities.