DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training

Effectively scaling Reinforcement Learning (RL) is crucial for enhancing the reasoning and alignment of Large Language Models. The massive data and complex execution flows inherent in these tasks require a distributed architecture capable of efficient scaling. However, to simplify programming and dependency management, mainstream frameworks often rely on a centralized architecture where a single node dispatches both control and data. This inherent coupling creates significant communication bottlenecks, severely limiting system scalability and efficiency. We present DistFlow, a novel, fully distributed RL framework that adopts a multi-controller paradigm. By decoupling data transmission from control dispatch, DistFlow establishes a parallelism-aware, decentralized Data Coordinator that leverages local caching, load balancing, and asynchronous double buffer to minimize communication overhead and mitigate straggler effects. For control logic, it introduces a task scheduler built upon Directed Acyclic Graph (DAG) that facilitates fine-grained, independent execution. Experimental results demonstrate that DistFlow achieves near-linear scalability up to 512 GPUs and delivers up to a 2.63x throughput improvement over state-of-the-art (SOTA) frameworks.

Modern AI assistants often improve after initial training by practicing tasks, receiving feedback, and updating themselves. This stage is important for making models reason better and follow human goals more reliably, but it is also very expensive. The work has to be spread across many computer chips, and current systems often send both instructions and large amounts of training data through one central machine. As training grows, that machine becomes a traffic jam: other machines wait for data, expensive chips sit idle, and the whole process can slow down or even fail. DISTFLOW is a new training system designed to remove this traffic jam. Instead of using one central machine to manage everything, it lets many machines share the work. Small instructions are handled separately from the large training data, and the data is kept close to where it is needed whenever possible. The system also balances uneven workloads and prepares the next batch of data while the current one is being used, reducing waiting time. In experiments on language and vision-language models, DISTFLOW scaled smoothly to 512 chips and trained models up to 2.63 times faster than a leading existing system, while producing similar learning results. This means researchers can run the feedback-based training needed for stronger AI systems more efficiently and at larger scale, lowering wasted computing time and making advanced model development more practical.