Q-Flow: Stable and Expressive Reinforcement Learning with Flow-based Policy

There is growing interest in utilizing flow-based models as decision-making policies in reinforcement learning due to their high expressive capacity. However, effectively leveraging this expressivity for value maximization remains challenging, as naive gradient-based optimization requires backpropagating through numerical solvers and often leads to instability. Existing approaches typically address this issue by restricting the expressive capacity of flow-based policies, resulting in a trade-off between optimization stability and representational flexibility. To resolve this, we introduce Q-Flow, a framework that leverages the deterministic nature of flow dynamics to explicitly propagate terminal trajectory value to intermediate latent states along the policy-induced flow. This formulation enables stable policy optimization using intermediate value gradients without unrolling the numerical solver, effectively bridging the gap between stability and expressivity. We evaluate Q-Flow in the offline learning setting on the challenging OGBench suite, where it consistently outperforms state-of-the-art baselines by an average of 10.6 percentage points, while also enabling stable online adaptation within the same framework.

Researchers are increasingly interested in building AI systems that can represent very flexible and complex behaviors. However, making these systems learn reliably remains difficult: approaches that are highly expressive often become unstable during training, while more stable methods usually limit the system’s flexibility. Our work introduces Q-Flow, a new framework that allows highly expressive models to learn stably without relying on computationally expensive training procedures. Instead, the method efficiently propagates information about successful outcomes throughout the learning process, enabling both stable and effective optimization. We evaluate Q-Flow on a set of challenging decision-making tasks and find that it consistently outperforms existing state-of-the-art methods by a substantial margin. In addition, the same framework can continue adapting after deployment, making it promising for real-world applications such as robotics and autonomous systems.