Direct Flow Q-Learning

Flow Matching shows great promise in offline reinforcement learning (RL), yet optimizing these iterative policies via Backpropagation Through Time (BPTT) is unstable. While prevailing paradigms circumvent this by distilling multi-step flows into single-step approximations, such methods may limit the benefits of iterative refinement. To avoid these sacrifices, we propose Direct Flow Q-Learning (DFQL), a streamlined framework that attains superior results by optimizing flow matching policies without BPTT or distillation. DFQL derives a surrogate objective that directly injects terminal Q-value gradients as a guidance term into each step velocity field, ensuring stable optimization while preserving iterative expressive capacity. Across 73 challenging tasks in OGBench and D4RL, DFQL achieves state-of-the-art results. Additionally, DFQL extends seamlessly to the offline-to-online setting, delivering substantial performance gains without further modification.

This paper studies offline reinforcement learning, where an agent learns from a fixed dataset without interacting with the environment. Recent flow-matching policies can model complex action distributions, but optimizing them with value functions is difficult because gradients through multiple generation steps can be unstable. We propose Direct Flow Q-Learning, which injects value-function guidance directly into the flow velocity field instead of backpropagating through the entire generation process. This leads to more stable training while keeping the expressiveness of multi-step flow policies. Experiments on OGBench and D4RL show that the method achieves strong performance in both offline and offline-to-online reinforcement learning.