Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning

Diffusion-based Q-learning has emerged as a powerful paradigm for offline reinforcement learning, but its reliance on multi-step denoising makes both training and inference computationally expensive and brittle. Recent efforts to accelerate diffusion Q-learning toward single-step action generation typically introduce auxiliary networks, policy distillation, or multi-phase training, which frequently compromise simplicity, stability, or performance. To address these limitations, we introduce Bootstrapped Flow Q-Learning (BFQ), a novel framework that enables accurate single-step action generation during both training and inference—without auxiliary networks or distillation procedures. BFQ adopts a divide-and-conquer view of the displacement vector along the flow path: it begins by learning short-range displacements that can be accurately estimated from the Flow Matching marginal velocity, and bootstraps these components to directly learn a noise-to-action mapping in a single step. This formulation eliminates multi-step denoising, resulting in a learning procedure that is substantially faster, simpler, and more robust. Extensive D4RL evaluations show that BFQ improves performance while significantly reducing computational cost compared to multi-step diffusion baselines, demonstrating that single-step action generation suffices for high-performance offline Reinforcement Learning.

Offline reinforcement learning aims to train decision-making systems using previously collected data instead of costly or risky real-world trial and error. Recent methods based on diffusion models have shown strong performance, but they are often slow and computationally expensive because they require many repeated refinement steps before producing an action. In this work, we introduce Bootstrapped Flow Q-Learning (BFQ), a new approach that can generate high-quality actions in a single step during both training and deployment. Unlike prior acceleration methods, BFQ does not require additional helper models, complicated training stages, or model distillation. Instead, it learns simple local motion patterns and progressively combines them to directly predict effective actions. This design makes BFQ substantially faster, simpler, and more stable than existing diffusion-based approaches while maintaining strong decision-making performance. Experiments on standard offline reinforcement learning benchmarks show that BFQ achieves competitive or improved results while significantly reducing computational cost. Our findings suggest that efficient single-step action generation can be sufficient for high-performance offline reinforcement learning.