What Does Flow-Matching Bring to TD-Learning?

Recent work shows that flow-matching networks can be effective for value function estimation in reinforcement learning, but it remains unclear why they work well or whether flow-matching Q-functions differ fundamentally from standard critics. We show that their success is not explained by distributional RL: explicitly modeling return distributions often degrades performance. Instead, we argue that flow-matching Q-functions are effective because they couple a learned velocity field with an integration procedure that is used both during training and to read out Q-values at inference time. This coupling enables robust value prediction through \emph{test-time recovery} from imperfect intermediate estimates where errors dampen out as more integration steps are performed. This mechanism is absent in monolithic critics. Beyond test-time recovery, training with the integration procedure induces more \emph{plastic} representations, allowing critics to represent non-stationary future TD targets without overwriting previous features. We formalize these effects and validate them empirically, showing that flow-matching critics outperform monolithic critics by over $2\times$ in performance and achieve $5$–$10\times$ higher sample efficiency in high-UTD regimes.

Reinforcement learning systems learn by repeatedly estimating how good different actions are, but these estimates can become unstable as learning progresses. This problem becomes especially severe in difficult settings where the system updates its estimates many times using constantly changing information, often causing learning to slow down or fail. Our research asked a simple question: can a newer machine learning technique called flow matching, originally designed for generating images and videos, help reinforcement learning make more reliable decisions? We found that flow matching improves learning in an unexpected way. Instead of producing a value estimate in a single step, it gradually refines the estimate through a sequence of smaller corrections, allowing the system to recover from earlier mistakes. We also found that this process helps the model remain more adaptable as learning changes over time, preventing it from forgetting useful information too quickly. These improvements make reinforcement learning systems more stable, faster to train, and better at solving difficult decision-making problems, helping pave the way toward more reliable AI systems.