FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matching

Iterative generative policies, such as diffusion models and flow matching, offer superior expressivity for continuous control but complicate Maximum Entropy Reinforcement Learning because their action log-densities are not directly accessible. To address this, we propose \textbf{Field Least-Energy Actor-Critic (FLAC)}, a likelihood-free framework that regulates policy stochasticity by penalizing the kinetic energy of the velocity field. Our key insight is to formulate policy optimization as a Generalized Schr\"odinger Bridge (GSB) problem relative to a high-entropy reference process (e.g., uniform). Under this view, the maximum-entropy principle emerges naturally as staying close to a high-entropy reference while optimizing return, without requiring explicit action densities. In this framework, kinetic energy serves as a physically grounded proxy for divergence from the reference: minimizing path-space energy bounds the deviation of the induced terminal action distribution. Building on this view, we derive an energy-regularized policy iteration scheme and a practical off-policy algorithm that automatically tunes the kinetic energy via a Lagrangian dual mechanism. Empirically, FLAC achieves superior or comparable performance on high-dimensional benchmarks relative to strong baselines, while avoiding explicit density estimation.

Robots and AI agents that learn to perform physical tasks — like opening a door or walking — need to explore many different strategies before settling on good ones. A common technique encourages this exploration by rewarding diversity in the agent's actions. However, newer and more powerful AI models that generate actions through a step-by-step refinement process make it mathematically difficult to measure and reward this diversity, because the likelihood of each action cannot be easily computed. We propose FLAC, a new method that sidesteps this problem entirely. Instead of measuring action diversity directly, we monitor the "energy" of the generation process itself — how much effort the model uses to produce an action. Lower energy naturally corresponds to more diverse, exploratory behavior. Our method automatically balances this energy to maintain healthy exploration while still pursuing high performance. In experiments on challenging robotic control tasks, FLAC matches or outperforms existing methods without ever needing to compute action likelihoods, making it both simpler and more broadly applicable to modern generative AI policies.