Generative Online Reinforcement Learning

Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e.g., Gaussians) are often too simple to represent the multimodal action distributions required for complex control. Conversely, expressive generative policies—such as diffusion and flow matching—can be difficult to optimize in online RL due to intractable likelihoods and gradients propagating through long sampling chains. We address this tension with a key structural principle: *decoupling optimization from generation*. Building on this, we introduce GoRL (Generative Online Reinforcement Learning), an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder. Using a two-timescale alternating schedule and anchoring decoder refinement to a fixed prior, GoRL enables stable optimization while continuously expanding expressiveness. Empirically, GoRL consistently outperforms unimodal and generative baselines across diverse continuous-control tasks. Notably, GoRL achieves returns exceeding **870** on HopperStand, more than $3\times$ the strongest baseline; on high-dimensional humanoid tasks, it further outperforms the strongest non-GoRL baseline by over an order of magnitude.

Reinforcement learning trains decision-making systems by letting them interact with an environment and improve through trial and error. Many common reinforcement learning methods use simple action patterns because they are easier to train, but these simple patterns can be too limited for difficult control tasks where several different actions may be useful in the same situation. More flexible generative models can represent richer behavior, but they are often hard to train reliably when learning online from new experience. This paper introduces GoRL, a framework that separates the part of the system responsible for stable learning from the part responsible for generating flexible actions. The learning component searches in a simple latent space, while a generative decoder converts those latent choices into real actions. By updating these two parts in stages, GoRL can remain stable while gradually learning more expressive behavior. Experiments on simulated continuous-control tasks show that GoRL performs better than standard simple policies and prior generative-policy baselines, including on challenging humanoid tasks.