RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning

On-policy Reinforcement Learning (RL) remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy, and policy updates must be conservative. In this paper, we revisit actor policy representation as a first-class design choice for on-policy RL. We study discretized categorical actors, which represent each action dimension as a distribution over discrete bins and induce a policy objective analogous to classification cross-entropy loss. Building on architectural advances from supervised learning, we further pair discretized categorical actors with regularized networks, yielding RN-D. Across diverse continuous-control benchmarks, we show that simply replacing the standard Gaussian actor with our proposed actor substantially improves performance, achieving state-of-the-art results within on-policy RL. We release our code at https://github.com/alwaysbyx/RND-RL.

Reinforcement learning (RL) trains machines to make decisions through trial and error, and is widely used for robotics, control systems, and aligning large language models. A popular family known as on-policy RL is the standard tool for continuous control tasks — for example, teaching a simulated robot to walk or to pick up an object. However, these methods are notoriously hard to train: unstable, sensitive to small implementation details, and often slow to converge. We revisit a design choice the field has long taken for granted: how the AI represents its actions. The standard approach outputs a continuous bell-shaped curve over possible actions. Instead, we let the AI choose from a small set of discrete options — much like keys on a piano — and pair this with a deeper, carefully regularized network. This turns each policy update into something resembling a familiar classification problem, which neural networks handle very reliably. Across diverse robot locomotion and manipulation benchmarks, using both sensor and camera inputs, this simple change yields faster, more stable, and more successful learning. It reaches state-of-the-art performance among on-policy methods while needing 1.3–1.9× fewer training samples. Because the change is a "drop-in" replacement that works across different on-policy algorithms, it offers a broadly useful improvement for the many researchers and practitioners who rely on this paradigm.