Stabilizing PPO via Latent-Space Regularization and KDE-Driven Exploration

Proximal Policy Optimization (PPO) is widely used in continuous-control tasks, yet its performance is often highly sensitive to training dynamics when neural networks approximate the policy and value functions. This paper introduces SPPO, a drop-in augmentation that preserves PPO’s clipped objective and network architecture while stabilizing actor-critic geometry via three mechanisms: (i) a CKA-based constraint on critic representations, (ii) a no-flip regularizer on actor updates, and (iii) KDE-driven advantage shaping. Theoretical analysis shows that these mechanisms tighten bounds on one-step bootstrapping error, improve expected directional alignment of action updates, and ensure non-decreasing occupancy mass over high-novelty regions. Experiments on standard continuous-control benchmarks demonstrate consistent gains over PPO and recent PPO stabilization methods. Ablation studies further quantify the contribution and complementary effects of each component. Additional training-dynamics analyses indicate that SPPO reduces instability and oscillations in both actor and critic updates, improving training stability and final performance.

A widely used method for reinforcement learning is Proximal Policy Optimization, or PPO, which is often used to train agents for continuous-control tasks such as simulated robots and locomotion. However, PPO can sometimes be unstable, because it updates two neural networks at the same time: one that chooses actions and one that estimates how good the current situation is. This paper introduces SPPO, a simple extension of PPO designed to make this learning process more stable. SPPO adds three stabilizing mechanisms. The first keeps the value-estimation network from changing its internal representation too abruptly. The second discourages the action-selection network from making updates that reverse the intended direction of improvement. The third encourages the agent to pay more attention to useful but underexplored experiences. Experiments on standard continuous-control benchmarks show that SPPO improves both training stability and final performance compared with PPO and several recent stabilization methods. These results suggest that improving the geometry of learning updates can make reinforcement learning algorithms more reliable without making them substantially more complex.