GIPO: Gaussian Importance Sampling Policy Optimization

Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation. However, RL remains limited by poor data efficiency, particularly in settings where interaction data are scarce and quickly become outdated. To address this challenge, GIPO (Gaussian Importance sampling Policy Optimization) is proposed as a policy optimization objective based on truncated importance sampling, replacing hard clipping with a log-ratio-based Gaussian trust weight to softly damp extreme importance ratios while maintaining non-zero gradients. Theoretical analysis shows that GIPO introduces an implicit, tunable constraint on the update magnitude, while concentration bounds guarantee robustness and stability under finite-sample estimation. Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency. Code is available at https://github.com/distanceLu/GIPO.

Problem: Training AI for real-world tasks, like robotics, requires highly expensive trial-and-error interactions. To save costs, AI systems reuse their past "old experiences," but current standard training methods often aggressively discard data that seems too outdated, leading to massive data waste. Solution: We developed a new method called GIPO to fix this. Instead of a hard cut-off that completely throws old data away, it uses a smooth, sliding scale to gently reduce the influence of older experiences while still extracting valuable knowledge from them. Impact: This significantly boosts data efficiency and stability, allowing us to train smarter AI systems for the real world using less data and at a much lower cost.