SPAR: Support-Preserving Action Rectification

Offline policy improvement faces an inherent conflict between maximizing value and fitting the data distribution. While in-sample weighted regression is stable, it suffers from over-conservatism that suppresses high-value actions in the distribution tail; conversely, gradient-based approaches often exhibit a fitting-optimization conflict of gradients, which drive the policy off the data manifold. To address this, we propose \textbf{S}upport-\textbf{P}reserving \textbf{A}ction \textbf{R}ectification (SPAR), which reframes global learning as a local residual rectification anchored to a frozen pure behavior cloning policy. This framework performs fine-grained fitting and local policy improvement in the residual space, thereby contracting the search space. We further introduce Latent Self-Imitation, utilizing a latent-sampling weighted-regression mechanism to address fitting-improvement gradient conflict in the residual space. Theoretically, we prove this mechanism eliminates the manifold-normal drift of standard value gradients, while extensive D4RL experiments show SPAR extracts significant gains from suboptimal baselines to achieve state-of-the-art performance.

This paper studies how to improve decision-making systems when only past data is available and no new trial-and-error interaction is allowed. In this setting, a system should learn better actions from the data, but it must also avoid choosing actions that are too different from what the data can reliably support. We propose SPAR, a method that starts from a simple policy learned from the data and then makes small, carefully selected corrections to its actions. Instead of directly chasing possibly unreliable value estimates, SPAR searches for useful improvements near the original policy and only applies corrections that are likely to remain within the data-supported region. Experiments on standard offline reinforcement learning benchmarks show that SPAR can substantially improve weaker baseline policies while maintaining stable and reliable behavior.