R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning

For reinforcement learning in data-scarce domains like real-world robotics, intensive data reuse enhances efficiency but induces overfitting. While prior works focus on critic bias, representation-level instability in Self-Predictive Learning (SPL) under high Update-to-Data (UTD) regimes remains underexplored. To bridge this gap, we propose Robust Representation via Redundancy Reduction (R2R2), a regularization method within SPL. We theoretically identify that standard zero-centering conflicts with SPL's spectral properties and design a non-centered objective accordingly. We verify R2R2 on SPL-native algorithms like TD7. Furthermore, to demonstrate its orthogonality to prior advancements, we extend the state-of-the-art SimbaV2, which originally lacks SPL, by integrating a tailored SPL module, termed SimbaV2-SPL. Experiments across 11 continuous control tasks confirm that R2R2 effectively mitigates overfitting; specifically, at a UTD ratio of 20, it improves TD7 by $\sim$22\% and provides additional gains on top of SimbaV2-SPL, which itself establishes a new state-of-the-art. The code can be found at [this link](https://github.com/songsang7/R2R2).

Training AI for real-world robots is challenging because collecting physical data is slow and expensive. To learn efficiently, AI systems must reuse the same limited data many times. However, this intensive reuse often causes the AI to simply memorize the data rather than learn general skills, a problem known as overfitting. We found that a major cause of this failure is that the AI's internal representation of the environment becomes highly unstable when data is reused too heavily. To solve this, we developed a method called R2R2. It improves how the AI learns to view its environment by actively reducing unnecessary, redundant information. We carefully designed our mathematical objective to preserve the essential features of the data, keeping the AI's internal representations highly stable even under aggressive data reuse. We tested our method across 11 complex virtual robot tasks. By adding R2R2 to various standard and state-of-the-art AI architectures, we successfully mitigated the overfitting issue. It improved the performance of existing methods, TD7, by about 22% and pushed the current top-performing systems to even higher scores. This work provides a more reliable way to train capable AI agents using limited data.