SOLAR for Offline MARL: Plateau-Triggered Potential Shaping under World-Model Uncertainty

Reward shaping can accelerate reinforcement learning, but in sparse-reward \emph{offline} multi-agent RL it is often brittle: dense intrinsic rewards may alter the underlying Markov game, while world-model guidance can amplify model bias. We find that shaping becomes reliable when it is (i) activated only after \emph{statistically validated} learning plateaus and (ii) constrained to \emph{potential-based} shaping, which preserves the task optimum. Motivated by this, we propose \textsc{SOLAR}, a simulate--evaluate--shape framework. A learned world model enables low-cost rollouts to test plateaus; once a plateau is detected, we inject shaping in the form $r+\gamma\Phi(s')-\Phi(s)$ with adaptively updated potentials; and we attenuate shaping using uncertainty-aware throttling in unreliable regions. We provide theoretical analysis on policy invariance and on the deviation of plateau decisions under model error, and establish stability for the resulting two-timescale adaptation. Experiments on sparse-reward offline MARL benchmarks show consistent gains in stability and final performance across dataset qualities.

Many real-world AI systems need to make decisions as a team, such as groups of robots, game characters, or autonomous systems coordinating with one another. A safe way to train such systems is to learn from previously collected data instead of repeatedly trying new actions in the real world. However, this is difficult when the data contains very little feedback. In these cases, the AI system may stop improving because it cannot tell which actions helped the team succeed. This paper introduces SOLAR, a method that helps cooperative AI systems continue learning when progress has stalled. SOLAR first builds a simulated version of the environment from the available data and uses it to check whether the current learning process has reached a plateau. Only when learning appears stuck does SOLAR provide extra guidance. This guidance is carefully designed so that it supports the original task rather than changing what the system is supposed to optimize. SOLAR also reduces its guidance when the simulation is uncertain, preventing the system from relying too much on unreliable predictions. Experiments show that SOLAR helps AI teams learn more stable and successful behaviors, especially when the available data is sparse or low quality. This suggests a safer and more data-efficient way to improve cooperative AI systems without requiring additional real-world interaction.