SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance. We provide evidence consistent with the hypothesis that, in the loss landscape, offline maxima for prior algorithms and online maxima are separated by low-performance valleys that gradient-based fine-tuning traverses. Following this, we present Score Matched Actor-Critic (SMAC), an offline RL method designed to learn actor–critics that transition to online value-based RL algorithms with no drop in performance. SMAC avoids valleys between offline and online maxima by regularizing the Q-function during the offline phase to respect a first-order derivative equality between the score of the policy and action-gradient of the Q-function. We experimentally demonstrate that SMAC converges to offline maxima that are connected to better online maxima via paths with monotonically increasing reward found by first-order optimization. SMAC achieves smooth transfer to Soft Actor-Critic and TD3 in 6/6 D4RL tasks. In 4/6 environments, it reduces regret by 34-58% over the best baseline.

This paper studies a practical problem in reinforcement learning: how to take a policy trained from old, offline data and safely improve it through new online experience. Offline reinforcement learning can learn strong policies without interacting with the environment, which is useful when real-world interaction is expensive or risky. But when these offline-trained agents are later fine-tuned online with standard RL algorithms, they often get worse before they get better. That early performance drop is dangerous in settings like robotics, where a bad policy during fine-tuning can cause real failures. The paper argues that this drop happens because many offline RL methods find solutions that are good on their own, but poorly positioned for online fine-tuning. In the space of neural network parameters, the offline solution and the online solution may be separated by a low-performance region. So when gradient-based fine-tuning moves from one to the other, the agent temporarily passes through a bad policy. The proposed method, Score-Matched Actor-Critic (SMAC), tries to train offline agents that are not just good initially, but also easier for online RL algorithms to keep improving. It does this by shaping the critic so that its action preferences resemble the structure expected by online actor-critic methods like SAC. In simpler terms, SMAC trains the value function to point the policy toward actions that look both data-supported and compatible with later online learning. Across several benchmark tasks, SMAC avoids the sharp performance drops seen in prior offline RL methods. It fine-tunes smoothly with SAC and TD3, and often achieves much lower online regret, meaning it wastes fewer online interactions performing poorly. The broader message is that offline RL should not only optimize for the best offline checkpoint; it should optimize for checkpoints that sit on a good path toward future online improvement.