From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning

Offline-to-online reinforcement learning (O2O RL) faces a central challenge between retaining offline conservatism and adapting to online feedback under distribution shift. This challenge arises because data behavior evolves during fine-tuning, rendering data origin a misleading basis for constraint handling and thereby leading to objective–data mismatch. We therefore propose Dynamic Alignment for RElaxation (DARE), a distribution-aware framework for sample-level constraint relaxation based on the behavioral consistency with a behavior model. To our knowledge, DARE is the first to condition constraint relaxation on behavioral consistency via a posterior-induced exchange mechanism, moving beyond a binary offline/online data distinction. Importantly, DARE requires only per-sample behavioral alignment, enabling instantiation on top of many offline algorithms with flexible choices of behavior models and fine-tuning objectives. We provide a theoretical analysis showing that behavior-based sample exchange consistently improves the distinction between offline-like and online-like subsets. Experiments on D4RL demonstrate that DARE consistently improves fine-tuning stability and achieves superior final performance over strong offline-to-online baselines. (The code is publicly available at \url{https://github.com/lpzu/DARE}.)

We want AI agents, such as robots, to learn from past experience and then improve through their own practice. The main challenge is matching the learning rule to the role of the data. Some data should help keep learning stable, while other data should help the agent improve through practice. Treating data only based on when it was collected can create a mismatch and slow down learning. To address this, we developed DARE, a framework that looks at the behavior shown by each training example. It keeps a more cautious rule for examples that resemble past experience, and uses a more flexible rule for examples that better support practice-driven improvement. Our findings suggest that agents should learn from data based on what each example is useful for, not just where it came from. This helps agents maintain stable learning while improving more efficiently through their own practice.