Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets

Cross-domain offline reinforcement learning (RL) aims to learn a policy in the target domain with a limited target domain dataset and a source domain dataset that exhibits a dynamics shift. Training directly on the original source dataset typically leads to performance collapse. Recent studies perform data filtering from the perspective of dynamics alignment or value alignment to enable efficient policy transfer. However, these studies are typically validated on single-domain or single-behavior-policy source datasets. In this work, we explore a more general heterogeneous cross-domain offline RL setting, where the source datasets may be collected from multiple source domains by diverse behavior policies. We first uncover a critical yet overlooked issue in this setting: value misassignment. Empirically and theoretically, we demonstrate that value misassignment can undermine value alignment, mislead data filtering toward selecting suboptimal samples, and loosen the suboptimality gap, thereby degrading the agent’s performance. To address this issue, we propose V2A, which integrates dynamics alignment, value alignment, and value assignment. V2A first employs temporally-consistent modality representation learning to extract dynamics modalities from the source dataset, followed by modality-aware advantage learning to rectify value alignment. Finally, it adopts a data filtering paradigm to selectively share source data for policy learning. Empirical results show that V2A significantly outperforms strong baseline methods under general heterogeneous cross-domain offline RL settings.

Robots and other AI agents often need to learn good actions in a new environment, but collecting fresh trial-and-error data there can be costly or risky. A natural alternative is to reuse old data from related environments, such as robots with different body shapes or simulations with different physics. However, simply mixing all old data can hurt learning, because some data may follow different physical rules and some may show poor behavior. We identify a new problem in this mixed-data setting: the learning system can assign the wrong quality score to an action when it ignores which environment produced that action. We propose V2A, a method that first discovers different environment types hidden in the data, then evaluates each action within the right type, and finally selects old data that are both relevant to the target environment and useful for learning. This lets the agent benefit from prior experience while avoiding misleading examples. In simulated control tasks, V2A consistently improves over strong existing methods. Our work helps make learning from existing datasets more reliable when the data comes from many related but different sources, a common situation in robotics.