Debiased Model-based Representations for Sample-efficient Continuous Control

Model-based representations recently stand out as a promising framework that embeds latent dynamics information into the representations for downstream off-policy actor-critic learning. It implicitly combines the advantages of both model-free and model-based approaches while avoiding the training costs associated with model-based methods. Nevertheless, existing model-based representation methods can fail to capture sufficient information about relevant variables and can overfit to early experiences in the replay buffer. These incur biases in representation and actor-critic learning, leading to inferior performance. To address this, we propose Debiased model-based Representations for Q-learning, tagged DR.Q algorithm. DR.Q explicitly maximizes the mutual information between the representations of the current state-action pair and the next state besides minimizing their deviations, and samples transitions with faded prioritized experience replay. We evaluate DR.Q on numerous continuous control benchmarks with a single set of hyperparameters, and the results demonstrate that DR.Q can match or surpass recent strong baselines, sometimes outperforming them by a large margin. Our code is available at https://github.com/dmksjfl/DR.Q.

In reinforcement learning, AI agents learn by building internal "maps" (representations) of their environment. However, current methods often suffer from two types of "bias": they may fail to capture enough useful information about their surroundings, and they often "overfit" to their earliest, often clumsy, experiences. This results in inaccurate internal models and poor performance in complex tasks. To address this, we developed DR.Q (Debiased model-based Representations for Q-learning). DR.Q introduces a mathematical "mutual information" objective that forces the AI to capture as much relevant detail as possible about how its actions influence the future. Additionally, we created a "faded" memory system that prioritizes important new experiences while gradually letting go of old, potentially misleading ones. We tested DR.Q across 73 diverse robotic control tasks. Using a single set of settings, DR.Q matched or outperformed strong existing AI systems, sometimes by a large margin—especially in tasks involving complex humanoid movements. This research provides a more robust and efficient way for AI to learn from its experiences, marking a significant step toward building more capable autonomous robots and agents.