Covariance Volume Maximization for Embodied Latent Exploration in Deep Reinforcement Learning

Efficient exploration remains a key challenge in deep reinforcement learning, especially for embodied agents operating in realistic environments with high-dimensional observations and complex dynamics. Recent latent exploration methods define bonuses in a learned latent space, but often struggle in these settings where (i) representations can be noisy or policy-dependent, and (ii) common strategies such as randomized latent objectives or fixed directional spanning are brittle and fail to improve global coverage. We propose Covariance Volume Maximization (CVM), a coverage-driven latent exploration framework with two key components. First, we learn a behavioral state encoder using a policy-mixture objective to reduce representation drift under rapidly changing exploration policies, yielding stable and behaviorally meaningful latent displacements. Second, CVM rewards each transition by its exact increase in the log-determinant of the covariance of recent latent displacements, explicitly expanding the explored region and prioritizing under-covered directions. This objective coincides with the classical D-optimal design criterion, providing an information-efficiency justification. Extensive experiments on embodied navigation and manipulation tasks demonstrate that CVM substantially improves exploration efficiency and robustness, and scales effectively to different environments.

Teaching robots and other AI agents to explore efficiently is difficult, especially in realistic environments where they must learn from complex camera views and physical interactions. Many existing methods either react too strongly to noisy visual changes or keep trying the same easy kinds of moves, which limits how much of the environment they truly discover. In this paper, we propose a new exploration method called Covariance Volume Maximization (CVM). The main idea is simple: help the agent build a more stable internal understanding of how its actions change the world, and then encourage it to try movements that open up less-explored possibilities instead of repeating familiar ones. This leads to broader and more useful exploration. We test the method on a range of navigation and manipulation tasks, including realistic embodied settings, and find that it helps agents explore more effectively and behave more robustly across different environments.