Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access

Asymmetric reinforcement learning leverages privileged information available during training to improve learning under partial observability. Existing asymmetric actor-critic methods typically assume access to the full environment state to condition the critic during training, which is often unrealistic in practice. We introduce the informed asymmetric actor-critic framework that allows the critic to be conditioned on arbitrary state-dependent privileged signals, and show that any such signal yields unbiased policy gradient estimates. This substantially expands the set of admissible privileged information and raises the problem of selecting the most informative signals for learning. To this end, we propose two novel informativeness criteria: a dependence-based test that can be applied prior to training, and a test based on improvements in value prediction that can be applied post hoc. Experiments on partially observable benchmarks and synthetic environments demonstrate that carefully selected privileged signals can match or outperform full-state asymmetric baselines while relying on strictly less state information.

An intelligent agent must learn to make decisions from past experience while only observing part of its environment. In some settings, learning is made easier by giving the agent access to full state information during training, even though this is not available at test time. However, this assumption is often unrealistic, and in practice only limited auxiliary information may be available during training. In this work, we study settings where arbitrary state-dependent privileged information is available during training, even if it does not correspond to the full state of the environment. Indeed, as humans, agents could leverage smaller pieces of helpful hints during learning, such as information from additional sensors, simulator variables, or external data sources. We introduce a training method that can use these hints in a safe way, without changing what the agent ultimately learns to do. We also propose two criteria to measure which hints are most useful for learning. Our results show that agents trained with carefully selected auxiliary signals can perform as well as, or even better than, agents that rely on full information about the environment. This suggests that effective learning is possible even when training-time information is incomplete, making it easier to build agents for real-world problems.