Learning to Perceive the World Through Control: Empowerment-Based Representation Learning

In many practical reinforcement learning (RL) environments, observations are far higher-dimensional than the variables that matter for control. In this work, we ask: can we learn representations that capture only control-relevant features of the environment? We study this question through the \emph{empowerment} objective, which maximizes an agent’s influence over the environment and is widely used for unsupervised skill learning. We show that empowerment agents induce two distinct representations --- forward and backward --- that capture complementary aspects of the state, and both of which are invariant to control-irrelevant features. Thus, empowerment maximization leads agents to learn an implicit, \emph{control-centric} model of the world. Our analysis highlights the importance of learning representations through interaction rather than from passive datasets: interaction aimed at maximizing control is essential for learning useful invariance properties, a perspective that aligns closely with the causal learning literature.

In many real-world reinforcement learning environments, agents observe far more information than is actually needed to make good decisions. For example, a robot navigating a maze should pay attention to its position and nearby obstacles, which determine what it can control, while ignoring irrelevant details such as a randomly changing light color on the wall. In this work, we study whether agents can automatically learn to focus only on the parts of the environment that matter for control, without relying on manually designed reward functions. We show that agents trained to maximize empowerment — a measure of how much influence they have over the future — naturally learn compact representations that ignore irrelevant details while preserving information important for decision-making. Our results suggest that active interaction with the environment, rather than passive observation alone, is crucial for learning useful and robust representations, drawing a strong connection between reinforcement learning and ideas from causal learning.