Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning

The family of linear recurrent neural networks has shown strong performance as recurrent memory units in partially observable reinforcement learning. We provide a theoretical justification for their empirical effectiveness by constructing and studying two linear filters: (i) the first exactly reproduces the pre–softmax logits of the belief vector in a hidden Markov model (HMM) under a deterministic transition matrix, thereby serving as a sufficient statistic for optimal policy learning, (ii) the second achieves vanishing state-decoding error under a nearly deterministic transition matrix, thus reducing state ambiguity to near zero. The results extend to action-controlled HMMs, where the corresponding linear filters become time-varying with action-dependent dynamics. We illustrate our main results through numerical experiments and further show that the constructed linear filter serves as a strong feature extractor in a small reinforcement learning game.

In reinforcement learning (RL), an agent learns to make decisions through interactions with its environment. When the environment is not fully observable, agents often need memory to make decisions based on a history of observations. Recent work has shown that surprisingly simple linear memory systems can perform well in such situations, but the reason for their success has remained unclear. We provide a theoretical explanation by developing two linear memory filters. We show that one preserves all the information needed for making optimal decisions when the hidden environment changes deterministically, while the other, called the Adaptive Logit Filter (ALF), can accurately infer the hidden environment from incomplete and noisy observations when these changes are nearly deterministic. Experiments further show that ALF enables effective decision making in a reinforcement learning task.