Frictional Q-Learning

Off-policy reinforcement learning suffers from extrapolation errors when a learned policy selects actions that are weakly supported in the replay buffer. In this study, we address this issue by drawing an analogy to static friction. From this perspective, the replay buffer is represented as a smooth, low-dimensional action manifold, where the support directions correspond to the tangential component, while the normal component captures the dominant first-order extrapolation error. This decomposition reveals an intrinsic anisotropy in value sensitivity that naturally induces a stability condition analogous to a friction threshold. To mitigate deviations toward unsupported actions, we propose Frictional Q-Learning, an off-policy algorithm that encodes supported actions as tangent directions using a contrastive variational autoencoder. We further show that an orthonormal basis of the orthogonal complement corresponds to normal components under mild local isometry assumptions. Extensive empirical results on standard continuous-control benchmarks consistently demonstrate robust and stable performance compared with competitive baselines.

Reinforcement learning teaches an AI agent to make decisions by learning from past experiences. A common approach lets the agent reuse old experiences stored in a memory bank — technically called a replay buffer. However, problems can arise when the agent chooses actions that are very different from anything it has actually seen before. In those cases, the agent has to make uncertain guesses, which can lead to unstable learning and poor decisions. We address this problem using an analogy to static friction. Just as friction prevents an object from slipping unless enough force is applied, a learning algorithm should resist moving toward actions that are not well supported by past data. We view the recorded actions as forming a smooth surface: safe directions stay along this surface, while risky directions move away from it. Based on this idea, we propose Frictional Q-Learning, a method that identifies well-supported action directions and discourages unsupported deviations. Experiments on standard control tasks show that this approach learns more stably and performs robustly compared with existing methods.