Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learning

Real-world reinforcement learning often faces environment drift, but most existing methods rely on static entropy coefficients/target entropy, causing over-exploration during stable periods and under-exploration after drift (thus slow recovery), and leaving unanswered the principled question of how exploration intensity should scale with drift magnitude. We show that, under standard assumptions, entropy scheduling in non-stationary maximum-entropy RL can be cast as the dynamic-regret trade-off between tracking a drifting comparator and stabilizing updates, yielding a square-root scaling rule for the entropy weight in terms of a (possibly conservative) online non-stationarity proxy. Building on this, we propose AES (Adaptive Entropy Scheduling), which adaptively adjusts the entropy coefficient/temperature online using observable drift proxies during training, requiring almost no structural changes and incurring minimal overhead. Across 4 algorithm variants, 12 tasks, and 4 drift modes, AES significantly reduces the fraction of performance degradation caused by drift and accelerates recovery after abrupt changes.

Reinforcement learning systems learn by trying different actions and retaining effective strategies, but many training algorithms assume that the world does not change. In real applications, the best action may change over time. A common way to help an agent explore is to add controlled randomness, but this randomness is often fixed: when the environment is stable, it can be wasteful; when the environment changes suddenly, it may be too weak for fast adaptation. We study the question of how exploration intensity should change as the environment drifts. Our method uses signals that are already produced during training, such as increases in prediction error, to adjust the agent’s exploration “knob” online. When the environment appears stable, it reduces unnecessary randomness; when the environment changes, it increases exploration so that the agent can recover more quickly. We tested this idea across multiple reinforcement learning algorithms and simulated control tasks. The results show that our method improves the agent’s robustness to environmental changes and shortens recovery time after sudden shifts. This suggests that exploration should be treated as an adaptive response to environmental change.