Intentional Updates for Streaming Reinforcement Learning

In gradient-based learning, a step size chosen in parameter units does not produce a predictable per-step change in function output. This often leads to instability in the streaming setting (i.e., batch size$=1$), where stochasticity is not averaged out and update magnitudes can momentarily become arbitrarily big or small. Instead, we propose intentional updates: first specify the intended outcome of an update and then solve for the step size that approximately achieves it. This strategy has precedent in online supervised linear regression via Normalized Least Mean Squares (NLMS) algorithm, which selects a step size to yield a specified change in the function output proportional to the current error. We extend this principle to streaming deep reinforcement learning by defining appropriate intended outcomes: Intentional TD aims for a fixed fractional reduction of the TD error, and Intentional Policy Gradient aims for a bounded per-step change in the policy, limiting local KL divergence. We propose practical algorithms combining eligibility traces and diagonal scaling. Empirically, these methods yield state-of-the-art streaming performance, frequently performing on par with batch and replay-buffer approaches.

Machine learning systems often learn by making many small adjustments as new data arrives. Usually, these adjustments are controlled by a setting called a step size, which determines how much the system’s internal numbers change. But changing these internal numbers by a fixed amount does not guarantee that the system’s actual predictions or behavior will change by a predictable amount. This can make learning unstable, especially when the system learns continuously from one example at a time. This paper proposes a different approach: instead of choosing how much to change the internal numbers, first choose the intended effect of the update, such as how much a prediction or action choice should change, and then compute the update needed to achieve that effect. We apply this idea to reinforcement learning, where an agent learns by interacting with an environment and receiving rewards. The resulting methods make learning more stable in fully streaming settings, without relying on stored past experience or large batches of data. In experiments across several reinforcement learning benchmarks, these methods perform strongly and often approach the performance of computationally-expensive methods that use batches and replay.