Adaptive Reinforcement Learning for Unobservable Random Delays

In standard reinforcement learning (RL) settings, the interaction between the agent and the environment is typically modeled as a Markov decision process (MDP), which assumes that the agent observes the system state instantaneously, selects an action without delay, and executes it immediately. In real-world dynamic environments, such as cyber-physical systems, this assumption often breaks down due to delays in the interaction between the agent and the system. These delays can vary stochastically over time and are typically unobservable when deciding on an action. Existing methods deal with this uncertainty conservatively by assuming a known fixed upper bound on the delay, even if the delay is often much lower. In this work, we introduce the interaction layer, a general framework that enables agents to adaptively handle unobservable and time-varying delays. Specifically, the agent generates a matrix of possible future actions, anticipating a horizon of potential delays, to handle both unpredictable delays and lost action packets sent over networks. Building on this framework, we develop a model-based algorithm, Actor-Critic with Delay Adaptation (ACDA), which dynamically adjusts to delay patterns. Our method significantly outperforms state-of-the-art approaches across a wide range of locomotion benchmark environments, including real-world measured delays.

A key problem when using AI agents to control real-world systems, like robots, is the delay between the agent and the system it controls. Reinforcement learning, the machine learning technique used to train these agents, generally assumes that interactions occur instantaneously. In reality, these delays vary randomly over time, causing agents that do not account for the delay to fail. We present a framework, the first of its kind, that allows agents to adapt on the fly to these delays without even knowing what the delay is. Instead of sending a single command, the agent sends a package of many commands, each command tailored to a specific delay. Our results show that an agent trained with this capability significantly outperforms previous methods for addressing delays. This framework is a realistic interpretation of the delay problem, without any assumptions on knowing future delays or perfect time synchronization. It allows for the physical separation of agents and the systems they control, enabling control of systems that cannot carry the necessary hardware.