Learn to change the world: Multi-level reinforcement learning with model-changing actions

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.

Traditional RL formulations generally focus on learning an effective policy under a fixed environment, while overlooking the possibility that an agent may actively configure or improve aspects of the environment itself. However, many practical systems allow partial control over underlying environmental dynamics. For example, a robotic agent that tries to cross a fast-flowing river may choose among different environment-modifying actions, for instance, placing stepping stones in the river, to make the environment easier for crossing. Although such model-changing actions may be subject to physical inaccuracies or implementation errors, they can nevertheless substantially improve achievable performance. To address this limitation, this paper introduces the Multi-layer Configurable Time-varying Markov Decision Process (MCTV-MDP) framework together with a corresponding learning algorithm, Bi-level Value Iteration, for finding the jointly optimal policies. In this framework, the RL agent can actively modify lower-level environment dynamics through higher-level model-changing actions in order to transition toward more favorable environments. We derive algorithms for the optimal model-changing policies using convex optimization and multi-level value iteration techniques.