Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation

Continuous-time reinforcement learning (CTRL) provides a natural framework for sequential decision-making in dynamic environments where interactions evolve continuously over time. While CTRL has shown growing empirical success, its ability to adapt to varying levels of problem difficulty remains poorly understood. In this work, we investigate the instance-dependent behavior of CTRL and introduce a simple, model-based algorithm built on maximum likelihood estimation (MLE) with a general function approximator. Unlike existing approaches that estimate system dynamics directly, our method estimates the state marginal density to guide learning. We establish instance-dependent performance guarantees by deriving a regret bound that scales with the total reward variance and measurement resolution. Notably, the regret becomes independent of the specific measurement strategy when the observation frequency adapts appropriately to the problem’s complexity. To further improve performance, our algorithm incorporates a randomized measurement schedule that enhances sample efficiency without increasing measurement cost. These results highlight a new direction for designing CTRL algorithms that automatically adjust their learning behavior based on the underlying difficulty of the environment.

Many important decision-making problems, such as controlling robots, managing financial systems, or planning medical interventions, unfold continuously over time rather than in neat, fixed steps. Learning in these settings is challenging because an algorithm must decide both what action to take and when to observe the system. Existing methods often use fixed observation schedules or try to learn the full underlying dynamics, which can waste measurements or become unreliable when the system is noisy. We propose CT-MLE, a new method for continuous-time reinforcement learning that learns from observed states by estimating where the system is likely to be in the future, instead of directly reconstructing every detail of the dynamics. The method also adds a small amount of randomness to its observation times, helping it estimate long-term rewards without greatly increasing measurement cost. Our theory shows that CT-MLE adapts to the difficulty of each environment: easier or less uncertain tasks require less learning effort, while harder, noisier tasks call for more. This gives researchers a principled way to choose observation schedules that use data more efficiently. In simulations of classic control tasks, the method performed especially well in challenging noisy environments.