HodgeFlow Policy Search by Topologically Dissecting Temporal-Difference Signals in Non-Markovian Environments

Non-Markovian dynamics are commonly found in real-world environments due to long-range dependencies, partial observability, and memory effects. The Bellman equation that is the central pillar of Reinforcement learning (RL) becomes only approximately valid under Non-Markovian. Existing work often focus on practical algorithm designs and offer limited theoretical treatment to address key questions, such as what dynamics are indeed capturable by the Bellman framework and how to inspire new algorithm classes with optimal approximations. In this paper, we present a novel topological viewpoint on temporal-difference (TD) based RL. We show that TD errors can be viewed as 1-cochain in the topological space of state transitions, while Markov dynamics are then interpreted as topological integrability. This novel view enables us to obtain a Hodge-type decomposition of TD errors into an integrable component and a topological residual, through a Bellman–de Rham projection. We further propose HodgeFlow Policy Search (HFPS) by fitting a {potential network} to minimize the non-integrable projection residual in RL, achieving stability/sensitivity guarantees. In numerical evaluations, HFPS is shown to significantly improve RL performance under non-Markovian.

Reinforcement learning is a way for AI systems to learn decisions through feedback from their environment. Most standard methods assume that the current situation contains all the information needed to decide what should happen next. In many real-world problems, this assumption is not fully true: the best action may depend on earlier events, hidden factors, delayed effects, or past interactions that are not visible at the current moment. When this happens, ordinary learning signals can become unreliable and may lead the system to unstable or poor behavior. This paper proposes a new way to separate the reliable part of the learning signal from the part caused by hidden history or memory effects. Based on this idea, we design a method called HodgeFlow Policy Search, which trains the learning system using only the more consistent part of the signal while also measuring how much inconsistency remains. Experiments on several decision-making tasks show that this approach can make learning more stable and robust when the environment depends on past events.