Geometric Coherence Learning for Structuring Value Functions in Plain MDPs

Geometric properties can be leveraged to stabilize and speed reinforcement learning. Existing examples include encoding symmetry structure, geometry-aware data augmentation, and enforcing structural restrictions. In this paper, we take a novel view of RL through the lens of order theory and recast value function estimates into learning a desired poset (partially ordered set). We propose \emph{GCR-RL} (Geometric Coherence Regularized Reinforcement Learning) that computes a sequence of super-poset refinements -- by refining posets in previous steps and learning additional order relationships from temporal difference signals -- thus ensuring geometric coherence across the sequence of posets underpinning the learned value functions. Two novel algorithms by Q-learning and by actor--critic are developed to efficiently realize these super-poset refinements. Their theoretical properties and convergence rates are analyzed. We empirically evaluate GCR-RL in a range of tasks and demonstrate significant improvements in sample efficiency and stable performance over strong baselines.

Reinforcement learning is a way for artificial intelligence systems to learn by trying actions, receiving feedback, and gradually improving their decisions. Although this approach has achieved strong results in games, control, and planning, it can still be unstable and may require many trials before learning a reliable strategy. This paper proposes a new method, GCR-RL, to make this learning process more stable and efficient. The main idea is that many decision-making tasks contain hidden structure. Some situations are essentially similar, while others have a natural “better than” relationship, such as being closer to a goal or avoiding unsafe choices. Instead of treating every experience as unrelated, GCR-RL learns these relationships and uses them to guide value estimates during training. This helps reduce inconsistent updates and prevents the learning process from cycling between conflicting estimates. Experiments show that this approach can help reinforcement learning agents learn faster and perform more stably across different tasks.