Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting

Most value-based and actor-critic reinforcement learning methods rely on Bellman-style recursions, yet these recursions collapse under non-exponential discounting common in human preferences and survival processes. We show the breakdown is structural: exponential discounting sits at a fragile intersection of multiplicativity and time homogeneity, and violating either property breaks standard dynamic programming. To overcome this, we propose Pontryagin-Guided Direct Policy Optimization (PG-DPO), a variational framework that abandons recursion and couples the Pontryagin Maximum Principle with Monte Carlo rollouts via an Adjoint-MC projection enforcing pointwise Hamiltonian maximization. Across multidimensional hyperbolic and survival-discount benchmarks, PG-DPO improves accuracy and stability where equation-driven solvers and critic-based baselines diverge.

Many AI systems learn to make a sequence of decisions by breaking a long-term problem into smaller future subproblems. This works well under a common assumption: future rewards become less important at a constant exponential rate. However, many real situations do not follow this pattern. For example, people often give much more weight to immediate rewards, and survival or failure risks can change over time. This paper studies what happens when this standard assumption is removed. We show that the usual recursive decision-making principle can fail in a structural way, not just because of poor implementation. To address this issue, we propose a new method, PG-DPO, that avoids relying on this broken recursion. Instead, it uses simulated future paths to estimate how valuable the current state is, and then directly chooses actions that are locally consistent with this estimate. Across several benchmark problems with non-standard discounting, our method produces more accurate and stable decisions than existing reinforcement learning and equation-based solvers. This suggests a practical route toward decision-making algorithms that work better when future preferences or risks evolve in realistic, non-exponential ways.