SVL: Goal-Conditioned Reinforcement Learning as Survival Learning

Standard approaches to goal-conditioned reinforcement learning (GCRL) that rely on temporal-difference learning can be unstable and sample-inefficient due to bootstrapping. While recent work has explored contrastive and supervised formulations to improve stability, we present a probabilistic alternative, called survival value learning (SVL), that reframes GCRL as a survival learning problem by modeling the distribution of time-to-goal from each state. This structured distributional Monte Carlo perspective yields a closed-form identity that expresses the goal-conditioned value function as a discounted sum of survival probabilities, enabling value estimation via a hazard model trained via maximum likelihood on both event and right-censored trajectories. We introduce three practical value estimators, including finite-horizon truncation and two binned infinite-horizon approximations to capture long-horizon objectives. Experiments on offline GCRL benchmarks show that SVL combined with hierarchical actors matches or surpasses strong hierarchical TD and Monte Carlo baselines, excelling on complex, long-horizon tasks.

Training AI agents to reliably reach goals, such as a robot navigating across a building, remains a major challenge. The most common methods adjust their predictions using prior predictions, a process that can cause small errors to compound, especially when goals require many sequential steps. We propose an alternative way to estimate the value of each situation in achieving a goal. Instead of asking "how good is this state for reaching the goal?", we model the full distribution of the time it would take the agent to reach the goal from each state. From this distribution, the value of any state can be computed directly, sidestepping the unstable feedback loops of standard methods. We train these time-to-goal predictions on past experience, including trajectories where the goal was never actually reached, and we provide three practical versions of the estimator that cover both short- and long-running tasks. On standard benchmarks for offline goal-reaching, our method, called Survival Value Learning, matches or beats existing approaches, with the largest gains on the longest and most complex tasks where current methods struggle most.