The Cost of Commitment in Option-Based Hierarchical RL

Empirically, option-based hierarchical reinforcement (HRL) learning often produces longer and more diverse options when a deliberation cost is charged at option boundaries. However, when options are executed for many steps under an approximate dynamics model, small model errors compound along the option, degrading the quality of the resulting plan. In this work, we introduce the commitment loss to formalize the tradeoff between deliberation cost and model error as a function of option duration. We characterize how optimal termination probabilities vary with this tradeoff under two model-error mechanisms. First, the model is learned from finite data via maximum-likelihood estimation, producing statistical error that interacts with option duration. Second, we consider an input-driven setting where an exogenous input is only observed at option boundaries and evolves unobserved between them, creating a drift-induced mismatch between planned and realized dynamics. In both cases, we solve for the optimal termination behavior as a function of deliberation cost and the error scale, clarifying the behavior of some popular HRL algorithms that approach the deliberation cost as a heuristic.

When an AI agent follows a plan for too long, discrepancies can emerge between what it expected to happen and what actually happens. However, replanning too often can also be costly, since each new decision may require additional computation, information gathering, or reasoning. In this work, we study this tradeoff and provide theoretical guidelines for choosing appropriate plan lengths depending on the environment and uncertainty level. Our results help explain the behavior of modern hierarchical AI systems and may help design more robust agents that know when to re-plan.