Hierarchical Decision Making with Structured Policies: A Principled Design via Inverse Optimization

Hierarchical decision-making frameworks are pivotal for addressing complex control tasks, enabling agents to decompose intricate problems into manageable subgoals. Despite their promise, existing hierarchical policies face critical limitations: (i) reinforcement learning (RL)-based methods struggle to guarantee strict constraint satisfaction, and (ii) optimal control (OC)-based approaches often rely on myopic and computationally prohibitive formulations. To reconcile these trade-offs, hierarchical RL-OC architectures have emerged as a promising paradigm. However, the formulation of the lower-level optimization within these frameworks remains underexplored, often relying on heuristic or myopic objectives. In this work, we propose a principled framework that systematically integrates upper-level goal abstraction with structured lower-level decision making. We adopt an inverse optimization approach to inform the structure of the lower-level problem from expert demonstrations, ensuring that the objective of the lower-level policy remains aligned with the overall long-term task goal. To validate the approach, our framework is evaluated on distinct decision making tasks: network-based resource allocation and continuous collision avoidance. Empirical results demonstrate that our method consistently outperforms strong baselines based on end-to-end RL, learning-augmented optimal control, and existing hierarchical RL approaches in both efficiency and decision quality.

Many real-time systems, such as robots, power grids, and transportation, must make fast decisions while obeying physical or operational constraints. Reinforcement learning can learn good long-term strategies, but it often behaves like a black box and may not strictly satisfy safety or capacity constraints. Optimal control can enforce such constraints, but solving a detailed long-horizon optimization problem at every decision step can be too slow for real-time use. Our work combines the strengths of both approaches: a learning model chooses a high-level goal, and a fast optimization module turns that goal into a feasible action. The key challenge is that this lower-level optimization problem is often hand-designed and can become short-sighted. We address this by using inverse optimization to learn the lower-level objective from a small set of expert demonstrations, rather than relying on heuristic design. This allows the controller to recover useful expert decision patterns while remaining computationally efficient and constraint-aware. Across vehicle rebalancing, supply chain inventory management, and robot navigation, our method makes better decisions than standard reinforcement learning and existing hierarchical designs, while being much faster than long-horizon optimal control. This matters because it provides a practical way to build AI decision systems that are safer, faster, and better aligned with expert behavior.