$\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

A central goal of large language model (LLM) research is to build agentic systems that can plan, act, and adapt through sustained interaction with dynamic environments. While recent LLM-based agents exhibit impressive contextual reasoning, their long-horizon decision-making remains fragile, often suffering from $\textit{objective drift}$, where goals and plans drift over extended interactions. We introduce $\texttt{Multi}^2$, a hierarchical multi-agent decision-making framework that explicitly decomposes agent behavior into complementary roles. A high-level agent ($\texttt{System 1}$) focuses on context-aware sub-goal generation using supervised fine-tuning (SFT), while a low-level agent ($\texttt{System 2}$) executes atomic actions through offline-to-online reinforcement learning (RL) in interactive environments. This separation enables stable long-horizon control, mitigates objective drift, and allows efficient adaptation. Across diverse interactive environments, $\texttt{Multi}^2$ consistently outperforms strong agentic baselines, demonstrating improved robustness and coordination in multi-turn interaction. Beyond performance, we introduce and release three hierarchical benchmark datasets, filling a long-standing gap in training and evaluating hierarchical decision-making for LLM-based agents.

Artificial intelligence (AI) agents powered by large language models (LLMs) can follow instructions and take actions in step-by-step tasks, but they often struggle when the task is long. When a task requires many steps, the agent may forget the original goal, repeat the same action, or make small mistakes that grow into bigger failures. We developed $\texttt{Multi}^2$ to make these agents more reliable by splitting the work between two specialized agents. One agent decides the next smaller goal, and the other agent carries out the specific action needed to reach that goal. The action-taking agent also learns from practice, so it can improve after trying tasks. We tested $\texttt{Multi}^2$ in several task environments where agents need to plan, act, and adjust over many steps. $\texttt{Multi}^2$ completed tasks more successfully, stayed closer to the original goal, and worked more efficiently than previous methods. This work matters because future AI agents will need to handle complex tasks without losing track of what they are supposed to do.