HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems

Multi-agent systems (MAS) built on large language models (LLMs) have demonstrated remarkable performance across diverse tasks. Existing approaches optimize communication topology, role assignment, or LLM routing in isolation, while treating each agent as a monolithic unit—failing to exploit internal LLM mixtures that can enhance individual role capabilities. We propose HieraMAS, a hierarchical agent collaboration framework with intra-node LLM mixtures and inter-node communication topology. HieraAgent introduces supernodes, where each functional role comprises multiple heterogeneous LLMs in a propose-synthesis structure. The optimization of HieraMAS poses unique credit assignment challenges, as final task performance heavily depends on LLM capabilities, potentially causing erroneous reinforcement of suboptimal configurations. We address this via a two-stage algorithm: (1) multi-level reward attribution providing fine-grained feedback at both node and system levels; and (2) graph classification treating topology selection as a holistic task rather than per-edge optimization. Experiments on reasoning and coding benchmarks demonstrate that HieraMAS significantly outperforms existing methods while achieving better cost-performance trade-offs.

Modern AI assistants often work in teams, with different "agents" handling different parts of a problem—much like a group of people collaborating on a project. Until now, most research has focused on how these AI agents talk to each other or who plays which role, while treating each agent as a single worker powered by one AI model. We introduce HieraMAS, a system that takes a different approach: each role on the team is itself a small panel of different AI models that propose ideas independently, after which one model synthesizes their best contributions. The challenge is figuring out which model deserves credit (or blame) when the team succeeds or fails. We solve this with a two-step training method that rewards each member based on both its individual contribution and the team's overall outcome, and that picks the best team structure as a whole rather than tweaking one connection at a time. On reasoning and programming tasks, HieraMAS produces noticeably better answers while using compute more efficiently, pointing toward AI teams that are both smarter and cheaper to run.