HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems

Multi-agent systems (MAS) built on large language models (LLMs) have demonstrated remarkable performance across diverse tasks. Existing approaches optimize communication topology, role assignment, or LLM routing in isolation, while treating each agent as a monolithic unit—failing to exploit internal LLM mixtures that can enhance individual role capabilities. We propose HieraMAS, a hierarchical agent collaboration framework with intra-node LLM mixtures and inter-node communication topology. HieraAgent introduces supernodes, where each functional role comprises multiple heterogeneous LLMs in a propose-synthesis structure. The optimization of HieraMAS poses unique credit assignment challenges, as final task performance heavily depends on LLM capabilities, potentially causing erroneous reinforcement of suboptimal configurations. We address this via a two-stage algorithm: (1) multi-level reward attribution providing fine-grained feedback at both node and system levels; and (2) graph classification treating topology selection as a holistic task rather than per-edge optimization. Experiments on reasoning and coding benchmarks demonstrate that HieraMAS significantly outperforms existing methods while achieving better cost-performance trade-offs.