OMAC: A Holistic Optimization Framework for LLM-Based Multi-Agent Collaboration

Agents powered by advanced large language models (LLMs) have demonstrated impressive capabilities across diverse complex applications. Recently, Multi-Agent Systems (MAS), wherein multiple agents collaborate and communicate with each other, have exhibited enhanced capabilities in complex tasks, such as high-quality code generation and arithmetic reasoning. However, the development of such systems often relies on handcrafted methods, and the literature on systematic design and optimization of LLM-based MAS remains limited. In this work, we introduce OMAC, a general framework designed for holistic optimization of LLM-based MAS. Specifically, we identify five key optimization dimensions for MAS, encompassing both agent functionality and collaboration structure. Building upon these dimensions, we first propose a general algorithm, utilizing two actors termed the Semantic Initializer and the Contrastive Comparator, to optimize any single dimension. Then, we present an algorithm for joint optimization across multiple dimensions. Extensive experiments demonstrate the superior performance of OMAC on diverse tasks against recent approaches. Codes are available at: https://github.com/xiwenchao/OMAC.

Large language models are increasingly used as "agents" that can plan, reason, write code, or solve problems, and many systems now combine several such agents to work together. However, these multi-agent systems are often built by hand: researchers decide which agents to include, what each agent should do, and how they should communicate, usually through trial and error. This makes it hard to know whether the system is well designed or whether a better collaboration pattern exists. We introduce OMAC, a framework that automatically improves both the agents themselves and the way they collaborate. OMAC looks at five parts of a multi-agent system: improving existing agents, adding new agents, choosing which agents should participate, deciding which agents should act at each step, and controlling which agents share information. It uses large language models to propose candidate designs, tests them on training tasks, compares stronger and weaker designs, and then refines the system based on what worked better. Across code generation, general reasoning, and math reasoning tasks, OMAC consistently improves performance over recent multi-agent optimization methods. This work matters because it provides a more systematic way to build reliable, efficient teams of AI agents for complex tasks.