A Two-Tier Perspective on Inference-Time Parallelism in Multi-Agent LLM Systems

Large language model (LLM)-driven multi-agent systems typically require multiple model invocations and complex coordination during inference, and their execution strategies directly affect system accuracy, latency, and computational cost. Parallel execution provides a means to improve inference-time efficiency. From the perspective of inference-time execution, this paper models parallelism in multi-agent systems as two distinct levels of decision processes: Replica Parallelism, which explores multiple complete solution paths at the task level, and Structural Parallelism, which enables concurrent execution within a single solution path through task decomposition. However, the roles of different forms of parallelism and their interrelationships still lack systematic study in terms of unified organization and coordination. We therefore propose TIPEX, a controllable execution framework that unifies these two levels of parallelism and coordinates their roles within the inference process under a unified execution semantics while supporting systematic combinations and analyses of different parallel strategies and parameter configurations. Systematic experiments on the GAIA benchmark demonstrate that inference-time parallelism can significantly improve accuracy and reduce end-to-end latency at the cost of increased token consumption. Further analysis shows that Replica and Structural Parallelism exhibit complementary effects across task complexities, with tasks of intermediate difficulty benefiting most from their coordination, while overly aggressive parallel strategies do not necessarily yield better performance.

Many AI systems now solve complex tasks by using several language-model agents that work together, search for information, use tools, and check intermediate results. These systems can be powerful, but they often run slowly because many steps are performed one after another, and errors made early may affect later steps. This paper studies how such AI systems can work in parallel during the answering process. We separate parallel execution into two types: trying several complete solution attempts at the same time, and allowing different parts of one solution attempt to run at the same time when they do not depend on each other. We introduce TIPEX, a framework that organizes and combines these two forms of parallel work. Experiments on challenging tasks show that parallel execution can improve answer accuracy and reduce response time, but it also uses more computation. The results also show that more parallelism is not always better: balanced parallel execution works best for tasks of moderate difficulty.