Which LLM Multi-Agent Protocol to Choose?

As large-scale multi-agent systems evolve, the communication protocol layer has become a critical, yet understudied, component affecting system performance and reliability. Despite a range of protocols, such as JSON-RPC, A2A, ANP, and ACP, protocol selection remains ad hoc. To address this, we introduce ProtocolBench, a benchmark designed to evaluate agent communication protocols across task utility, communication overhead, system performance, and resilience under failure. ProtocolBench uses a three-layer architecture with protocol adapters for fair com- parison, diverse scenarios (e.g., document aggregation, collaborative coding), and detailed telemetry. Our results show protocol choice can impact task completion time by up to 36%, communication overhead by 3.5 seconds, and resilience with statistically observable differences. We also propose ProtocolRouter, a learnable protocol routing system that dynamically selects protocols based on runtime con- ditions, improving performance by up to 18% compared to individual protocols. Our findings highlight that hybrid protocol deployments outperform homogeneous ones by approximately 6.6%, with negligible protocol translation overhead. We release ProtocolBench as an open-source framework to standardize protocol evaluation and improve multi-agent system reliability at scale.