ProtocolBench: Which LLM MultiAgent Protocol to Choose?

As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability. Despite the existence of diverse protocols (A2A, ACP, ANP, Agora, etc.), the selection of them is often intuition-driven and lacks standardized guidance. We introduce ProtocolBench, a benchmark that systematically compares agent protocols along four measurable axes: task success, end-to-end latency, message or byte overhead, and robustness under failures. On ProtocolBench, the choice of protocol significantly influences system behavior. In the Streaming Queue scenario, overall completion time varies by up to 36.5\% across protocols, and mean end-to-end latency differs by 3.48 s. Under Fail-Storm Recovery, resilience also differs consistently across protocols. Beyond evaluation, we present ProtocolRouter, a lightweight constraint-aware protocol router that selects per-scenario (or per-module) protocols from requirement and runtime signals. ProtocolRouter reduces Fail-Storm recovery time by up to 18.1\% versus the best single-protocol baseline and achieves scenario-specific gains such as higher success in GAIA, while exposing trade-offs across other metrics. We also release ProtocolRouterBench to standardize constrained protocol-selection evaluation and improve reliability at scale.

Many AI applications now use multiple AI agents that talk to one another to answer questions, process documents, or coordinate services. The way these agents communicate can strongly affect how fast, reliable, and safe the system is, but developers often choose communication methods by intuition. This paper introduces ProtocolBench, a testing framework that compares several communication protocols for multi-agent AI systems across realistic tasks such as document question answering, high-throughput request processing, recovery from agent failures, and privacy-sensitive medical dialogue. We show that no single protocol is best for every situation: some are faster, some recover better from failures, and some provide stronger privacy protections. We also introduce ProtocolRouter, a lightweight system that chooses protocols according to the needs of each task or module. Our goal is to help researchers and practitioners make communication choices more systematic and build more reliable multi-agent AI systems.