A Diagnostic Study of Multi-Agent LLMs for Real-World Debates

Multi-agent LLM debates are increasingly used in domains such as policy, politics, and city planning, where ground truth is often unavailable. Yet existing evaluations rely heavily on outcome-based proxies such as consensus, majority vote, or LLM-as-judge scores, which can miss failures like sycophancy, domination, and premature convergence. We introduce a diagnostic framework that evaluates both debate outcomes and the deliberative process using interpretable metrics for engagement, responsiveness, influence asymmetry, balance, stability, and agent utility. Across real-world debate settings and validation benchmarks, our process-level diagnostics align more closely with human judgments and reveal interaction failures that standard outcome-only measures overlook. These results show that reliable evaluation of multi-agent debates requires measuring not only what answer agents reach, but how they reach it.

Large language models are increasingly used in groups of agents that debate or discuss complex questions, such as policy choices, city planning decisions, and political simulations. However, these debates are often evaluated only by their final answer, by whether the agents agree, or by another language model’s judgment. These signals can be misleading because a group may reach agreement through copying, pressure from one dominant agent, or premature consensus rather than meaningful discussion. We propose a diagnostic framework that evaluates not only what answer a group reaches, but how the agents reached it. Our metrics measure whether agents engage with the discussion, respond to one another’s arguments, avoid domination by a single voice, and maintain a healthy balance between disagreement and convergence. We test these diagnostics on both objective question-answering tasks and subjective debate settings such as city planning and politics. Our results show that process-level signals better capture debate quality than simple outcome-based measures, especially when there is no single correct answer. This work can help researchers and practitioners audit multi-agent AI systems before relying on their conclusions in real-world decision-making settings.