Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoning

Multi-Agent Debate (MAD) has shown promise in improving reasoning and reducing hallucinations, yet it remains unclear how information exchange shapes individual reasoning behavior. Empirically, MAD exhibits paradoxical phenomena, including rising accuracy with increasing token entropy and marked differences between homogeneous and heterogeneous agent combinations. In this paper, we introduce a Bayesian uncertainty analysis framework for MAD, which decomposes answer-level predictive uncertainty into epistemic uncertainty and aleatoric uncertainty, corresponding to the potential gain and cost of debate. Across multiple agent configurations, we find that effective debate depends on achieving high epistemic gain under controlled aleatoric cost. Building on this insight, we design an uncertainty-guided multi-agent reinforcement learning algorithm that encourages lower aleatoric cost and more effective epistemic information utilization. Experiments show that our approach simultaneously enhances each agent's accuracy and promotes a more productive debate process, providing an operational Bayesian perspective for understanding and improving MAD.

Large language models can sometimes improve their answers by debating with other models, but it is not always clear why debate helps or why it sometimes fails. This paper studies multi-agent debate in math reasoning tasks from an uncertainty perspective. We show that useful debate depends on two competing factors: agents should provide new information to each other, but they should also avoid becoming unstable or confused when processing peer responses. We introduce a Bayesian-inspired framework that separates these two effects into epistemic uncertainty, which reflects useful disagreement and potential information gain, and aleatoric uncertainty, which reflects response instability during debate. Based on this analysis, we develop a reinforcement learning method that trains agents to better use helpful peer information while reducing unstable reasoning. Experiments show that this approach makes multi-agent debate more reliable and improves the reasoning performance of individual agents.