D-FUSEr: Diverse Failure, Unified Success via Error-Distribution Shaping in LLM Reasoning

Test-time scaling methods such as majority vote aggregation and iterative refinement (e.g., self-reflection or multi-agent inference) improve reasoning performance by leveraging multiple solution samples. However, their efficacy depends not only on raw performance, but critically on the distribution of errors across samples. When errors concentrate, (a) aggregation accuracy degrades, as the majority vote may select a shared mistake, and (b) confidence in common mistakes may suppress exploration in iterative refinement. We argue that improving correctness alone is not sufficient to mitigate these issues; to this end, we propose to explicitly shape error distributions to improve aggregation. First, we introduce a theoretically grounded \textbf{diverse failure reward} that incentivizes calibrated disagreement within model errors. We prove that this reward directly optimizes majority-vote accuracy: policies achieving higher reward attain higher expected majority-vote performance, and vice versa. We further show that this theoretical property generalizes to iterative refinement. Second, we introduce \textbf{anti-votes}, in which the model predicts the most common mistake alongside its solution, allowing probability mass on dominant errors to be explicitly reweighted. We identify conditions under which anti-votes are guaranteed to improve majority-vote accuracy. Empirically, across three model families of varying sizes and four benchmarks, we show that both approaches substantially improve majority vote and iterative refinement performance without degrading single-sample accuracy.

When AI systems tackle hard reasoning problems, one effective trick is to have them generate several possible answers and then pick the best one: for example, by going with whichever answer comes up most often, or by letting the model review and revise its own work. This works well, but it has a hidden weakness: if the AI tends to make the same mistake repeatedly, these methods backfire. The wrong answer can win the majority vote, and the AI's confidence in that shared error can stop it from exploring better alternatives. Our key insight is that simply training an AI to be more accurate isn't enough. What also matters is how its mistakes are spread out. If the model is going to be wrong, it's far better for it to be wrong in many different ways than to keep making one dominant error. We introduce two methods built around this idea. The first is a new training reward that encourages the model's errors to be varied rather than clustered together. We show mathematically that improving on this reward directly translates into better answers when multiple samples are combined. The second method has the model flag what it thinks the most common mistake is likely to be, alongside its actual answer, so that tempting-but-wrong answers can be down-weighted before a final choice is made. Testing these approaches across several different AI models and datasets, we find that both methods reliably improve the quality of the final answer derived from multiple samples, all without making the model any worse at answering on a single try.