Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution

Large Language Models have achieved strong performance on reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step reasoning trace might fail remains difficult. Confidence estimation offers a natural diagnostic signal, yet existing methods are restricted to final answers or require internal model access. We introduce Stepwise Confidence Attribution (SCA), a framework for closed-source LLMs that assigns step-level confidence based only on generated reasoning traces. SCA applies the Information Bottleneck principle: steps aligning with consensus structures across correct solutions receive high confidence, while deviations are flagged as potentially erroneous. We propose two complementary methods: (1) NIBS, a non-parametric IB approach measuring consistency without graph structures, and (2) GIBS, a graph-based IB model that learns subgraphs through a differentiable mask to capture logical variability. Extensive experiments on mathematical reasoning and multi-hop question answering show that SCA reliably identifies low-confidence steps strongly correlated with reasoning errors. Moreover, using step-level confidence to guide self-correction improves the correction success rate by up to 13.5\% over answer-level feedback.

Large language models often solve problems by generating step-by-step reasoning, but when they make a mistake, it is hard to know which step went wrong. Existing methods usually check only the final answer, require access to the model’s internal information, or rely on costly human annotations. We propose Stepwise Confidence Attribution, a method that scores the reliability of each reasoning step using only the model’s generated solutions. The key idea is that correct solutions may be written differently, but they often share important intermediate facts or logical patterns. Our method finds these shared patterns and flags steps that deviate from them as less reliable. Experiments on math and multi-hop question-answering tasks show that our method can locate likely reasoning errors and help models correct their own mistakes more effectively.