Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions

While large language models (LLMs) perform strongly on diverse tasks, their trustworthiness is limited by erratic behavior that is unfaithful to their internal knowledge. In particular, LLMs often fail on multiple-choice questions (MCQs) even if they encode correct answers in their hidden representations, revealing a misalignment between internal knowledge and output behavior. We investigate and mitigate this knowledge-prediction gap on MCQs through a three-step analysis of hidden representations. First, we quantify the prevalence and magnitude of the gap across models and datasets. Second, we provide a geometric interpretation by identifying distinct knowledge and prediction subspaces in the residual stream. Third, we introduce KAPPA, a lightweight inference-time intervention that aligns the two subspaces within the residual stream to reduce the knowledge-prediction gap. Our results provide a geometric and interpretable explanation of the knowledge-prediction gap in LLMs. Furthermore, KAPPA effectively reduces the gap across diverse MCQ benchmarks and models, and generalizes to free-form settings.

Large language models often get multiple-choice questions wrong even when they can answer the same question correctly in other formats. This suggests that a model may "know" the correct answer internally but fail to use it. We study why this happens and whether it can be fixed. By inspecting the internal signals of these models during multiple-choice question answering, we found that two different signals are present at the same time: one indicates which answer is correct, while another points to the answer the model is about to output. In models that answer poorly, these two signals diverge: the model's internal "knowledge" is not in line with its generation. To address this, we developed a lightweight method, KAPPA, which aligns these internal signals at test time, without retraining the model. Applying KAPPA makes models answer more in line with their internal knowledge, improving accuracy on reasoning, truthfulness, and bias benchmarks. More broadly, our findings suggest that many model errors come not from lack of knowledge, but from failing to use knowledge that is already present.