Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning

Scientific problem solving poses unique challenges for LLMs, requiring both deep domain knowledge and the ability to apply such knowledge through complex reasoning. While automated scientific reasoners hold great promise for assisting human scientists, there is currently no widely adopted holistic benchmark for evaluating scientific reasoning, and few approaches systematically disentangle the distinct roles of knowledge and reasoning in these tasks. To address these gaps, we introduce SciReas, a diverse suite of existing benchmarks for scientific reasoning tasks, and SciReas-Pro, a selective subset that requires more complex reasoning. Our holistic evaluation surfaces insights about scientific reasoning performance that remain hidden when relying on individual benchmarks alone. We then propose KRUX, a probing framework for studying the distinct roles of reasoning and knowledge in scientific tasks. Combining the two, we conduct an in-depth analysis that yields several key findings: (1) Retrieving task-relevant knowledge from model parameters is a critical bottleneck for LLMs in scientific reasoning; (2) Reasoning models consistently benefit from external knowledge added in-context on top of the reasoning enhancement; (3) Enhancing verbalized reasoning improves LLMs' ability to surface task-relevant knowledge.

AI language models are increasingly used to answer scientific questions, but it is often unclear why they fail: do they lack the needed scientific knowledge, or do they know it but fail to use it correctly? We built SciReas and SciReas-Pro, a unified evaluation suite and its selective subset covering ten scientific benchmarks across fields such as biology, chemistry, physics, medicine, mathematics, and engineering. Beyond benchmarking, we introduce KRUX, a framework for separating the roles of knowledge and reasoning in scientific problem-solving. KRUX gives models short, relevant “knowledge ingredients” before they answer a question, allowing us to test whether failures come from missing knowledge, poor knowledge retrieval, or weak reasoning. We find that giving models the right facts can greatly improve their scientific problem-solving. At the same time, reasoning-trained models still benefit from extra knowledge, showing that knowledge and reasoning are complementary. We also find that reasoning training helps models surface useful facts they may already know. Overall, SciReas and KRUX provide tools for diagnosing and improving scientific AI systems.