Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interaction

Existing tasks fall short in evaluating reasoning ability of Large Language Models (LLMs) in an interactive, unknown environment. This deficiency leads to the isolated assessment of deductive, inductive, and abductive reasoning, neglecting the integrated reasoning process that is indispensable for humans discovery of real world. We introduce a novel evaluation paradigm, black-box interaction, to tackle this challenge. A black-box is defined by a hidden function that maps a specific set of inputs to outputs. LLMs are required to unravel the hidden function behind the black-box by interacting with it in given exploration turns, and reasoning over observed input-output pairs. Leveraging this idea, we build the Oracle benchmark which comprises 6 types of black-box task and 96 black-boxes. 19 modern LLMs are benchmarked. o3, a leading LLM from OpenAI, ranks first in 5 of the 6 tasks, achieving over 70\% accuracy on most easy black-boxes. But it still struggles with some hard black-box tasks, where its average performance drops below 40\%. Further analysis indicates a universal difficulty among LLMs: They lack the high-level planning capability to develop efficient and adaptive exploration strategies for hypothesis refinement. Code is available in https://github.com/lemonsis/Oracle_Benchmark.

Modern AI systems can solve hard math problems and write code, but it's unclear whether they can truly reason the way scientists do — by poking at something unknown, forming guesses, and refining those guesses based on what they observe. Most existing tests check only one slice of reasoning at a time, and tests built from popular games are unreliable because the AI may have already studied those games online. We introduce "black-box environment interaction": we hide a rule inside a box and let the AI ask questions and see answers, just like a chemist running experiments to figure out an unknown reaction. We built Oracle, a collection of 96 such hidden-rule puzzles spanning six flavors — cracking secret codes, reverse-engineering algorithms, identifying physical motions, mapping electronic circuits, solving interactive puzzles, and outsmarting a hidden game opponent. We tested 19 leading AI models, including OpenAI's o3 and Google's Gemini 2.5 Pro. Even the strongest model scores below 40% on hard puzzles. The core weakness we uncovered is striking: today's AI cannot plan its investigations — it explores blindly instead of adjusting strategy based on what it just learned. Closing this gap is a key step toward AI that can genuinely discover new things.