SEDRAS: Symbolically Evaluated Deep Research And Science

As the reasoning capabilities of Large Language Models (LLMs) expand, evaluating true inductive generalization on entirely unseen data becomes increasingly challenging. To this end, we introduce a modular in-context learning evaluation framework, that is scalable and extendable across its separate modules. This is based upon the notion of synthetic scenarios with controllable complexity across three independent axes: 1) the logic of the underlying data distribution (UDD) 2) their projection into diverse representations, and 3) the interaction dynamic determining how the model accesses and explores the data. For these scenarios, the model is tasked to perform in-context scientific discovery and produce an interpretable theory in natural language that explains the observations. In a separate conversation, the model is then tasked to convert this generated theory into executable code, which can be programmatically compared against the underlying data distribution. Using this modular framework we produce an initial suite of 600 diverse scenarios that we use to evaluate and analyze various state-of-the-art LLMs. Although these experiments show that Gemini 3.0 Pro achieves the best overall score, each model performs the best at different tasks. For example: GPT 5.2 is the clear winner on pure symbolic data, Claude Opus 4.5 is the best at working with files, Gemini is the strongest model for the non-dynamic scenarios, and Grok 4.1 is the strongest model when UDD complexity scales. Furthermore, all models struggle with active exploration and are seemingly incapable of identifying informative data points, resulting in less efficient exploration than a random baseline.

As artificial intelligence (AI) continues to advance, the methods we use to test its true reasoning skills must adapt. Many current tests are "static," meaning they rely on fixed datasets published online, which AI models can easily memorize or encounter during their training. To address this, we introduce SEDRAS, a new testing framework that creates completely unique, customizable mystery scenarios from the ground up. In these scenarios, the AI is asked to act like a scientist: it must examine unfamiliar data, run its own experiments, and write a human-readable theory that explains the underlying rules of the system. Using this framework, we tested several world-leading AI models and discovered that they have different strengths and weaknesses depending on the task. Some excel at pure logic, while others are better at reading text or sorting through files. Crucially, our findings reveal a major bottleneck in current AI development: when left to guide their own research, today's best models struggle to figure out which experiments are worth running, often exploring less efficiently than a completely random strategy. By exposing these hidden weaknesses, SEDRAS provides a more resilient and evolving yardstick to measure true problem-solving capabilities as AI looks to assist humans in real-world scientific discovery.