FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

Autonomous AI agents powered by large language models (LLMs) are increasingly capable of running a full cycle of scientific research, yet we still lack reliable ways to verify that their discoveries are correct. Because novel findings demand costly real-world validation, existing benchmarks fall back on LLM-as-judge scoring of generated papers or single leaderboard metrics, both coarse proxies for scientific reasoning. We introduce FIRE-BENCH (Full-cycle Insight Rediscovery Evaluation), which instead asks agents to rediscover established, verifiable findings from recent, high-impact machine learning research. Given only a high-level research question from a published study, an agent must independently design experiments, run them, and draw evidence-backed conclusions, scored against the study's documented findings. Across state-of-the-art agents with frontier backbones such as gpt-5, even the strongest reaches limited rediscovery success (<50 F1), with high run-to-run variance and recurring failures in experimental design, execution, and evidence-based reasoning. Beyond diagnosing current systems, FIRE-BENCH shows that open-ended discovery can be evaluated rigorously and verifiably, laying a foundation for building reliable environments that improve agents.

AI assistants like ChatGPT now write code and analyze data, and researchers increasingly use them as "automated scientists" that brainstorm ideas, design experiments, and draw conclusions. The hard question is whether these discoveries are correct, since real verification requires costly lab work or expert panels. Existing tests either use AI to grade AI (LLM-as-judge) or measure single leaderboard scores. We built FIRE-Bench, a new benchmark and evaluation protocol that gives an AI agent only the research question from a top peer-reviewed paper (for example, "do language models predict different medical costs for different racial groups?") and asks it to rediscover the answer: design the experiment, write code, run it, and report a conclusion. Because we already know what the human scientists found ("the truth"), we score the agent's answer against it. Even today's most capable agents (OpenAI's Codex, Anthropic's Claude Code) get less than half right and answer inconsistently across runs. Our novel finding: their weakness is not coding but designing the experiment and drawing trustworthy conclusions from data. Beyond a rigorous yardstick for progress, FIRE-Bench provides the foundation for training future research agents in reliable RL environments.