BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science

The realization of autonomous scientific experimentation is currently limited by LLMs' struggle to grasp the strict procedural logic and accuracy required by biological protocols. To address this fundamental challenge, we present BioProBench, a comprehensive resource for procedural reasoning in biology. BioProBench is grounded in BioProCorpus, a foundational collection of 27,000 human-written protocols. From this corpus, we systematically constructed a dataset of over 550,000 task instances, offering both a large-scale training resource and a rigorous benchmark with novel metrics. Evaluating 10 mainstream LLMs, we find that while general comprehension is high, performance drops significantly on tasks demanding deep reasoning, quantitative precision, and safety awareness. To demonstrate the value of BioProCorpus in mitigating these issues, we developed \textbf{ProAgent}, grounded in our corpus, ProAgent substantially advances the state-of-the-art. Code and data are available at: https://github.com/YuyangSunshine/bioprotocolbench and https://huggingface.co/datasets/BioProBench/BioProBench .

Automating biological experiments requires Artificial Intelligence (AI) to accurately understand and follow complex, step-by-step laboratory instructions. However, current AI models struggle with the strict logic, precise measurements, and safety requirements of real-world biology. To solve this, we created BioProBench, a massive collection of 27,000 human-written biology protocols and over 550,000 test questions. This resource helps train and test AI on how to safely follow lab procedures. When we tested 10 leading AI models, we found that while they understand general scientific text, they frequently fail at precise, step-by-step reasoning. To demonstrate how to fix this, we built ProAgent, an AI system that uses our new database to double-check its work. This significantly improved the AI's ability to understand and plan experiments accurately, paving the way for safer and more reliable automated science laboratories.