BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

We introduce BioAgent Bench, an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The suite consists of manually curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) accompanied by task-specific prompts and concrete output artifacts to support automated assessment. We evaluate frontier closed- and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that agents based on frontier LLMs can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, bioinformatics workflows often involve sensitive patient data or unpublished intellectual property, thereby making the use of cost-effective yet reliable local agents an imperative. Therefore, by releasing the code and the complementary resources comprising our suite, we aim to accelerate the development of such privacy-preserving agents.

Scientists who study biological data like DNA and RNA sequences rely on long chains of computer analyses, trimming raw sequencing data, aligning it to a reference, calling genetic variants, and so on. AI "agents" built on large language models could automate this tedious work, but we didn't know how reliable they really are. We built BioAgent Bench, a test suite of ten realistic, start-to-finish bioinformatics data analysis tasks, each built around a different organism or microbial community. We ran ten leading AI systems through these tasks and used a separate AI to grade their work against known correct answers. Crucially, we also sabotaged the inputs, corrupting files, slipping in misleading "decoy" data, and padding instructions with irrelevant text to see whether the agents would successfully recognize these modifications and successfully complete the tasks. The best AI agents finished most pipelines successfully without special hand-holding. But our sabotage tests exposed that agents failed to ignore broken or wrong data instead of catching the problem. By releasing our suite openly, we hope to help build private, locally-run AI tools that scientists in biology can actually rely on.