Evaluating System Design Choices in Biomedical AI Agents

Tool-augmented biomedical agents are increasingly being proposed as systems that can plan analyses, interact with computational resources, and return scientific answers from heterogeneous data. However, the design choices that make such agents reliable remain poorly characterized. We report a preliminary ablation study of the Biomni agent on the BixBench-Verified-50 dataset, focusing on two practical components of an agentic workflow: whether the agent uses a tool retriever, and whether a critic is inserted in the reasoning loop. We evaluate two LLM models, GPT-4.1 mini and GPT-4.1, across six critic-retriever settings. The strongest condition was GPT-4.1 with tool retrieval and an end-of-run critic, which achieved 60% accuracy. Overall, our results provide preliminary evidence that the placement and interaction of agentic components can substantially affect both reliability and efficiency in biomedical AI workflows.