BiomedBench Suite: Benchmarks for Evaluating LLM Performance on Biomedical Reasoning Tasks

Biomedical research demands both fluent reasoning over complex domain knowledge and precise interfacing with structured scientific databases. We present $\textit{BiomedBench Suite}$, a pair of benchmarks built on a shared set of biomedical research questions and evaluated through two complementary modalities, surfacing consistent failure modes in state-of-the-art LLMs. $\textit{CARDBiomedBench}$ evaluates open-ended QA through 68,000 expert-curated question-answer pairs in neurodegenerative disease research, scored for both accuracy and appropriate abstention. $\textit{BiomedSQL}$ extends the same question set to a harmonized biomedical knowledge base, evaluating text-to-SQL generation that requires implicit domain reasoning about significance thresholds, effect directionality, and trial-phase filtering. On CARDBiomedBench, no model effectively balances accuracy with safe abstention. The highest-accuracy model reaches just 51\% response quality rate (RQR), while the safest reaches a 75\% safety rate at only 24\% RQR. On BiomedSQL, the strongest model reaches only 62.6\% execution accuracy against a 90\% expert baseline. We synthesize lessons across both efforts on benchmark construction, evaluation design, and the presistent gap between syntactic competence and scientific reasoning.