Poster
in
Workshop: 3rd Workshop on Multi-modal Foundation Models and Large Language Models for Life Sciences

Medmarks: An Open-Source LLM Benchmark Suite for Medical Tasks

Benjamin Warner ⋅ Ratna Grandhi ⋅ Max Kieffer ⋅ Aymane Ouraq ⋅ Saurav Panigrahi ⋅ Geetu Ambwani ⋅ Kunal Bagga ⋅ Nikhil Khandekar ⋅ Arya Hariharan ⋅ Nishant Mishra ⋅ Manish Ram ⋅ Shamus Zi Yang Sim ⋅ Ahmed Essouaied ⋅ Adepoju J Moyondafoluwa ⋅ Robert Scholz ⋅ Bofeng Huang ⋅ Molly Beavers ⋅ Srishti Gureja ⋅ Anish Mahishi ⋅ Sameed Khan ⋅ Maxime Griot ⋅ Hunar Batra ⋅ Jean-Benoit Delbrouck ⋅ Siddhant Bharadwaj ⋅ Ronald Clark ⋅ Ashish Vashist ⋅ Anas Zafar ⋅ Leema K Murali ⋅ Harsh Deshpande ⋅ Ameen Patel ⋅ William Brown ⋅ Johannes Hagemann ⋅ Connor Lane ⋅ Paul Scotti ⋅ Tanishq Abraham

Project Page [ OpenReview]

Abstract

Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce \textsc{Medmarks}, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). Code is available at \url{https://anonymous.4open.science/r/med-lm-envs-B76C/}.

Chat is not available.