Automatic Construction of Clinical Scoring Systems with LLM Agents

Modern clinical practice relies on evidence-based guidelines implemented as compact scoring systems composed of a small number of interpretable decision rules. While machine-learning models achieve strong performance, many fail to translate into routine clinical use due to misalignment with workflow constraints such as memorability, auditability, and bedside execution. We argue that this gap arises not from insufficient predictive power, but from optimizing over model classes that are incompatible with guideline deployment. Deployable guidelines often take the form of unit-weighted clinical checklists, formed by thresholding the sum of binary rules, but learning such scores requires searching an exponentially large discrete space of possible rule sets. We introduce AgentScore, which performs semantically guided optimization in this space by using LLMs to propose candidate rules and a deterministic, data-grounded verification-and-selection loop to enforce statistical validity and deployability constraints. Across eight clinical prediction tasks, AgentScore outperforms existing score-generation methods and achieves AUROC comparable to more flexible interpretable models despite operating under stronger structural constraints. On two additional externally validated tasks, AgentScore achieves higher discrimination than established guideline-based scores.

Doctors often use simple scoring systems to support clinical decisions, such as deciding whether a patient is at high risk or needs closer monitoring. These scores are usually made from a small number of easy-to-check rules, so they can be used quickly at the bedside and understood by clinicians. However, creating such scores is usually slow, manual, and dependent on expert design. This paper introduces AgentScore, a method for automatically building simple clinical scoring systems from medical data. Instead of producing a complex prediction model that requires software to run, AgentScore searches for short checklists of clinically meaningful rules. The language model suggests candidate checklist rules, and a separate validation step tests each rule against patient data before it can be included in the final score. The final result is a compact score in which each rule contributes one point. These scores are not intended for direct clinical use without further testing, but the results suggest that machine learning can help generate candidate clinical tools that are easier to inspect, validate, and potentially use in real-world care.