Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models

Clinical deployment of foundation models requires decision policies that operate under explicit error budgets, such as a cap on false-positive clinical calls. Strong average accuracy alone does not guarantee safety: errors can concentrate among patients selected for action, leading to harm and inefficient use of healthcare resources. Here we develop StratCP, a stratified conformal framework that turns foundation model predictions into decision-ready outputs through error-controlled selection and calibrated deferral. StratCP first selects a subset of patients for immediate clinical action while controlling false discovery rates at a user-specified level. For the remaining patients, it returns prediction sets that achieve target coverage conditional on deferral, supporting confirmatory testing or expert review. We evaluate StratCP on Electronic Health Record (EHR) foundation model predictions across EHRSHOT tasks spanning operational outcomes, assignment of new diagnoses, and anticipating lab test results, including length of stay, pancreatic cancer, and thrombocytopenia severity prediction. Across tasks, StratCP controls class-specific false discovery rates among selected predictions and provides valid, selection-conditional coverage for deferred patients, with the largest gains on rarer and higher-stakes classes. StratCP establishes error-controlled decision policies for safe deployment of medical foundation models.