Pretraining EHR Foundation Models with Patient-Aware Sampling

Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a global token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially biasing learning toward long trajectories. We propose alternative pretraining sequence construction methods, focusing on how training signal is distributed across patients. Specifically, we compare Global Stream, deterministic Patient Chunks, and stochastic Patient Sampling with controllable weighting. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.