Autoregressive EHR Foundation Models with Multimodal Inputs

Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. Through controlled ablations on MIMIC-IV, we study two key design choices for multimodal EHR fusion: how to compress long modality sequences before cross-attention, and whether the choice of pretrained modality encoder matters for downstream performance. We show that latent compression substantially outperforms both uncompressed cross-attention and mean pooling, and that encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further find that, under our current architecture, simply adding auxiliary modalities does not guarantee improvement on aggregate ICU mortality prediction over a strong EHR-only baseline, motivating future work on more flexible fusion architectures and clinically contextual evaluation.