Block Redundancy in Tabular Foundation Models for Clinical Applications

In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to clinical use because they require no training on patient data, however their architectures still demand GPU inference and discourage the on-premise deployment that hospitals need. Simpler alternatives (e.g. Gradient-Boosted Decision Tree) run on CPU and often match their performance, but at the cost of manual feature engineering, preprocessing, and hyperparameter tuning that transformer models avoid by design. In this paper we show that it is possible to simplify TFMs, by substituting up to ∼94% of the blocks using a closed-form linear translator, while preserving downstream performance and requiring minimal compute. We show results on eight clinical benchmarks of different sizes (including MIMIC-III and multi-center eICU-CRD) and three TabZilla controls, for classification (binary and multi-class) and regression.