Tabular Foundation Models Are Effectively Shallow

In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources since they require no additional training on the target data. However, their architectures impose high inference latency and memory costs, hindering deployment in some environments. Simpler alternatives (e.g. Gradient-Boosted Decision Trees) run on CPU and often match their performance, but at the cost of manual feature engineering, preprocessing, and hyperparameter tuning that transformer models avoid by design. In this paper we show that it is possible to simplify TFMs, by substituting up to $\sim$94\% of the blocks using a closed-form linear translator, while preserving downstream performance and requiring minimal compute. We evaluate our approach across 15 classification and regression benchmarks spanning both general and clinical domains. This provides a practical pathway to dramatic throughput enhancements and lightweight inference, preserving the zero-shot convenience of foundation models without the prohibitive computational costs.