Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model’s parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/isonelayer_enough.

Tabular foundation models such as TabPFN and TabICL have demonstrated strong performance on predictive tasks, yet their inference mechanisms remain largely unexplored. In this work, we present the first mechanistic interpretability study of six tabular foundation models. We analyze how predictions emerge across depth and identify distinct stages of inference. Based on these findings, we design a proof-of-concept looped single-layer model that uses only 20% of the original model’s parameters while achieving comparable performance.