Do Tabular Foundation Models Learn Rules or Memorize Exemplars?

We investigate how tabular foundation models (TFMs) generalize during in-context learning (ICL), focusing on whether they rely on memorizing context examples or learning abstract rules. Standard evaluation metrics assess empirical performance, but provide limited insight into the underlying mechanisms driving model predictions. Understanding these mechanisms enables better prediction of model behavior on new downstream tasks and supports more robust applications. To address this, we adapt and extend controlled experimental frameworks to the tabular domain and evaluate six state-of-the-art TFMs under settings that distinguish exemplar-based interpolation from rule-based extrapolation. We empirically examine how generalization behavior evolves with the number of training samples and across model layers. We observe that models with similar predictive performance can exhibit substantially different generalization strategies, with a common trend of transitioning from exemplar-based behavior in early layers to more rule-based reasoning in deeper layers. These findings suggest that predictive accuracy alone is insufficient to characterize model behavior.