Row-Attention Extracts, Column-Attention Projects: How ConTextTab Solves In-Context Linear Classification

Tabular Foundation Models (TFMs) achieve strong zero-shot classification via in-context learning, but their internal computations remain poorly understood. We present, to our knowledge, the first causal intervention study of a TFM. ConTextTab is uniquely suited to mechanistic analysis because its per-cell tokenisation preserves the feature axis as a directly observable computational dimension, unlike TFMs that bundle features into row-level tokens. We study it on the Rotated Linear Threshold task in $\mathbb{R}^2$, whose Bayes-optimal algorithm is known exactly, and address its post-LayerNorm architecture by restricting activation patching to sub-block boundaries -- a clean causal baseline that does not require auxiliary modeling choices. We localise a linear plug-in classifier $\mathrm{LD}(x^*) \approx f\bigl(x^* \cdot \hat n(\alpha) - \hat\theta\bigr)$ to the two architectural axes: row attention extracts the boundary normal $\hat n(\alpha)$ and threshold. $\hat\theta$ in a single sub-block (L0), distributed across all twelve heads with none individually dispensable; column-attention then progressively projects the query onto the boundary, with the final three of twelve layers carrying $0.588$ of the causal weight.