Optimizing Pre-Training of Tabular Foundation Models by Shaping Geometry

Tabular foundation models (TFMs) are trained to solve new supervised learning problems in-context, despite not knowing the downstream table, feature distribution, or label rule during pretraining. This suggests that successful classifier TFMs might learn broad, well-spread intermediate representations that later layers can adapt to task-specific decision boundaries. We test this hypothesis in supervised TabPFN-style pretraining and find that it does not hold: while the model learns in-context prediction, its intermediate embeddings do not naturally become more isotropic or well dispersed. We therefore introduce row- and column-wise Sketched Isotropic Gaussian Regularization (SIGReg) for tabular transformer hidden states and use it to directly shape representation geometry. Early-layer SIGReg reorganizes the model across depth, producing better-distributed early embeddings while preserving later layers for task-specific decision structure. This improves convergence speed, classification performance, and seed-to-seed stability with little additional overhead.