The Illusion of Generalization in Tabular Language Models

Tabular Language Models (TLMs) have been claimed to achieve strong generalization for tabular prediction. We conduct a systematic re-evaluation of Tabula-8B as a representative TLM, utilizing 165 datasets from the UniPredict benchmark. Our investigation reveals three findings. First, binary and categorical classification achieve near-zero median lift over majority-class baselines and strong aggregate performance is driven entirely by quartile classification tasks. Second, top-performing datasets exhibit pervasive contamination, including complete train-test overlap and task-level leakage that evades standard deduplication. Third, instruction-tuning without tabular exposure recovers 92.2% of standard classification performance and on quartile classification, format familiarity closes 71.3% of the gap with the residual attributable to contaminated datasets. These findings suggest claimed generalization likely reflects evaluation artifacts rather than learned tabular reasoning. We conclude with recommendations for strengthening TLM evaluation.

Spreadsheets and data tables are pervasive in the modern world, from patient histories to loan applications to fraud alerts. Predicting from them has long belonged to specialized statistical tools, but a new class of tabular foundation models (TLMs) recently claimed to beat those tools, even on data they were never trained on. If true, that would matter enormously for fields like medicine and finance, so we set out to verify it. Re-examining a leading TLM across 165 datasets, we found its success largely dissolves under scrutiny. On four out of ten datasets it did no better than guessing the most common answer, and its impressive metrics leaned almost entirely on one unusually easy task type. Eight of the ten highest-scoring datasets turned out to be contaminated, with test answers sitting in the training data where the model could simply recall them. Most surprisingly, a plain model with no exposure to tables at all recovered 92% of the TLM's accuracy. The headline results reflect flaws in how these systems were tested, not a genuine new ability to understand and predict unseen tables. We offer seven recommendations to keep unreliable tools out of high-stakes settings.