TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model

Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.

A lot of important data is stored in tables, such as Excel spreadsheets. A classical problem is to predict missing values in one column from values in the other columns. For example, one could try to predict diseases from patient data or solar cell efficiency from material composition. Traditionally, the way to go from tables to predictions is through handcrafted algorithms. However, a different paradigm has emerged in which handcrafted algorithms are replaced by big neural networks, which are optimized beforehand on millions of other tables to learn how make good predictions. We introduce TabICLv2, an instance of the latter approach that achieves the best average predictive accuracies across hundreds of tables while also being open-source and faster than its main competitor. We achieve this by introducing better methods to generate millions of synthetic tables and improving the structure of the neural network and its optimization. TabICLv2 gives researchers and practitioners a way to easily obtain more accurate predictions on many tables.