TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems

TabPFN has emerged as a promising in-context learning model for tabular data, capable of directly predicting the labels of test samples given labeled training examples. It has demonstrated competitive performance, particularly on small-scale classification tasks. However, despite its effectiveness, TabPFN still requires further refinement in several areas, including handling high-dimensional features, aligning with downstream datasets, and scaling to larger datasets.In this paper, we revisit existing variants of TabPFN and observe that most approaches focus either on reducing bias or variance, often neglecting the need to address the other side, while also increasing inference overhead. To fill this gap, we propose Beta (Bagging and Encoder-based Fine-tuning for TabPFN Adaptation), a novel and effective method designed to minimize both bias and variance. To reduce bias, we introduce a lightweight encoder to better align downstream tasks with the pre-trained TabPFN. By increasing the number of encoders in a lightweight manner, Beta mitigates variance, thereby further improving the model’s performance. Additionally, bootstrapped sampling is employed to further reduce the impact of data perturbations on the model, all while maintaining computational efficiency during inference. Our approach enhances TabPFN’s ability to handle high-dimensional data and scale to larger datasets. Experimental results on over 200 benchmark classification datasets demonstrate that Beta either outperforms or matches state-of-the-art methods.

Modern deep learning models often struggle to perform well when working with spreadsheet-like data, especially when the number of features (columns) is large or when only a small amount of labeled data is available. One promising solution, called TabPFN, can make predictions by looking at a few examples—much like how humans can learn quickly from just a few cases. While TabPFN works well in many situations, it still faces challenges with larger and more complex datasets.In our work, we propose a new method called Beta that improves how TabPFN learns and makes predictions. Beta combines two key ideas: using simplified helper models (encoders) to better understand the data, and blending multiple training samples (a technique known as bagging) to make the final predictions more stable. We tested Beta on over 200 real-world datasets and found that it performs as well or better than current top methods, even when the data is high-dimensional.