TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention

Tabular foundation models, exemplified by TabPFN, perform prediction via in-context learning, inferring test labels directly from labeled training examples. They have demonstrated competitive performance, particularly on small-to-medium datasets. However, recent tabular foundation models often improve accuracy with increasingly complex architectures, incurring higher inference cost and limiting practical deployment. In this work, we revisit the original TabPFN design and show that a lightweight row-wise attention–only backbone can remain highly competitive with two simple enhancements: a gated attention stabilization mechanism and a small set of learnable register tokens that provide global context and improve pretraining quality. The resulting model, TabSwift, supports both classification and regression, and is competitive with stronger tabular foundation models (e.g., TabPFN v2 and TabICL) while being more efficient at inference. For latency-sensitive serving, we further introduce an adaptive layer-wise early-exit mechanism that dynamically adjusts inference depth per sample. Overall, TabSwift enables efficient and anytime tabular in-context learning for practical deployments.

Tabular data, such as spreadsheets of customer records, medical measurements, financial indicators, or scientific observations, is one of the most common data formats in real-world applications. Recent tabular foundation models can make predictions by looking at a small set of labeled examples, but many of them become increasingly complex and expensive to run, which limits their use in practical systems where fast responses are important. This paper introduces TabSwift, a simpler and more efficient foundation model for tabular data. Instead of using a heavy model structure, TabSwift keeps a lightweight design and adds two simple improvements that help the model learn more stably and use the information in a table more effectively. It can handle both classification tasks, such as predicting a category, and regression tasks, such as predicting a numerical value. We also introduce an adaptive prediction mechanism that allows the model to stop early for easy examples and spend more computation only on harder ones. Experiments on a large collection of public tabular datasets show that TabSwift achieves competitive prediction quality compared with stronger tabular foundation models while running faster. These results suggest that practical tabular foundation models do not always need to be more complex; carefully designed lightweight models can offer a better balance between accuracy and efficiency.