TabularBERT: Binning-Based Self-Supervised Learning for Tabular Representation

Tabular data is one of the most fundamental and widely used formats for representing structured information. Classical machine learning algorithms continue to achieve substantial success in extracting predictive patterns and constructing accurate models from structured data; however, representation learning approaches that extend language-model-based methods to the tabular setting have opened new opportunities. Nevertheless, conventional tokenization procedures and token embedding mechanisms are not well-suited to numerical variables, as they fail to preserve key numerical properties, including proximity structure and ordinal relationships. To address this limitation, we propose TabularBERT, a Transformer-based model that discretizes numerical variables via binning-based tokenization and learns representations that account for numerical proximity and ordinal information while capturing conditional dependencies among variables through masked self-supervised pretraining. We empirically demonstrate the effectiveness and interpretability of the proposed approach, highlighting the benefits of language-model-based representation learning in the tabular domain.

Many important datasets, such as medical records, financial tables, and customer databases, are stored in tables. While language models have been very successful for text, applying similar ideas to tables is difficult because tables often contain numerical values, such as age, income, or temperature. Unlike words, numerical values have an order and distance: for example, 50 is closer to 51 than to 90. Standard tokenization methods used in language models do not naturally preserve this structure. We propose TabularBERT, a model that adapts language-model-style learning to tabular data. The main idea is to divide numerical values into bins and treat these bins as tokens, while encouraging nearby bins to have similar representations. The model is first trained to recover masked table entries from the remaining entries, helping it learn relationships among variables, and is then fine-tuned for prediction tasks. We also use learning objectives that account for the order of numerical bins and reduce information loss caused by discretization. Our experiments show that TabularBERT performs strongly on a range of tabular prediction tasks. The results suggest that language-model-based representation learning can be useful for tabular data when numerical structure is carefully handled.