TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning

Deep learning models for supervised learning on tabular data are rapidly improving. Notably, ensembles (mixtures of multiple models) often play an important role in achieving top performance, which motivates designing ensemble-first systems rather than treating ensembling as an ad hoc trick. In this work, we present TabPack — a new ensembling approach that packs many base model–optimizer pairs with different hyperparameters into a single neural network and a single optimizer. The base model and optimizer hyperparameters are sampled randomly, after which all base models are trained in parallel, and the final ensemble is built on the fly during training. As a result, TabPack produces powerful ensembles in a single run, with substantial efficiency gains over traditional approaches. With its remarkable efficiency, strong performance on medium-to-large datasets, and reduced reliance on traditional hyperparameter tuning, TabPack is an appealing solution for practitioners and researchers that makes tabular DL more accessible on consumer-grade hardware and suggests a new avenue for designing better tabular deep learning systems.

Modern AI methods are becoming better at working with table-like data, such as customer records, medical measurements, or financial transactions, but getting the best results often requires training and combining many separate models. This can be slow, expensive, and difficult to tune, especially for users without high-end hardware. We developed TabPack, a system that trains many related models at once inside a single larger model, while also testing different settings for each of them. During training, TabPack automatically builds a final combined predictor from the most useful models, instead of requiring users to train many models separately and combine them afterwards. This makes the usual "try many models and ensemble them" workflow much faster and more practical. In our experiments, TabPack achieves strong accuracy on medium-to-large tabular datasets while substantially reducing the time and hardware needed for experimentation. It can even enable experiments on consumer-grade devices such as a modern MacBook. This matters because tabular data is central to many real-world machine learning applications, and TabPack makes strong deep learning solutions for such data more accessible to both researchers and practitioners.