TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning

Deep learning models for supervised learning on tabular data are rapidly improving. Notably, ensembles (mixtures of multiple models) often play an important role in achieving top performance, which motivates designing ensemble-first systems rather than treating ensembling as an ad hoc trick. In this work, we present TabPack --- a new ensembling approach that packs many base model-optimizer pairs with different hyperparameters into a single neural network and a single optimizer. The base model-optimizer hyperparameters are sampled randomly, after which all base models are trained in parallel, and the final ensemble is built on the fly during training. As a result, TabPack produces powerful ensembles in a single run, with substantial efficiency gains over traditional approaches. With its remarkable efficiency, strong performance on public benchmarks, and reduced reliance on traditional hyperparameter tuning, TabPack becomes an appealing solution for practitioners, and suggests a new avenue for designing better tabular deep learning systems.