Unveiling the Role of Data Uncertainty in Tabular Deep Learning

Recent advancements in tabular deep learning have demonstrated exceptional practical performance, yet the field often lacks a clear understanding of why these techniques actually succeed. To address this gap, our paper highlights the importance of the concept of data (aleatoric) uncertainty for explaining the effectiveness of recent tabular DL methods. While data uncertainty leads to irreducible prediction errors on test samples, it also introduces stochasticity into the training signal that can impede effective learning. We demonstrate that tabular methods differ significantly in their ability to cope with this optimization challenge. Specifically, we reveal that the success of many beneficial design choices in tabular DL, such as numerical feature embeddings, advanced ensembling strategies, retrieval-augmented models, and tabular Prior-Fitted Networks, can be partially attributed to their respective implicit mechanisms for performing well under high data uncertainty. By dissecting these varied mechanisms, we provide a unifying understanding of recent performance improvements. Furthermore, leveraging insights from this perspective, we design a novel, more effective numerical feature embedding method as an immediate practical outcome of our analysis. Overall, our work paves the way toward a principled understanding of the benefits introduced by modern tabular methods that results in the concrete advancements of existing techniques and outlines future research directions for tabular DL.

Tabular data are spreadsheet-like records, with rows for examples and columns for properties such as age, price, or location. Neural networks have recently become much better at working with such data, thanks to many creative new architectures and training techniques, but we still lack a clear understanding of why some of these ideas work well and why they differ from the standard recipes used for neural networks that process images or text. In this paper we highlight one important characteristic of many tabular tasks: the recorded columns often do not contain all the information needed to determine the correct answer. For example, two apartments with similar recorded properties may still sell for different prices because of unrecorded factors, or a measured label may simply be noisy. This makes predictions inherently uncertain: even the best model may not be able to know the exact answer from the available information. Such noise not only limits final accuracy, but also makes training harder by giving models inconsistent feedback during training. We empirically study several recent tabular neural-network methods through this lens and find that their gains are often strongest on the most uncertain examples. Although these methods work in different ways, our results suggest that part of their success comes from reducing the harm caused by noisy training signals. This understanding also leads us to a better way of representing numerical columns, helping make progress in tabular deep learning more principled.