Counterfactual Residual Data Augmentation for Regression

Data-driven modeling in real-world regression tasks often suffers from limited training samples, high collection costs, and noisy observations. Inspired by the impact of data augmentation in vision and language, we propose a novel Counterfactual Residual Data Augmentation (CRDA) technique for tabular regression. Our key insight is that once a regressor has modeled the systematic component of the data, the remaining noise can be viewed as an invariant residual that remains stable under small perturbations of carefully selected features. We exploit this residual invariance to generate new, yet realistic, training samples, effectively expanding the dataset without requiring additional real data. Our method is model-agnostic and readily applicable to various types of regressors. In experiments across datasets from a variety of benchmark repositories, on average, CRDA reduces an MLP Regressor's MSE by 22.9% and an XGBoost Regressor's MSE by 6.4%. When compared to existing state-of-the-art data generators and augmentation techniques, CRDA consistently outperforms in MSE reduction. By adding principled counterfactual variations to the training data, our method offers a simple and efficient remedy for noise-prone, small-sample regression settings.

Many everyday predictions — a house's price, a patient's recovery time, a factory's output — come from models trained on data in tables. In many real settings, collecting enough data is slow, costly, or impossible, and models trained on too few examples predict poorly. Researchers know how to stretch small datasets of images or text with realistic tweaks like flips or rephrasings, but no equally simple trick existed for tables of numbers. We developed CRDA, a method that creates believable new training examples. After a model captures the general pattern, each real example still differs slightly from what the model predicted, forming an unexplained gap or error specific to that example. CRDA changes a few of the example's features, recalculates the model's predicted outcome, and then re-applies that same gap, producing a new and realistic example. Built-in checks keep only the synthetic examples that actually help, and otherwise leave the model unchanged. Across many datasets, CRDA cut prediction errors substantially, and more reliably than existing methods, especially when data was scarce. It offers a simple, safe way to get more from limited data in fields like medicine, finance, and manufacturing, where data is expensive to collect.