Iterative Robust Satisficing: Minimizing Performance Degradation Under Distribution Shift

Modern neural networks often achieve high accuracy on their training distribution but degrade sharply under distribution shifts. We address this problem through Robust Satisficing (RS), an optimization objective that seeks parameters which attain a target level of in-distribution performance while minimizing fragility, defined as the rate at which performance deteriorates as the data distribution departs from training. We develop a gradient-based algorithm, Iterative Robust Satisficing (IRS), that directly optimizes this criterion. Across a range of synthetic and real-world distribution shifts, including long-tailed image classification, group shifts induced by spurious correlations, and natural shifts in tabular regression, IRS consistently improves performance on minority and worst-case groups without sacrificing overall accuracy. Notably, IRS achieves these robustness gains with a per-step computational cost similar to standard stochastic gradient descent and requires only a single forward and backward pass per update. Together, these results suggest that minimizing fragility provides a practical and effective alternative to existing robust training methods for learning models that remain reliable under distribution shift.

Machine learning models are often trained on data that looks different from the data they encounter in the real world, causing their performance to degrade unpredictably. Rather than trying to guard against every possible change in the data distribution, which can be overly cautious and hurt normal performance, our method trains models to be as insensitive as possible to distribution shift while still meeting a target level of accuracy on the training data. We formalize this notion of "fragility" mathematically and develop an efficient algorithm, Iterative Robust Satisficing (IRS), that minimizes it at essentially the same computational cost as standard training. In experiments spanning image classification, group-shift benchmarks, and tabular regression, IRS consistently improves performance on minority and worst-case groups without sacrificing overall accuracy.