Robust Learning via Nested Distributionally Robust Optimization

Distributionally Robust Optimization (DRO) is widely used to improve model robustness, with existing methods addressing either geometric perturbations (e.g., input shifts) or statistical contamination (e.g., heavy-tailed noise and outliers) effectively. However, these uncertainty sources often co-exist, and coupling them through a single divergence or optimal transport constraint conflates geometric displacement with loss-based outlierness, frequently discarding informative high-leverage samples. We introduce nested DRO, a bilevel formulation that combines an outer optimistic $\phi$-divergence cleaning step with an inner pessimistic optimal-transport robustification step, thereby decoupling geometric smoothing from statistical cleaning. We prove that this structure naturally induces a geometry-invariant, loss-based reweighting mechanism that separates outlier suppression from transport-induced regularization. We derive a tractable strong dual for the resulting non-convex problem and show its equivalence to variance-regularized risk minimization, leading to a clear statistical interpretation of the induced weights. Empirical results on synthetic and real datasets demonstrate that nested DRO consistently outperforms geometry-coupled DRO baselines, particularly under heavy-tailed contamination where preserving high-leverage structure is crucial.

Machine learning systems often face two kinds of problems at once: small changes to the input data, such as rotated images, and corrupted data, such as mislabeled or noisy examples. Existing robust training methods usually handle one of these problems well, but struggle when both appear together, especially because they may mistake unusual but valid examples for harmful outliers. Our work introduces a new training framework, called nested distributionally robust optimization, that separates these two challenges. It first uses a statistics-based step to reduce the influence of suspicious examples, and then applies a geometry-based step to protect against meaningful input changes. This lets the model keep informative “hard” examples while still guarding against truly corrupted data. We also show that this method has a clear mathematical interpretation and can be trained efficiently. In experiments on synthetic data and CIFAR-10 with mixed corruptions, it performs better than several existing robust baselines. These results suggest a more reliable way to train machine learning models when real-world data are both noisy and geometrically distorted.