Non-Asymptotic and Non-Lipschitzian Bounds on Optimal Values in Stochastic Optimization Under Heavy Tails

This paper focuses on non-asymptotic confidence bounds (CB) for the optimal values of stochastic optimization (SO) problems. Existing approaches often rely on two conditions that may be restrictive: The need for a global Lipschitz constant and the assumption of light-tailed distributions. Beyond either of the conditions, it remains largely unknown whether computable CBs can be constructed. In view of this literature gap, we provide three key findings below: (i) Based on the conventional formulation of sample average approximation (SAA), we derive non-Lipschitzian CBs for convex SP problems under heavy tails. (ii) We explore diametrical risk minimization (DRM)---a recently introduced modification to SAA---and attain non-Lipschitzian CBs for nonconvex SP problems in light-tailed settings. (iii) We extend our analysis of DRM to handle heavy-tailed randomness by utilizing properties in formulations for training over-parameterized classification models.

Training machine learning models often involves solving stochastic optimization problems based on noisy or unpredictable data. But how confident can we be in how well the model will perform in the real world? Most existing results for estimating this confidence rely on strong assumptions about randomness and data behavior.Our work develops new ways to estimate confidence without these strict assumptions. By building on traditional methods and their robust extensions, we proposed better estimations of this confidence even when the data contains many outliers or extreme values.This allows machine learning decisions to be validated more reliably in real-world applications, especially when dealing with volatile or unpredictable data.