Minimal Achievable Sufficient Statistic Learning

We introduce Minimal Achievable Sufficient Statistic (MASS) Learning, a training objective for machine learning models whose minimizers are minimal sufficient statistics with respect to the class of functions being optimized over (e.g. deep networks). In deriving MASS Learning, we also introduce Conserved Differential Information (CDI), an information-theoretic quantity that - unlike standard mutual information - can be usefully applied to deterministically-dependent continuous random variables like the input and output of a deep network. In a series of experiments, we show that deep networks trained with MASS Learning match state–of–the–art performance on supervised learning, uncertainty quantification, and adversarial robustness benchmarks.