Timezone: »
We present an online post-hoc calibration method, called Online Platt Scaling (OPS), which combines the Platt scaling technique with online logistic regression. We demonstrate that OPS smoothly adapts between i.i.d. and non-i.i.d. settings with distribution drift. Further, in scenarios where the best Platt scaling model is itself miscalibrated, we enhance OPS by incorporating a recently developed technique called calibeating to make it more robust. Theoretically, our resulting OPS+calibeating method is guaranteed to be calibrated for adversarial outcome sequences. Empirically, it is effective on a range of synthetic and real-world datasets, with and without distribution drifts, achieving superior performance without hyperparameter tuning. Finally, we extend all OPS ideas to the beta scaling method.
Author Information
Chirag Gupta (Carnegie Mellon University)
Aaditya Ramdas (Carnegie Mellon University)
More from the Same Authors
-
2023 Poster: Fully-Adaptive Composition in Differential Privacy »
Justin Whitehouse · Aaditya Ramdas · Ryan Rogers · Steven Wu -
2023 Oral: Nonparametric Extensions of Randomized Response for Private Confidence Sets »
Ian Waudby-Smith · Steven Wu · Aaditya Ramdas -
2023 Poster: Nonparametric Extensions of Randomized Response for Private Confidence Sets »
Ian Waudby-Smith · Steven Wu · Aaditya Ramdas -
2023 Poster: Sequential Kernelized Independence Testing »
Aleksandr Podkopaev · Patrick Bloebaum · Shiva Kasiviswanathan · Aaditya Ramdas -
2023 Poster: Sequential Changepoint Detection via Backward Confidence Sequences »
Shubhanshu Shekhar · Aaditya Ramdas -
2022 Workshop: Workshop on Distribution-Free Uncertainty Quantification »
Anastasios Angelopoulos · Stephen Bates · Sharon Li · Ryan Tibshirani · Aaditya Ramdas · Stephen Bates -
2021 : Top-label calibration (Spotlight #7) »
Chirag Gupta -
2021 Workshop: Workshop on Distribution-Free Uncertainty Quantification »
Anastasios Angelopoulos · Stephen Bates · Sharon Li · Aaditya Ramdas · Ryan Tibshirani -
2021 Poster: Distribution-Free Calibration Guarantees for Histogram Binning without Sample Splitting »
Chirag Gupta · Aaditya Ramdas -
2021 Spotlight: Distribution-Free Calibration Guarantees for Histogram Binning without Sample Splitting »
Chirag Gupta · Aaditya Ramdas