Timezone: »
A key challenge facing deep learning is that neural networks are often not robust to shifts in the underlying data distribution. We study this problem from the perspective of the statistical concept of parameter identification. Generalization bounds from learning theory often assume that the test distribution is close to the training distribution. In contrast, if we can identify the ``true'' parameters, then the model generalizes to arbitrary distribution shifts. However, neural networks are typically overparameterized, making parameter identification impossible. We show that for quadratic neural networks, we can identify the function represented by the model even though we cannot identify its parameters. Thus, we can obtain robust generalization bounds even in the overparameterized setting. We leverage this result to obtain new bounds for contextual bandits and transfer learning with quadratic neural networks. Overall, our results suggest that we can improve robustness of neural networks by designing models that can represent the true data generating process. In practice, the true data generating process is often very complex; thus, we study how our framework might connect to neural module networks, which are designed to break down complex tasks into compositions of simpler ones. We prove robust generalization bounds when individual neural modules are identifiable.
Author Information
Kan Xu (University of Pennsylvania)
Hamsa Bastani (Wharton)
Osbert Bastani (University of Pennsylvania)
More from the Same Authors
-
2021 : Improving Human Decision-Making with Machine Learning »
Hamsa Bastani -
2021 : Mind the Gap: Safely Bridging Offline and Online Reinforcement Learning »
Wanqiao Xu · Kan Xu · Hamsa Bastani · Osbert Bastani -
2021 : Mind the Gap: Safely Bridging Offline and Online Reinforcement Learning »
Wanqiao Xu · Kan Xu · Hamsa Bastani · Osbert Bastani -
2021 : Deploying a Machine Learning System for COVID-19 Testing in Greece »
Hamsa Bastani · Kimon Drakopoulos · Vishal Gupta -
2021 : Improving Human Decision-Making with Machine Learning »
Hamsa Bastani · Osbert Bastani · Wichinpong Sinchaisri -
2021 : Improving Human Decision-Making with Machine Learning »
Hamsa Bastani · Osbert Bastani · Wichinpong Sinchaisri -
2023 : TRAC: Trustworthy Retrieval Augmented Chatbot »
Shuo Li · Sangdon Park · Insup Lee · Osbert Bastani -
2023 : TRAC: Trustworthy Retrieval Augmented Chatbot »
Shuo Li · Sangdon Park · Insup Lee · Osbert Bastani -
2023 Poster: PAC Prediction Sets for Large Language Models of Code »
Adam Khakhar · Stephen Mell · Osbert Bastani -
2023 Poster: LIV: Language-Image Representations and Rewards for Robotic Control »
Yecheng Jason Ma · Vikash Kumar · Amy Zhang · Osbert Bastani · Dinesh Jayaraman -
2023 Poster: Robust Subtask Learning for Compositional Generalization »
Kishor Jothimurugan · Steve Hsu · Osbert Bastani · Rajeev Alur -
2022 : Spotlight Presentations »
Adrian Weller · Osbert Bastani · Jake Snell · Tal Schuster · Stephen Bates · Zhendong Wang · Margaux Zaffran · Danielle Rasooly · Varun Babbar -
2022 Poster: Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching »
Yecheng Jason Ma · Andrew Shen · Dinesh Jayaraman · Osbert Bastani -
2022 Spotlight: Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching »
Yecheng Jason Ma · Andrew Shen · Dinesh Jayaraman · Osbert Bastani -
2022 Poster: Understanding Robust Generalization in Learning Regular Languages »
Soham Dan · Osbert Bastani · Dan Roth -
2022 Spotlight: Understanding Robust Generalization in Learning Regular Languages »
Soham Dan · Osbert Bastani · Dan Roth -
2022 Poster: Sequential Covariate Shift Detection Using Classifier Two-Sample Tests »
Sooyong Jang · Sangdon Park · Insup Lee · Osbert Bastani -
2022 Spotlight: Sequential Covariate Shift Detection Using Classifier Two-Sample Tests »
Sooyong Jang · Sangdon Park · Insup Lee · Osbert Bastani -
2021 : Spotlight »
Zhiwei (Tony) Qin · Xianyuan Zhan · Meng Qi · Ruihan Yang · Philip Ball · Hamsa Bastani · Yao Liu · Xiuwen Wang · Haoran Xu · Tony Z. Zhao · Lili Chen · Aviral Kumar -
2021 Poster: Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings »
Kan Xu · Xuanyi Zhao · Hamsa Bastani · Osbert Bastani -
2021 Spotlight: Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings »
Kan Xu · Xuanyi Zhao · Hamsa Bastani · Osbert Bastani -
2020 Poster: Robust and Stable Black Box Explanations »
Hima Lakkaraju · Nino Arsov · Osbert Bastani -
2020 Poster: Generating Programmatic Referring Expressions via Program Synthesis »
Jiani Huang · Calvin Smith · Osbert Bastani · Rishabh Singh · Aws Albarghouthi · Mayur Naik -
2019 Poster: Learning Neurosymbolic Generative Models via Program Synthesis »
Halley R Young · Osbert Bastani · Mayur Naik -
2019 Oral: Learning Neurosymbolic Generative Models via Program Synthesis »
Halley R Young · Osbert Bastani · Mayur Naik