Timezone: »
Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution samples. This raises the question of whether these models have learned to solve a dataset rather than the underlying task by overfitting to spurious dataset biases. We investigate one recently proposed approach, AFLITE, which adversarially filters such dataset biases, as a means to mitigate the prevalent overestimation of machine performance. We provide a theoretical understanding for AFLITE, by situating it in the generalized framework for optimum bias reduction. We present extensive supporting evidence that AFLITE is broadly applicable for reduction of measurable dataset biases, and that models trained on the filtered datasets yield better generalization to out-of-distribution tasks. Finally, filtering results in a large drop in model performance (e.g., from 92% to 62% for SNLI), while human performance still remains high. Our work thus shows that such filtered datasets can pose new research challenges for robust generalization by serving as upgraded benchmarks.
Author Information
Ronan Le Bras (Allen Institute for AI)
Swabha Swayamdipta (Allen Institute for AI)
Chandra Bhagavatula (AI2)
Rowan Zellers (University of Washington)
Matthew Peters (AI2)
Ashish Sabharwal (Allen Institute for AI)
Yejin Choi (University of Washington)
More from the Same Authors
-
2023 : SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks »
Yuchen Lin · Yicheng Fu · Karina Yang · Prithviraj Ammanabrolu · Faeze Brahman · Shiyu Huang · Chandra Bhagavatula · Yejin Choi · Xiang Ren -
2023 : Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker »
Melanie Sclar · Sachin Kumar · Peter West · Alane Suhr · Yejin Choi · Yulia Tsvetkov -
2023 : Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker »
Melanie Sclar · Sachin Kumar · Peter West · Alane Suhr · Yejin Choi · Yulia Tsvetkov -
2023 Poster: Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling »
Kolby Nottingham · Prithviraj Ammanabrolu · Alane Suhr · Yejin Choi · Hannaneh Hajishirzi · Sameer Singh · Roy Fox -
2023 Oral: Specializing Smaller Language Models towards Multi-Step Reasoning »
Yao Fu · Hao Peng · Litu Ou · Ashish Sabharwal · Tushar Khot -
2023 Poster: Specializing Smaller Language Models towards Multi-Step Reasoning »
Yao Fu · Hao Peng · Litu Ou · Ashish Sabharwal · Tushar Khot -
2022 Poster: Staged Training for Transformer Language Models »
Sheng Shen · Pete Walsh · Kurt Keutzer · Jesse Dodge · Matthew Peters · Iz Beltagy -
2022 Spotlight: Staged Training for Transformer Language Models »
Sheng Shen · Pete Walsh · Kurt Keutzer · Jesse Dodge · Matthew Peters · Iz Beltagy -
2022 Poster: Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information »
Kawin Ethayarajh · Yejin Choi · Swabha Swayamdipta -
2022 Oral: Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information »
Kawin Ethayarajh · Yejin Choi · Swabha Swayamdipta