Timezone: »
Humans can learn and reason under substantial uncertainty in a space of infinitely many compositional, productive concepts. For example, if a scene with two blue spheres qualifies as “daxy,” one can reason that the underlying concept may require scenes to have “only blue spheres” or “only spheres” or “only two objects.” In contrast, standard benchmarks for compositional reasoning do not explicitly capture a notion of reasoning under uncertainty or evaluate compositional concept acquisition. We introduce a new benchmark, Compositional Reasoning Under Uncertainty (CURI) that instantiates a series of few-shot, meta-learning tasks in a productive concept space to evaluate different aspects of systematic generalization under uncertainty, including splits that test abstract understandings of disentangling, productive generalization, learning boolean operations, variable binding, etc. Importantly, we also contribute a model-independent “compositionality gap” to evaluate the difficulty of generalizing out-of-distribution along each of these axes, allowing objective comparison of the difficulty of each compositional split. Evaluations across a range of modeling choices and splits reveal substantial room for improvement on the proposed benchmark.
Author Information
Shanmukha Ramakrishna Vedantam (Facebook AI Research)
Arthur Szlam (Facebook)
Maximilian Nickel (Facebook AI Research)
Ari Morcos (Facebook AI Research (FAIR))
Brenden Lake (New York University)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Poster: CURI: A Benchmark for Productive Concept Learning Under Uncertainty »
Fri. Jul 23rd 04:00 -- 06:00 AM Room
More from the Same Authors
-
2022 : Learning to Discretize for Continuous-time Sequence Compression »
Ricky T. Q. Chen · Maximilian Nickel · Matthew Le · Matthew Muckley · Karen Ullrich -
2023 : SemDeDup: Data-efficient learning at web-scale through semantic deduplication »
Amro Abbas · Daniel Simig · Surya Ganguli · Ari Morcos · Kushal Tirumala -
2023 : D4: Document Deduplication and Diversification »
Kushal Tirumala · Daniel Simig · Armen Aghajanyan · Ari Morcos -
2022 Poster: Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time »
Mitchell Wortsman · Gabriel Ilharco · Samir Gadre · Becca Roelofs · Raphael Gontijo Lopes · Ari Morcos · Hongseok Namkoong · Ali Farhadi · Yair Carmon · Simon Kornblith · Ludwig Schmidt -
2022 Spotlight: Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time »
Mitchell Wortsman · Gabriel Ilharco · Samir Gadre · Becca Roelofs · Raphael Gontijo Lopes · Ari Morcos · Hongseok Namkoong · Ali Farhadi · Yair Carmon · Simon Kornblith · Ludwig Schmidt -
2022 Poster: COAT: Measuring Object Compositionality in Emergent Representations »
Sirui Xie · Ari Morcos · Song-Chun Zhu · Shanmukha Ramakrishna Vedantam -
2022 Spotlight: COAT: Measuring Object Compositionality in Emergent Representations »
Sirui Xie · Ari Morcos · Song-Chun Zhu · Shanmukha Ramakrishna Vedantam -
2022 Poster: Matching Normalizing Flows and Probability Paths on Manifolds »
Heli Ben-Hamu · samuel cohen · Joey Bose · Brandon Amos · Maximilian Nickel · Aditya Grover · Ricky T. Q. Chen · Yaron Lipman -
2022 Spotlight: Matching Normalizing Flows and Probability Paths on Manifolds »
Heli Ben-Hamu · samuel cohen · Joey Bose · Brandon Amos · Maximilian Nickel · Aditya Grover · Ricky T. Q. Chen · Yaron Lipman -
2021 : Invited Talk 6 (Maximilian Nickel): Modeling Spatio-Temporal Events via Normalizing Flows »
Maximilian Nickel -
2021 Poster: ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases »
Stéphane d'Ascoli · Hugo Touvron · Matthew Leavitt · Ari Morcos · Giulio Biroli · Levent Sagun -
2021 Poster: Not All Memories are Created Equal: Learning to Forget by Expiring »
Sainbayar Sukhbaatar · Da JU · Spencer Poff · Stephen Roller · Arthur Szlam · Jason Weston · Angela Fan -
2021 Oral: Not All Memories are Created Equal: Learning to Forget by Expiring »
Sainbayar Sukhbaatar · Da JU · Spencer Poff · Stephen Roller · Arthur Szlam · Jason Weston · Angela Fan -
2021 Spotlight: ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases »
Stéphane d'Ascoli · Hugo Touvron · Matthew Leavitt · Ari Morcos · Giulio Biroli · Levent Sagun -
2020 : Collaboration in Situated Instruction Following Q&A »
Yoav Artzi · Arthur Szlam -
2020 : Collaborative Construction and Communication in Minecraft Q&A »
Julia Hockenmaier · Arthur Szlam -
2020 Workshop: Workshop on Learning in Artificial Open Worlds »
Arthur Szlam · Katja Hofmann · Ruslan Salakhutdinov · Noboru Kuno · William Guss · Kavya Srinet · Brandon Houghton -
2020 Poster: Fast Adaptation to New Environments via Policy-Dynamics Value Functions »
Roberta Raileanu · Max Goldstein · Arthur Szlam · Facebook Rob Fergus -
2019 Workshop: Identifying and Understanding Deep Learning Phenomena »
Hanie Sedghi · Samy Bengio · Kenji Hata · Aleksander Madry · Ari Morcos · Behnam Neyshabur · Maithra Raghu · Ali Rahimi · Ludwig Schmidt · Ying Xiao -
2019 Poster: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh -
2019 Oral: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh -
2018 Poster: Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks »
Brenden Lake · Marco Baroni -
2018 Oral: Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks »
Brenden Lake · Marco Baroni -
2018 Poster: Measuring abstract reasoning in neural networks »
Adam Santoro · Feilx Hill · David GT Barrett · Ari S Morcos · Timothy Lillicrap -
2018 Poster: Optimizing the Latent Space of Generative Networks »
Piotr Bojanowski · Armand Joulin · David Lopez-Paz · Arthur Szlam -
2018 Poster: Composable Planning with Attributes »
Amy Zhang · Sainbayar Sukhbaatar · Adam Lerer · Arthur Szlam · Facebook Rob Fergus -
2018 Oral: Measuring abstract reasoning in neural networks »
Adam Santoro · Feilx Hill · David GT Barrett · Ari S Morcos · Timothy Lillicrap -
2018 Oral: Composable Planning with Attributes »
Amy Zhang · Sainbayar Sukhbaatar · Adam Lerer · Arthur Szlam · Facebook Rob Fergus -
2018 Oral: Optimizing the Latent Space of Generative Networks »
Piotr Bojanowski · Armand Joulin · David Lopez-Paz · Arthur Szlam -
2018 Poster: Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry »
Maximilian Nickel · Douwe Kiela -
2018 Oral: Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry »
Maximilian Nickel · Douwe Kiela