Timezone: »
Whether neural networks can learn abstract reasoning or whether they merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation 'regimes' in which the training data and test questions differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test sets differ only minimally, and we present a novel architecture, with structure designed to encourage reasoning, that does significantly better. When we vary the way in which the test questions and training data differ, we find that our model is notably proficient at certain forms of generalisation, but notably weak at others. We further show that the model's ability to generalise improves markedly if it is trained to predict symbolic explanations for its answers. Altogether, we introduce and explore ways to both measure and induce stronger abstract reasoning in neural networks. Our freely-available dataset should motivate further progress in this direction.
Author Information
Adam Santoro (DeepMind)
Feilx Hill (Deepmind)
David GT Barrett (DeepMind)
Ari S Morcos (DeepMind)
Timothy Lillicrap (Google DeepMind)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Poster: Measuring abstract reasoning in neural networks »
Thu. Jul 12th 04:15 -- 07:00 PM Room Hall B #110
More from the Same Authors
-
2023 Poster: The Edge of Orthogonality: A Simple View of What Makes BYOL Tick »
Pierre Richemond · Allison Tam · Yunhao Tang · Florian Strub · Bilal Piot · Feilx Hill -
2022 Poster: Tell me why! Explanations support learning relational and causal structure »
Andrew Lampinen · Nicholas Roy · Ishita Dasgupta · Stephanie Chan · Allison Tam · James McClelland · Chen Yan · Adam Santoro · Neil Rabinowitz · Jane Wang · Feilx Hill -
2022 Poster: Retrieval-Augmented Reinforcement Learning »
Anirudh Goyal · Abe Friesen Friesen · Andrea Banino · Theophane Weber · Nan Rosemary Ke · Adrià Puigdomenech Badia · Arthur Guez · Mehdi Mirza · Peter Humphreys · Ksenia Konyushkova · Michal Valko · Simon Osindero · Timothy Lillicrap · Nicolas Heess · Charles Blundell -
2022 Spotlight: Retrieval-Augmented Reinforcement Learning »
Anirudh Goyal · Abe Friesen Friesen · Andrea Banino · Theophane Weber · Nan Rosemary Ke · Adrià Puigdomenech Badia · Arthur Guez · Mehdi Mirza · Peter Humphreys · Ksenia Konyushkova · Michal Valko · Simon Osindero · Timothy Lillicrap · Nicolas Heess · Charles Blundell -
2022 Spotlight: Tell me why! Explanations support learning relational and causal structure »
Andrew Lampinen · Nicholas Roy · Ishita Dasgupta · Stephanie Chan · Allison Tam · James McClelland · Chen Yan · Adam Santoro · Neil Rabinowitz · Jane Wang · Feilx Hill -
2022 Poster: A data-driven approach for learning to control computers »
Peter Humphreys · David Raposo · Tobias Pohlen · Gregory Thornton · Rachita Chhaparia · Alistair Muldal · Josh Abramson · Petko Georgiev · Adam Santoro · Timothy Lillicrap -
2022 Spotlight: A data-driven approach for learning to control computers »
Peter Humphreys · David Raposo · Tobias Pohlen · Gregory Thornton · Rachita Chhaparia · Alistair Muldal · Josh Abramson · Petko Georgiev · Adam Santoro · Timothy Lillicrap -
2021 Poster: Discretization Drift in Two-Player Games »
Mihaela Rosca · Yan Wu · Benoit Dherin · David GT Barrett -
2021 Spotlight: Discretization Drift in Two-Player Games »
Mihaela Rosca · Yan Wu · Benoit Dherin · David GT Barrett -
2020 : Invited Talk: Felix Hill »
Feilx Hill -
2020 Poster: Probing Emergent Semantics in Predictive Agents via Question Answering »
Abhishek Das · Federico Carnevale · Hamza Merzic · Laura Rimell · Rosalia Schneider · Josh Abramson · Alden Hung · Arun Ahuja · Stephen Clark · Greg Wayne · Feilx Hill -
2020 Poster: An Explicitly Relational Neural Network Architecture »
Murray Shanahan · Kyriacos Nikiforou · Antonia Creswell · Christos Kaplanis · David GT Barrett · Marta Garnelo -
2019 Poster: Learning Latent Dynamics for Planning from Pixels »
Danijar Hafner · Timothy Lillicrap · Ian Fischer · Ruben Villegas · David Ha · Honglak Lee · James Davidson -
2019 Poster: Meta-Learning Neural Bloom Filters »
Jack Rae · Sergey Bartunov · Timothy Lillicrap -
2019 Oral: Meta-Learning Neural Bloom Filters »
Jack Rae · Sergey Bartunov · Timothy Lillicrap -
2019 Oral: Learning Latent Dynamics for Planning from Pixels »
Danijar Hafner · Timothy Lillicrap · Ian Fischer · Ruben Villegas · David Ha · Honglak Lee · James Davidson -
2019 Poster: Deep Compressed Sensing »
Yan Wu · Mihaela Rosca · Timothy Lillicrap -
2019 Oral: Deep Compressed Sensing »
Yan Wu · Mihaela Rosca · Timothy Lillicrap -
2019 Poster: Composing Entropic Policies using Divergence Correction »
Jonathan Hunt · Andre Barreto · Timothy Lillicrap · Nicolas Heess -
2019 Poster: An Investigation of Model-Free Planning »
Arthur Guez · Mehdi Mirza · Karol Gregor · Rishabh Kabra · Sebastien Racaniere · Theophane Weber · David Raposo · Adam Santoro · Laurent Orseau · Tom Eccles · Greg Wayne · David Silver · Timothy Lillicrap -
2019 Oral: An Investigation of Model-Free Planning »
Arthur Guez · Mehdi Mirza · Karol Gregor · Rishabh Kabra · Sebastien Racaniere · Theophane Weber · David Raposo · Adam Santoro · Laurent Orseau · Tom Eccles · Greg Wayne · David Silver · Timothy Lillicrap -
2019 Oral: Composing Entropic Policies using Divergence Correction »
Jonathan Hunt · Andre Barreto · Timothy Lillicrap · Nicolas Heess -
2018 Poster: Fast Parametric Learning with Activation Memorization »
Jack Rae · Chris Dyer · Peter Dayan · Timothy Lillicrap -
2018 Oral: Fast Parametric Learning with Activation Memorization »
Jack Rae · Chris Dyer · Peter Dayan · Timothy Lillicrap -
2017 Poster: Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study »
Samuel Ritter · David GT Barrett · Adam Santoro · Matthew Botvinick -
2017 Poster: Learning to Learn without Gradient Descent by Gradient Descent »
Yutian Chen · Matthew Hoffman · Sergio Gómez Colmenarejo · Misha Denil · Timothy Lillicrap · Matthew Botvinick · Nando de Freitas -
2017 Talk: Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study »
Samuel Ritter · David GT Barrett · Adam Santoro · Matthew Botvinick -
2017 Talk: Learning to Learn without Gradient Descent by Gradient Descent »
Yutian Chen · Matthew Hoffman · Sergio Gómez Colmenarejo · Misha Denil · Timothy Lillicrap · Matthew Botvinick · Nando de Freitas