Timezone: »
Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia. However, our world is dynamic, evolves over time, and our models' knowledge becomes outdated. To study how semi-parametric QA models and their underlying parametric language models (LMs) adapt to evolving knowledge, we construct a new large-scale dataset, StreamingQA, with human written and generated questions asked on a given date, to be answered from 14 years of time-stamped news articles. We evaluate our models quarterly as they read new articles not seen in pre-training. We show that parametric models can be updated without full retraining, while avoiding catastrophic forgetting. For semi-parametric models, adding new articles into the search space allows for rapid adaptation, however, models with an outdated underlying LM under-perform those with a retrained LM. For questions about higher-frequency named entities, parametric updates are particularly beneficial. In our dynamic world, the StreamingQA dataset enables a more realistic evaluation of QA models, and our experiments highlight several promising directions for future research.
Author Information
Adam Liska (DeepMind)
Tomas Kocisky (DeepMind)
Elena Gribovskaya (DeepMind)
Tayfun Terzi (DeepMind)
Eren Sezener (DeepMind)
Devang Agrawal (DeepMind)
Cyprien de Masson d'Autume (DeepMind)
Tim Scholtes (DeepMind)
Manzil Zaheer (Google Research)
Susannah Young (DeepMind)
Ellen Gilsenan-McMahon (DeepMind)
Sophia Austin (DeepMind)
Phil Blunsom (DeepMind and Oxford University)
Angeliki Lazaridou (DeepMind)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Spotlight: StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models »
Tue. Jul 19th 03:30 -- 03:35 PM Room Room 301 - 303
More from the Same Authors
-
2021 : CoBERL: Contrastive BERT for Reinforcement Learning »
Andrea Banino · AdriĆ Puigdomenech Badia · Jacob C Walker · Tim Scholtes · Jovana Mitrovic · Charles Blundell -
2022 Poster: Towards Coherent and Consistent Use of Entities in Narrative Generation »
Pinelopi Papalampidi · Kris Cao · Tomas Kocisky -
2022 Poster: Deep Hierarchy in Bandits »
Joey Hong · Branislav Kveton · Sumeet Katariya · Manzil Zaheer · Mohammad Ghavamzadeh -
2022 Poster: A Context-Integrated Transformer-Based Neural Network for Auction Design »
Zhijian Duan · Jingwu Tang · Yutong Yin · Zhe Feng · Xiang Yan · Manzil Zaheer · Xiaotie Deng -
2022 Spotlight: Deep Hierarchy in Bandits »
Joey Hong · Branislav Kveton · Sumeet Katariya · Manzil Zaheer · Mohammad Ghavamzadeh -
2022 Spotlight: A Context-Integrated Transformer-Based Neural Network for Auction Design »
Zhijian Duan · Jingwu Tang · Yutong Yin · Zhe Feng · Xiang Yan · Manzil Zaheer · Xiaotie Deng -
2022 Spotlight: Towards Coherent and Consistent Use of Entities in Narrative Generation »
Pinelopi Papalampidi · Kris Cao · Tomas Kocisky -
2022 Poster: Private Adaptive Optimization with Side information »
Tian Li · Manzil Zaheer · Sashank Jakkam Reddi · Virginia Smith -
2022 Poster: Robust Training of Neural Networks Using Scale Invariant Architectures »
Zhiyuan Li · Srinadh Bhojanapalli · Manzil Zaheer · Sashank Jakkam Reddi · Sanjiv Kumar -
2022 Spotlight: Private Adaptive Optimization with Side information »
Tian Li · Manzil Zaheer · Sashank Jakkam Reddi · Virginia Smith -
2022 Oral: Robust Training of Neural Networks Using Scale Invariant Architectures »
Zhiyuan Li · Srinadh Bhojanapalli · Manzil Zaheer · Sashank Jakkam Reddi · Sanjiv Kumar -
2022 Poster: Knowledge Base Question Answering by Case-based Reasoning over Subgraphs »
Rajarshi Das · Ameya Godbole · Ankita Rajaram Naik · Elliot Tower · Manzil Zaheer · Hannaneh Hajishirzi · Robin Jia · Andrew McCallum -
2022 Spotlight: Knowledge Base Question Answering by Case-based Reasoning over Subgraphs »
Rajarshi Das · Ameya Godbole · Ankita Rajaram Naik · Elliot Tower · Manzil Zaheer · Hannaneh Hajishirzi · Robin Jia · Andrew McCallum -
2021 Poster: Meta-Thompson Sampling »
Branislav Kveton · Mikhail Konobeev · Manzil Zaheer · Chih-wei Hsu · Martin Mladenov · Craig Boutilier · Csaba Szepesvari -
2021 Spotlight: Meta-Thompson Sampling »
Branislav Kveton · Mikhail Konobeev · Manzil Zaheer · Chih-wei Hsu · Martin Mladenov · Craig Boutilier · Csaba Szepesvari -
2021 Poster: Latent Programmer: Discrete Latent Codes for Program Synthesis »
Joey Hong · David Dohan · Rishabh Singh · Charles Sutton · Manzil Zaheer -
2021 Oral: Latent Programmer: Discrete Latent Codes for Program Synthesis »
Joey Hong · David Dohan · Rishabh Singh · Charles Sutton · Manzil Zaheer -
2021 Poster: Federated Composite Optimization »
Honglin Yuan · Manzil Zaheer · Sashank Jakkam Reddi -
2021 Spotlight: Federated Composite Optimization »
Honglin Yuan · Manzil Zaheer · Sashank Jakkam Reddi -
2020 : Invited Talk: Angeliki Lazaridou »
Angeliki Lazaridou -
2019 : Multi-agent communication from raw perceptual input: what works, what doesn't and what's next »
Angeliki Lazaridou -
2019 Poster: Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning »
Natasha Jaques · Angeliki Lazaridou · Edward Hughes · Caglar Gulcehre · Pedro Ortega · DJ Strouse · Joel Z Leibo · Nando de Freitas -
2019 Oral: Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning »
Natasha Jaques · Angeliki Lazaridou · Edward Hughes · Caglar Gulcehre · Pedro Ortega · DJ Strouse · Joel Z Leibo · Nando de Freitas -
2017 Workshop: Learning to Generate Natural Language »
Yishu Miao · Wang Ling · Tsung-Hsien Wen · Kris Cao · Daniela Gerz · Phil Blunsom · Chris Dyer