Timezone: »
Existing reasoning tasks often have an important assumption that the input contents can be always accessed while reasoning, requiring unlimited storage resources and suffering from severe time delay on long sequences. To achieve efficient reasoning on long sequences with limited storage resources, memory augmented neural networks introduce a human-like write-read memory to compress and memorize the long input sequence in one pass, trying to answer subsequent queries only based on the memory. But they have two serious drawbacks: 1) they continually update the memory from current information and inevitably forget the early contents; 2) they do not distinguish what information is important and treat all contents equally. In this paper, we propose the Rehearsal Memory (RM) to enhance long-sequence memorization by self-supervised rehearsal with a history sampler. To alleviate the gradual forgetting of early information, we design self-supervised rehearsal training with recollection and familiarity tasks. Further, we design a history sampler to select informative fragments for rehearsal training, making the memory focus on the crucial information. We evaluate the performance of our rehearsal memory by the synthetic bAbI task and several downstream tasks, including text/video question answering and recommendation on long sequences.
Author Information
Zhu Zhang (DAMO Academy, Alibaba Group,)
Chang Zhou (Alibaba Group)
Jianxin Ma (Alibaba Group)
Zhijie Lin (Zhejiang University)
Jingren Zhou (Alibaba Group)
Hongxia Yang (Alibaba Group)
Zhou Zhao (Zhejiang University)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: Learning to Rehearse in Long Sequence Memorization »
Thu. Jul 22nd 01:25 -- 01:30 PM Room
More from the Same Authors
-
2023 : Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis »
Zhenhui Ye · Ziyue Jiang · Yi Ren · Jinglin Liu · Chen Zhang · Xiang Yin · Zejun MA · Zhou Zhao -
2023 Poster: Cones: Concept Neurons in Diffusion Models for Customized Generation »
Zhiheng Liu · Ruili Feng · Kai Zhu · Yifei Zhang · Kecheng Zheng · Yu Liu · Deli Zhao · Jingren Zhou · Yang Cao -
2023 Poster: RLEG: Vision-Language Representation Learning with Diffusion-based Embedding Generation »
Liming Zhao · Kecheng Zheng · Yun Zheng · Deli Zhao · Jingren Zhou -
2023 Poster: Composer: Creative and Controllable Image Synthesis with Composable Conditions »
Lianghua Huang · Di Chen · Yu Liu · Yujun Shen · Deli Zhao · Jingren Zhou -
2023 Poster: Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models »
Rongjie Huang · Jiawei Huang · Dongchao Yang · Yi Ren · Luping Liu · Mingze Li · Zhenhui Ye · Jinglin Liu · Xiang Yin · Zhou Zhao -
2023 Oral: Cones: Concept Neurons in Diffusion Models for Customized Generation »
Zhiheng Liu · Ruili Feng · Kai Zhu · Yifei Zhang · Kecheng Zheng · Yu Liu · Deli Zhao · Jingren Zhou · Yang Cao -
2023 Poster: mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video »
Haiyang Xu · Qinghao Ye · Ming Yan · Yaya Shi · Jiabo Ye · yuanhong xu · Chenliang Li · Bin Bi · Qi Qian · Wei Wang · Guohai Xu · Ji Zhang · Songfang Huang · Fei Huang · Jingren Zhou -
2022 Poster: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework »
Peng Wang · An Yang · Rui Men · Junyang Lin · Shuai Bai · Zhikang Li · Jianxin Ma · Chang Zhou · Jingren Zhou · Hongxia Yang -
2022 Spotlight: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework »
Peng Wang · An Yang · Rui Men · Junyang Lin · Shuai Bai · Zhikang Li · Jianxin Ma · Chang Zhou · Jingren Zhou · Hongxia Yang -
2022 Poster: Principled Knowledge Extrapolation with GANs »
Ruili Feng · Jie Xiao · Kecheng Zheng · Deli Zhao · Jingren Zhou · Qibin Sun · Zheng-Jun Zha -
2022 Poster: Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably) »
Yu Huang · Junyang Lin · Chang Zhou · Hongxia Yang · Longbo Huang -
2022 Spotlight: Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably) »
Yu Huang · Junyang Lin · Chang Zhou · Hongxia Yang · Longbo Huang -
2022 Spotlight: Principled Knowledge Extrapolation with GANs »
Ruili Feng · Jie Xiao · Kecheng Zheng · Deli Zhao · Jingren Zhou · Qibin Sun · Zheng-Jun Zha -
2021 Poster: Uncertainty Principles of Encoding GANs »
Ruili Feng · Zhouchen Lin · Jiapeng Zhu · Deli Zhao · Jingren Zhou · Zheng-Jun Zha -
2021 Spotlight: Uncertainty Principles of Encoding GANs »
Ruili Feng · Zhouchen Lin · Jiapeng Zhu · Deli Zhao · Jingren Zhou · Zheng-Jun Zha -
2021 Poster: KNAS: Green Neural Architecture Search »
Jingjing Xu · Liang Zhao · Junyang Lin · Rundong Gao · Xu SUN · Hongxia Yang -
2021 Spotlight: KNAS: Green Neural Architecture Search »
Jingjing Xu · Liang Zhao · Junyang Lin · Rundong Gao · Xu SUN · Hongxia Yang -
2019 Poster: Almost Unsupervised Text to Speech and Automatic Speech Recognition »
Yi Ren · Xu Tan · Tao Qin · Sheng Zhao · Zhou Zhao · Tie-Yan Liu -
2019 Oral: Almost Unsupervised Text to Speech and Automatic Speech Recognition »
Yi Ren · Xu Tan · Tao Qin · Sheng Zhao · Zhou Zhao · Tie-Yan Liu