Timezone: »

 
Poster
Towards Scaling Difference Target Propagation by Learning Backprop Targets
Maxence ERNOULT · Fabrice Normandin · Abhinav Moudgil · Sean Spinney · Eugene Belilovsky · Irina Rish · Blake Richards · Yoshua Bengio

Tue Jul 19 03:30 PM -- 05:30 PM (PDT) @ Hall E #237
in Poster Session 1 »

The development of biologically-plausible learning algorithms is important for understanding learning in the brain, but most of them fail to scale-up to real-world tasks, limiting their potential as explanations for learning by real brains. As such, it is important to explore learning algorithms that come with strong theoretical guarantees and can match the performance of backpropagation (BP) on complex tasks. One such algorithm is Difference Target Propagation (DTP), a biologically-plausible learning algorithm whose close relation with Gauss-Newton (GN) optimization has been recently established. However, the conditions under which this connection rigorously holds preclude layer-wise training of the feedback pathway synaptic weights (which is more biologically plausible). Moreover, good alignment between DTP weight updates and loss gradients is only loosely guaranteed and under very specific conditions for the architecture being trained. In this paper, we propose a novel feedback weight training scheme that ensures both that DTP approximates BP and that layer-wise feedback weight training can be restored without sacrificing any theoretical guarantees. Our theory is corroborated by experimental results and we report the best performance ever achieved by DTP on CIFAR-10 and ImageNet 32x32.

Keywords: APP: Neuroscience, Cognitive Science

Author Information

Maxence ERNOULT (IBM Research)
Maxence ERNOULT

Ecole Polytechnique (2012-2015) University of Cambridge (2015-2016) PhD in Neuromorphic Computing at Université Paris-Saclay (2017-2020) Remote internship with Mila (2020-2021) Research scientist at IBM Research (2021-)

Fabrice Normandin (Mila)
Abhinav Moudgil (Mila, Concordia University)
Sean Spinney (Mila)
Eugene Belilovsky (Mila)
Irina Rish (MILA / Université de Montréal h)
Blake Richards (Mila)
Yoshua Bengio (Mila - Quebec AI Institute)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors