THE CONFERENCE
FOR AUTHORS
FOR PARTICIPANTS

 
Poster
Decoupled Neural Interfaces using Synthetic Gradients
Max Jaderberg · Wojciech Czarnecki · Simon Osindero · Oriol Vinyals · Alex Graves · David Silver · koray kavukcuoglu

Mon Aug 7th 06:30 -- 10:00 PM @ Gallery #1
in Posters Mon »

Training directed neural networks typically requires forward-propagating data through a computation graph, followed by backpropagating error signal, to produce weight updates. All layers, or more generally, modules, of the network are therefore locked, in the sense that they must wait for the remainder of the network to execute forwards and propagate error backwards before they can be updated. In this work we break this constraint by decoupling modules by introducing a model of the future computation of the network graph. These models predict what the result of the modelled subgraph will produce using only local information. In particular we focus on modelling error gradients: by using the modelled \emph{synthetic gradient} in place of true backpropagated error gradients we decouple subgraphs, and can update them independently and asynchronously \ie~we realise \emph{decoupled neural interfaces}. We show results for feed-forward models, where every layer is trained asynchronously, recurrent neural networks (RNNs) where predicting one's future gradient extends the time over which the RNN can effectively model, and also a hierarchical RNN system with ticking at different timescales. Finally, we demonstrate that in addition to predicting gradients, the same framework can be used to predict inputs, resulting in models which are decoupled in both the forward and backwards pass -- amounting to independent networks which co-learn such that they can be composed into a single functioning corporation.

Author Information

Max Jaderberg (DeepMind)
Wojciech Czarnecki (DeepMind)
Simon Osindero (DeepMind)
Oriol Vinyals (DeepMind)

Oriol Vinyals is a Research Scientist at Google. He works in deep learning with the Google Brain team. Oriol holds a Ph.D. in EECS from University of California, Berkeley, and a Masters degree from University of California, San Diego. He is a recipient of the 2011 Microsoft Research PhD Fellowship. He was an early adopter of the new deep learning wave at Berkeley, and in his thesis he focused on non-convex optimization and recurrent neural networks. At Google Brain he continues working on his areas of interest, which include artificial intelligence, with particular emphasis on machine learning, language, and vision.

Alex Graves (DeepMind)
David Silver (Google DeepMind)
koray kavukcuoglu (DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors