THE CONFERENCE
FOR AUTHORS
FOR PARTICIPANTS

Timezone: »

 
Talk
Efficient softmax approximation for GPUs
Edouard Grave · Armand Joulin · Moustapha Cisse · David Grangier · Herve Jegou

Tue Aug 08 06:42 PM -- 07:00 PM (PDT) @ Darling Harbour Theatre
in Deep learning 8: hardware »

We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computation time. Our approach further reduces the computational cost by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax. The code of our method is available at https://github.com/facebookresearch/adaptive-softmax.

[ PDF]  [ ss Summary/Notes]  [ Video] 

Author Information

Edouard Grave (Facebook AI Research)
Armand Joulin (Facebook)
Moustapha Cisse
David Grangier (Facebook)
Herve Jegou (Facebook AI Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors