Timezone: »
Poster
LIT: Learned Intermediate Representation Training for Model Compression
Animesh Koratana · Daniel Kang · Peter Bailis · Matei Zaharia
Researchers have proposed a range of model compression techniques to reduce the
computational and memory footprint of deep neural networks (DNNs). In
this work, we introduce Learned Intermediate representation
Training (LIT), a novel model compression technique that outperforms a
range of recent model compression techniques by leveraging the highly repetitive
structure of modern DNNs (e.g., ResNet). LIT uses a teacher DNN to train a
student DNN of reduced depth by leveraging two key ideas: 1) LIT directly
compares intermediate representations of the teacher and student model and 2)
LIT uses the intermediate representation from the teacher model's previous block
as input to the current student block during training, improving stability of
intermediate representations in the student network. We show that LIT can
substantially reduce network size without loss in accuracy on a range of DNN
architectures and datasets. For example, LIT can compress ResNet on CIFAR10 by
3.4$\times$ outperforming network slimming and FitNets. Furthermore, LIT can
compress, by depth, ResNeXt 5.5$\times$ on CIFAR10 (image classification), VDCNN
by 1.7$\times$ on Amazon Reviews (sentiment analysis), and StarGAN by
1.8$\times$ on CelebA (style transfer, i.e., GANs).
Author Information
Animesh Koratana (Stanford University)
Daniel Kang (Stanford University)
Peter Bailis (Stanford University)
Matei Zaharia (Stanford and Databricks)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Oral: LIT: Learned Intermediate Representation Training for Model Compression »
Thu Jun 13th 07:00 -- 07:05 PM Room Hall A
More from the Same Authors
-
2019 Poster: Compressed Factorization: Fast and Accurate Low-Rank Factorization of Compressively-Sensed Data »
Vatsal Sharan · Kai Sheng Tai · Peter Bailis · Gregory Valiant -
2019 Poster: Equivariant Transformer Networks »
Kai Sheng Tai · Peter Bailis · Gregory Valiant -
2019 Oral: Equivariant Transformer Networks »
Kai Sheng Tai · Peter Bailis · Gregory Valiant -
2019 Oral: Compressed Factorization: Fast and Accurate Low-Rank Factorization of Compressively-Sensed Data »
Vatsal Sharan · Kai Sheng Tai · Peter Bailis · Gregory Valiant -
2019 Poster: Rehashing Kernel Evaluation in High Dimensions »
Paris Siminelakis · Kexin Rong · Peter Bailis · Moses Charikar · Philip Levis -
2019 Oral: Rehashing Kernel Evaluation in High Dimensions »
Paris Siminelakis · Kexin Rong · Peter Bailis · Moses Charikar · Philip Levis