Timezone: »
Classical distillation methods transfer representations from a teacher'' neural network to a
student'' network by matching their output activations. Recent methods also match the Jacobians, or the gradient of output activations with the input. However, this involves making some ad hoc decisions, in particular, the choice of the loss function. In this paper, we first establish an equivalence between Jacobian matching and distillation with input noise, from which we derive appropriate loss functions for Jacobian matching. We then rely on this analysis to apply Jacobian matching to transfer learning by establishing equivalence of a recent transfer learning procedure to distillation. We then show experimentally on standard image datasets that Jacobian-based penalties improve distillation, robustness to noisy inputs, and transfer learning.
Author Information
Suraj Srinivas (Idiap)
Francois Fleuret (Idiap research institute)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Oral: Knowledge Transfer with Jacobian Matching »
Thu Jul 12th 03:50 -- 04:00 PM Room Victoria
More from the Same Authors
-
2019 Poster: Processing Megapixel Images with Deep Attention-Sampling Models »
Angelos Katharopoulos · Francois Fleuret -
2019 Oral: Processing Megapixel Images with Deep Attention-Sampling Models »
Angelos Katharopoulos · Francois Fleuret -
2018 Poster: Geodesic Convolutional Shape Optimization »
Pierre Baque · Edoardo Remelli · Francois Fleuret · EPFL Pascal Fua -
2018 Oral: Geodesic Convolutional Shape Optimization »
Pierre Baque · Edoardo Remelli · Francois Fleuret · EPFL Pascal Fua -
2018 Poster: Kronecker Recurrent Units »
Cijo Jose · Mouhamadou Moustapha Cisse · Francois Fleuret -
2018 Poster: Not All Samples Are Created Equal: Deep Learning with Importance Sampling »
Angelos Katharopoulos · Francois Fleuret -
2018 Oral: Not All Samples Are Created Equal: Deep Learning with Importance Sampling »
Angelos Katharopoulos · Francois Fleuret -
2018 Oral: Kronecker Recurrent Units »
Cijo Jose · Mouhamadou Moustapha Cisse · Francois Fleuret