Timezone: »

 
Spotlight
Bregman Neural Networks
Jordan Frecon · Gilles Gasso · Massimiliano Pontil · Saverio Salzo

Thu Jul 21 12:30 PM -- 12:35 PM (PDT) @ Ballroom 1 & 2
in Deep Learning »

We present a framework based on bilevel optimization for learning multilayer, deep data representations. On the one hand, the lower-level problem finds a representation by successively minimizing layer-wise objectives made of the sum of a prescribed regularizer as well as a fidelity term and some linear function both depending on the representation found at the previous layer. On the other hand, the upper-level problem optimizes over the linear functions to yield a linearly separable final representation. We show that, by choosing the fidelity term as the quadratic distance between two successive layer-wise representations, the bilevel problem reduces to the training of a feed-forward neural network. Instead, by elaborating on Bregman distances, we devise a novel neural network architecture additionally involving the inverse of the activation function reminiscent of the skip connection used in ResNets. Numerical experiments suggest that the proposed Bregman variant benefits from better learning properties and more robust prediction performance.

Keywords: DL: Other Representation Learning · OPT: Bilevel optimization · OPT: Convex · MISC: Representation Learning · MISC: Supervised Learning

Author Information

Jordan Frecon (INSA Rouen)
Gilles Gasso (INSA Rouen)
Massimiliano Pontil ( Istituto Italiano di Tecnologia & University College London)
Saverio Salzo (Istituto Italiano di Tecnologia)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors