Timezone: »
Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and they can be trained and sampled from via well-established statistical tools, such as MCMC. Neural networks may be used as energy function approximators, providing both a rich class of expressive models as well as a flexible device to incorporate data structure. In this work we focus on shallow neural networks. Building from the incipient theory of overparametrized neural networks, we show that models trained in the so-called 'active' regime provide a statistical advantage over their associated 'lazy' or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Our study covers both the maximum likelihood and Stein Discrepancy estimators, and we validate our theoretical results with numerical experiments on synthetic data.
Author Information
Carles Domingo-Enrich (NYU)
Alberto Bietti (NYU)
Eric Vanden-Eijnden (New York University)
Joan Bruna (New York University)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Oral: On Energy-Based Models with Overparametrized Shallow Neural Networks »
Wed. Jul 21st 12:00 -- 12:20 PM Room
More from the Same Authors
-
2022 Poster: Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning »
Alberto Bietti · Chen-Yu Wei · Miroslav Dudik · John Langford · Steven Wu -
2022 Poster: Extended Unconstrained Features Model for Exploring Deep Neural Collapse »
Tom Tirer · Joan Bruna -
2022 Spotlight: Extended Unconstrained Features Model for Exploring Deep Neural Collapse »
Tom Tirer · Joan Bruna -
2022 Spotlight: Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning »
Alberto Bietti · Chen-Yu Wei · Miroslav Dudik · John Langford · Steven Wu -
2021 Workshop: ICML Workshop on Representation Learning for Finance and E-Commerce Applications »
Senthil Kumar · Sameena Shah · Joan Bruna · Tom Goldstein · Erik Mueller · Oleg Rokhlenko · Hongxia Yang · Jianpeng Xu · Oluwatobi O Olabiyi · Charese Smiley · C. Bayan Bruss · Saurabh H Nagrecha · Svitlana Vyetrenko -
2021 Poster: Offline Contextual Bandits with Overparameterized Models »
David Brandfonbrener · William Whitney · Rajesh Ranganath · Joan Bruna -
2021 Poster: A Functional Perspective on Learning Symmetric Functions with Neural Networks »
Aaron Zweig · Joan Bruna -
2021 Spotlight: A Functional Perspective on Learning Symmetric Functions with Neural Networks »
Aaron Zweig · Joan Bruna -
2021 Spotlight: Offline Contextual Bandits with Overparameterized Models »
David Brandfonbrener · William Whitney · Rajesh Ranganath · Joan Bruna -
2020 Poster: Extra-gradient with player sampling for faster convergence in n-player games »
Samy Jelassi · Carles Domingo-Enrich · Damien Scieur · Arthur Mensch · Joan Bruna -
2019 Workshop: Theoretical Physics for Deep Learning »
Jaehoon Lee · Jeffrey Pennington · Yasaman Bahri · Max Welling · Surya Ganguli · Joan Bruna -
2019 : Opening Remarks »
Jaehoon Lee · Jeffrey Pennington · Yasaman Bahri · Max Welling · Surya Ganguli · Joan Bruna -
2019 Poster: Neuron birth-death dynamics accelerates gradient descent and converges asymptotically »
Grant Rotskoff · Samy Jelassi · Joan Bruna · Eric Vanden-Eijnden -
2019 Oral: Neuron birth-death dynamics accelerates gradient descent and converges asymptotically »
Grant Rotskoff · Samy Jelassi · Joan Bruna · Eric Vanden-Eijnden -
2019 Poster: Approximating Orthogonal Matrices with Effective Givens Factorization »
Thomas Frerix · Joan Bruna -
2019 Oral: Approximating Orthogonal Matrices with Effective Givens Factorization »
Thomas Frerix · Joan Bruna