Timezone: »
Poster
The SSL Interplay: Augmentations, Inductive Bias, and Generalization
Vivien Cabannnes · Bobak T Kiani · Randall Balestriero · Yann LeCun · Alberto Bietti
Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision. Yet in practice, engineers face issues such as instability in tuning optimizers and collapse of representations during training. Such challenges motivate the need for a theory to shed light on the complex interplay between the choice of data augmentation, network architecture, and training algorithm. % on the resulting performance in downstream tasks. We study such an interplay with a precise analysis of generalization performance on both pretraining and downstream tasks in kernel regimes, and highlight several insights for SSL practitioners that arise from our theory.
Author Information
Vivien Cabannnes (Meta AI)
Bobak T Kiani (MIT)
Randall Balestriero (Rice University)
Yann LeCun (New York University)
Alberto Bietti (Meta AI)
More from the Same Authors
-
2022 : Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior »
Ravid Shwartz-Ziv · Micah Goldblum · Hossein Souri · Sanyam Kapoor · Chen Zhu · Yann LeCun · Andrew Wilson -
2022 : What Do We Maximize In Self-Supervised Learning? »
Ravid Shwartz-Ziv · Ravid Shwartz-Ziv · Randall Balestriero · Yann LeCun · Yann LeCun -
2023 : Hessian Inertia in Neural Networks »
Xuchan Bao · Alberto Bietti · Aaron Defazio · Vivien Cabannnes -
2023 : Provable Instance Specific Robustness via Linear Constraints »
Ahmed Imtiaz Humayun · Josue Casco-Rodriguez · Randall Balestriero · Richard Baraniuk -
2023 : Understanding the Detrimental Class-level Effects of Data Augmentation »
Polina Kirichenko · Mark Ibrahim · Randall Balestriero · Diane Bouchacourt · Ramakrishna Vedantam · Hamed Firooz · Andrew Wilson -
2023 Oral: Equivariant Polynomials for Graph Neural Networks »
Omri Puny · Derek Lim · Bobak T Kiani · Haggai Maron · Yaron Lipman -
2023 Poster: RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank »
Quentin Garrido · Randall Balestriero · Laurent Najman · Yann LeCun -
2023 Poster: Equivariant Polynomials for Graph Neural Networks »
Omri Puny · Derek Lim · Bobak T Kiani · Haggai Maron · Yaron Lipman -
2023 Oral: RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank »
Quentin Garrido · Randall Balestriero · Laurent Najman · Yann LeCun -
2023 Poster: Self-supervised learning of Split Invariant Equivariant representations »
Quentin Garrido · Laurent Najman · Yann LeCun -
2023 Poster: A Generalization of ViT/MLP-Mixer to Graphs »
Xiaoxin He · Bryan Hooi · Thomas Laurent · Adam Perold · Yann LeCun · Xavier Bresson -
2022 : Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior »
Ravid Shwartz-Ziv · Micah Goldblum · Hossein Souri · Sanyam Kapoor · Chen Zhu · Yann LeCun · Andrew Wilson -
2022 Poster: Implicit Bias of Linear Equivariant Networks »
Hannah Lawrence · Bobak T Kiani · Kristian Georgiev · Andrew Dienes -
2022 Spotlight: Implicit Bias of Linear Equivariant Networks »
Hannah Lawrence · Bobak T Kiani · Kristian Georgiev · Andrew Dienes -
2021 Poster: Adversarial Robustness Guarantees for Random Deep Neural Networks »
Giacomo De Palma · Bobak T Kiani · Seth Lloyd -
2021 Poster: Disambiguation of Weak Supervision leading to Exponential Convergence rates »
Vivien Cabannnes · Francis Bach · Alessandro Rudi -
2021 Spotlight: Disambiguation of Weak Supervision leading to Exponential Convergence rates »
Vivien Cabannnes · Francis Bach · Alessandro Rudi -
2021 Spotlight: Adversarial Robustness Guarantees for Random Deep Neural Networks »
Giacomo De Palma · Bobak T Kiani · Seth Lloyd -
2020 Poster: Structured Prediction with Partial Labelling through the Infimum Loss »
Vivien Cabannnes · Alessandro Rudi · Francis Bach -
2018 Poster: Adversarially Regularized Autoencoders »
Jake Zhao · Yoon Kim · Kelly Zhang · Alexander Rush · Yann LeCun -
2018 Oral: Adversarially Regularized Autoencoders »
Jake Zhao · Yoon Kim · Kelly Zhang · Alexander Rush · Yann LeCun -
2018 Poster: A Spline Theory of Deep Learning »
Randall Balestriero · Richard Baraniuk -
2018 Oral: A Spline Theory of Deep Learning »
Randall Balestriero · Richard Baraniuk -
2018 Poster: Comparing Dynamics: Deep Neural Networks versus Glassy Systems »
Marco Baity-Jesi · Levent Sagun · Mario Geiger · Stefano Spigler · Gerard Arous · Chiara Cammarota · Yann LeCun · Matthieu Wyart · Giulio Biroli -
2018 Oral: Comparing Dynamics: Deep Neural Networks versus Glassy Systems »
Marco Baity-Jesi · Levent Sagun · Mario Geiger · Stefano Spigler · Gerard Arous · Chiara Cammarota · Yann LeCun · Matthieu Wyart · Giulio Biroli -
2017 Poster: Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs »
Li Jing · Yichen Shen · Tena Dubcek · John E Peurifoy · Scott Skirlo · Yann LeCun · Max Tegmark · Marin Soljačić -
2017 Talk: Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs »
Li Jing · Yichen Shen · Tena Dubcek · John E Peurifoy · Scott Skirlo · Yann LeCun · Max Tegmark · Marin Soljačić