Timezone: »
Contrastive learning is a powerful framework for learning self-supervised representations that generalize well to downstream supervised tasks. We show that multiple existing contrastive learning methods can be reinterpeted as learning a positive-definite kernel that approximates a particular contrastive kernel defined by the positive pairs. The principal components of the data under this kernel exactly correspond to the eigenfunctions of a positive-pair Markov chain, and these eigenfunctions can be used to build a representation thatprovably minimizes the worst-case approximation error of linear predictors under the assumption that positive pairs have similar labels. We give generalization bounds for downstream linear prediction using this optimal representation, and show how to approximate this representation using kernel PCA. We also explore kernel-based representations on a noisy MNIST task for which the positive pair distribution has a closed form, and compare the properties of the true eigenfunctions with their learned approximations.
Author Information
Daniel D. Johnson (Google Research / University of Toronto)
Daniel D. Johnson (Google Research / University of Toronto)
Ayoub El Hanchi (University of Toronto)
Ayoub El Hanchi (University of Toronto)
Chris Maddison (University of Toronto)
Chris Maddison (University of Toronto)
More from the Same Authors
-
2020 : (#27 / Sess. 2) Learning Graph Structure with A Finite-State Automaton Layer »
Daniel D Johnson -
2021 : Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models »
Daniel D. Johnson -
2023 Poster: R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents »
Daniel D. Johnson · Daniel Tarlow · Christian Walder -
2022 Poster: Augment with Care: Contrastive Learning for Combinatorial Problems »
Haonan Duan · Pashootan Vaezipoor · Max Paulus · Yangjun Ruan · Chris Maddison -
2022 Poster: Learning to Cut by Looking Ahead: Cutting Plane Selection via Imitation Learning »
Max Paulus · Giulia Zarpellon · Andreas Krause · Laurent Charlin · Chris Maddison -
2022 Spotlight: Augment with Care: Contrastive Learning for Combinatorial Problems »
Haonan Duan · Pashootan Vaezipoor · Max Paulus · Yangjun Ruan · Chris Maddison -
2022 Spotlight: Learning to Cut by Looking Ahead: Cutting Plane Selection via Imitation Learning »
Max Paulus · Giulia Zarpellon · Andreas Krause · Laurent Charlin · Chris Maddison -
2022 Poster: Bayesian Nonparametrics for Offline Skill Discovery »
Valentin Villecroze · Harry Braviner · Panteha Naderian · Chris Maddison · Gabriel Loaiza-Ganem -
2022 Spotlight: Bayesian Nonparametrics for Offline Skill Discovery »
Valentin Villecroze · Harry Braviner · Panteha Naderian · Chris Maddison · Gabriel Loaiza-Ganem -
2022 Poster: Stochastic Reweighted Gradient Descent »
Ayoub El Hanchi · David Stephens · Chris Maddison -
2022 Spotlight: Stochastic Reweighted Gradient Descent »
Ayoub El Hanchi · David Stephens · Chris Maddison -
2021 Poster: Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding »
Yangjun Ruan · Karen Ullrich · Daniel Severo · James Townsend · Ashish Khisti · Arnaud Doucet · Alireza Makhzani · Chris Maddison -
2021 Oral: Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding »
Yangjun Ruan · Karen Ullrich · Daniel Severo · James Townsend · Ashish Khisti · Arnaud Doucet · Alireza Makhzani · Chris Maddison -
2021 Poster: Oops I Took A Gradient: Scalable Sampling for Discrete Distributions »
Will Grathwohl · Kevin Swersky · Milad Hashemi · David Duvenaud · Chris Maddison -
2021 Oral: Oops I Took A Gradient: Scalable Sampling for Discrete Distributions »
Will Grathwohl · Kevin Swersky · Milad Hashemi · David Duvenaud · Chris Maddison -
2020 : Q&A: Chris Maddison »
Chris Maddison · Jessica Forde · Jesse Dodge -
2020 : Invited Talk: Chris Maddison »
Chris Maddison