Exploring the Gap between Collapsed & Whitened Features in Self-Supervised Learning

Bobby He · Mete Ozay

Hall E #516

Keywords: [ MISC: Representation Learning ] [ DL: Robustness ] [ DL: Other Representation Learning ] [ DL: Everything Else ] [ MISC: Unsupervised and Semi-supervised Learning ] [ DL: Algorithms ] [ DL: Self-Supervised Learning ]

Abstract: Avoiding feature collapse, when a Neural Network (NN) encoder maps all inputs to a constant vector, is a shared implicit desideratum of various methodological advances in self-supervised learning (SSL). To that end, whitened features have been proposed as an explicit objective to ensure uncollapsed features \cite{zbontar2021barlow,ermolov2021whitening,hua2021feature,bardes2022vicreg}. We identify power law behaviour in eigenvalue decay, parameterised by exponent $\beta{\geq}0$, as a spectrum that bridges between the collapsed & whitened feature extremes. We provide theoretical & empirical evidence highlighting the factors in SSL, like projection layers & regularisation strength, that influence eigenvalue decay rate, & demonstrate that the degree of feature whitening affects generalisation, particularly in label scarce regimes. We use our insights to motivate a novel method, PMP (PostMan-Pat), which efficiently post-processes a pretrained encoder to enforce eigenvalue decay rate with power law exponent $\beta$, & find that PostMan-Pat delivers improved label efficiency and transferability across a range of SSL methods and encoder architectures.

Chat is not available.