Can local learning match self-supervised backpropagation?

While end-to-end self-supervised learning with backpropagation (global BP-SSL) has become central for training modern AI systems, theories of local self-supervised learning (local-SSL) have struggled to build functional representations in deep neural networks. To establish a link between global and local rules, we first develop a theory for deep linear networks: We identify conditions for local-SSL algorithms (like Forward-forward or CLAPP) to implement exactly the same weight update as a global BP-SSL. Starting from the theoretical insights, we then develop novel variants of local-SSL algorithms to approximate global BP-SSL in deep non-linear convolutional neural networks. Variants that improve the similarity between updates of local-SSL with those of global BP-SSL also show better performance on image datasets (CIFAR-10, STL-10, Tiny ImageNet, and ImageNet). The best local-SSL rule with the CLAPP loss function matches the performance of a comparable global BP-SSL with InfoNCE or CPC-like loss functions, and improves upon state-of-the-art for layer-wise SSL on these benchmarks.

Modern deep learning methods update the weights of neural networks by backpropagation (BP), which sends error signals from the output layer backward through the entire network. However, such a global error propagation process is unlike how the brain learns, because biological neurons update their connections using information only locally available at the connection. We investigate whether brain-like "local" learning rules, where each layer of a network is trained on its own objective, can achieve results comparable to the standard backpropagation method. We focus specifically on self-supervised learning, a method where deep neural networks learn the underlying structure of unlabeled data. Mathematically, we first identify certain ideal conditions, under which local rules implement the exact same weight updates as the global BP method. Building upon this theoretical framework, we propose two designs, direct feedback and spatial dependence, that help local learning algorithms better approximate the global BP method. When applied to training convolutional networks for image recognition tasks, these improvements also enable us to achieve a new state-of-the-art performance for local learning rules and match the performance of the BP method. Beyond evaluations of task performance, our local learning rules could also reduce computational cost and potentially be adapted to more energy-efficient neuromorphic hardware.