Layerwise Hebbian/anti-Hebbian (HaH) Learning In Deep Networks: A Neuro-inspired Approach To Robustness
Metehan Cekic · Can Bakiskan · Upamanyu Madhow
We propose a neuro-inspired approach for engineering robustness into deep neural networks (DNNs), in which end-to-end cost functions are supplemented with layer-wise costs promoting Hebbian (“fire together,” “wire together”) updates for highly active neurons, and anti-Hebbian updates for the remaining neurons. Unlike standard end-to-end training, which does not directly exert control over the features extracted at intermediate layers, Hebbian/anti-Hebbian (HaH) learning is aimed at producing sparse, strong activations which are more difficult to corrupt. We further encourage sparsity by introducing competition between neurons via divisive normalization and thresholding, together with implicit $\ell_2$ normalization of neuronal weights, instead of batch norm. Preliminary CIFAR-10 experiments demonstrate that our neuro-inspired model, trained without augmentation by noise or adversarial perturbations, is substantially more robust to a range of corruptions than a baseline end-to-end trained model. This opens up exciting research frontiers for training robust DNNs, with layer-wise costs providing a strategy complementary to that of data-augmented end-to-end training.
Metehan Cekic (University of California, Santa Barbara)
I am currently a Ph.D. candidate, working with Prof. Upamanyu Madhow in the Electrical and Computer Engineering department at UCSB. Through my research experience, I have developed an interest in Deep Learning and its applications. I received my B.S. degrees in electrical & electronics engineering and physics from Bogazici University, Istanbul, Turkey in 2017.
Can Bakiskan (University of California, Santa Barbara)
Upamanyu Madhow (University of California, Santa Barbara)
More from the Same Authors
2022 : Early Layers Are More Important For Adversarial Robustness »
Can Bakiskan · Metehan Cekic · Upamanyu Madhow
2022 : Dynamic Positive Reinforcement For Long-Term Fairness »
Bhagyashree Puranik · Upamanyu Madhow · Ramtin Pedarsani