Towards the Training of Deeper Predictive Coding Neural Networks

Predictive coding networks are neural models that perform inference through an iterative energy minimization process, whose operations are local in space and time. While effective in shallow architectures, they suffer significant performance degradation beyond five to seven layers. In this work, we show that this degradation is caused by exponentially imbalanced errors between layers during weight updates, and by predictions from the previous layers not being effective in guiding updates in deeper layers. Furthermore, when training models with skip connections, the energy propagated by the residuals reaches higher layers faster than that propagated by the main pathway, affecting test accuracy. We address the first issue by introducing a novel precision-weighted optimization of latent variables that balances error distributions during the relaxation phase, the second issue by proposing a novel weight update mechanism that reduces error accumulation in deeper layers, and the third one by using auxiliary neurons that slow down the propagation of the energy in the residual connections. Empirically, our methods achieve performance comparable to backpropagation on deep models such as ResNets, opening new possibilities for predictive coding in complex tasks.

Brain-inspired AI models, known as predictive coding networks, are highly efficient because their components only need to talk to their immediate neighbors to learn. However, when we try to stack these components to make deep networks—which are essential for handling complex, real-world data—their performance plummets. We discovered that as these networks grow deeper, the learning signals become wildly unbalanced between layers, and the crucial guiding information gets lost or mixed up along the way. To fix this, we developed novel mechanisms: a balancing system that ensures all layers receive an equal share of learning signals, an updated communication method that prevents errors from piling up in the deepest parts of the network, and "traffic-controlling" neurons that prevent shorthand shortcuts (residuals) from overwhelming the main learning pathway. Together, these fixes allow deep brain-inspired networks to match the accuracy of traditional, industry-standard AI training methods (like backpropagation) for the first time on complex image tasks. This unlocks new possibilities for developing powerful AI that operates more like the human brain.