On the Infinite Width and Depth Limits of Predictive Coding Networks

Predictive coding (PC) is a biologically plausible alternative to standard backpropagation (BP) that minimises an energy function with respect to network activities before updating weights. Recent work has improved the training stability of deep PC networks (PCNs) by leveraging some BP-inspired reparameterisations. However, the full scalability and theoretical basis of these methods remain unclear. To address this gap, we study the infinite width and depth limits of PCNs. For linear residual networks, we show that the set of width- and depth-stable feature-learning parameterisations for PC is exactly the same as for BP. Moreover, under any of these parameterisations, the PC energy with equilibrated activities converges to the quadratic BP loss when the model width is much larger than the depth, resulting in PC computing the same gradients as BP. Experiments show that, as long as an activity equilibrium is reached, convergence to BP holds for nonlinear models including convolutional networks and transformers. Overall, this work constrains the types of parameterisation that are scalable with PC, while showing a way in which BP can be effectively implemented with only local updates in much wider than deep networks like the brain.

Modern AI systems like ChatGPT are built on neural networks—structures of interconnected artificial neurons arranged in layers, loosely inspired by the biological brain. Currently, these networks are trained using an algorithm called “backpropagation”. While highly effective, backpropagation requires distant neurons to communicate with one another, which not only doesn’t align with how the brain actually learns, but is also incredibly energy-inefficient. A alternative algorithm called “predictive coding” is much more brain-like because it only updates connections based on the activity of neighboring neurons. However, a major question remains: can predictive coding scale up to match the performance of massive modern AI models? We answer this question by mathematically analysing what happens when predictive coding networks become incredibly wide (many neurons per layer) and deep (many layers). We show that when a predictive coding network is designed to be much wider than it is deep—closely mirroring the actual structural proportions of the human brain—it updates its connections in the exact same way as backpropagation. This work suggests how the brain could learn effectively using only local updates, while contributing to the development of scalable, energy-efficient AI.