ePC: Fast and Deep Predictive Coding in Digital Simulation

Predictive Coding (PC) offers a brain-inspired alternative to backpropagation for neural network training, described as a physical system minimizing its internal energy. While ideally suited for analog implementation, such hardware does not exist yet, and thus, in practice, PC is predominantly digitally simulated, requiring excessive amounts of compute while struggling to scale to deeper architectures. This paper reformulates PC to overcome this hardware-algorithm mismatch. First, we uncover how the canonical state-based formulation of PC (sPC) is, by design, deeply inefficient in digital simulation, inevitably resulting in exponential signal decay that stalls the entire numerical process. Then, to overcome this fundamental limitation, we introduce error-based PC (ePC), a novel reparameterization of PC which does not suffer from signal decay. Though no longer directly implementable in analog, ePC numerically computes exact PC weights gradients and runs orders of magnitude faster than sPC. Experiments across multiple architectures and datasets demonstrate that ePC matches backpropagation's performance even for deeper models where sPC struggles. Besides practical improvements, our work provides theoretical insight into PC dynamics and establishes a foundation for scaling PC-based learning to deeper architectures in digital simulation and beyond.

Predictive Coding (PC) is a promising learning algorithm that could lead to fast, ultra-energy-efficient AI hardware. Unfortunately, such hardware does not exist yet, and instead, researchers must use digital simulations of PC -- which are horribly slow. Moreover, a recent paper noted poor performance on deeper networks, casting doubts on PC's capabilities. In our work, we uncover why PC is so slow in simulation and how this directly affects test performance in common setups. Then, we introduce a simple solution to the problem: a mathematical reframing that allows PC to be simulated 100-1000x faster and with great performance, even for deep networks. Our work not only proves that PC is indeed a capable learning algorithm, but it also provides a practical tool to substantially speed up PC research and future hardware development.