Training Deep Spiking Neural Networks without Normalization

The training of deep Spiking Neural Networks (SNNs) has traditionally relied on Batch Normalization (BN), which stabilizes input currents and gradients during training. However, BN is not a universal solution. It is unsuitable for variable-length tasks and scenarios with reduced batch size, constraining the development of deep SNNs, where removing BN typically causes the training to fail to converge. This dependence stems not from a fundamental necessity of BN but from the current lack of reasonable initialization methods for SNNs. This paper addresses this core limitation by proposing SpikeInit, a novel initialization framework for SNNs. By modeling the response curve and gradient of spiking layers, SpikeInit initializes the weights and shape parameters of surrogate gradients to maintain stable firing rates during forward propagation and stable gradient magnitudes during backpropagation. Extensive experiments demonstrate that deep SNNs with SpikeInit can be trained stably without normalization and achieve superior performance compared to their normalized counterparts under identical settings. Furthermore, we demonstrate the scalability of SpikeInit by successfully training an ultra-deep, 1000-layer SNN without normalization. Our work provides a foundational step toward large-scale normalization-free SNN, liberating SNN design from the constraints of normalization.

Training brain-inspired deep Spiking Neural Networks (SNNs) is difficult because they often rely on normalization methods, such as Batch Normalization, to stabilize training. These methods are useful but not suitable for tasks with variable input length or small batch sizes, which limits where deep SNNs can be used. This paper shows that the problem is not that deep SNNs fundamentally need normalization, but that they need a better initialization. We developed SpikeInit, a method that carefully initializes both the network weights and the learning signals so that spikes and gradients remain stable across many layers. With SpikeInit, deep spiking neural networks can be trained without normalization and can even outperform normalized versions on various tasks. The method also scales to extremely deep models, including 1000-layer SNNs. This work makes deep SNNs more flexible and may help bring their energy-efficient computing advantages to a wider range of real-world AI applications.