Training Deep Spiking Neural Networks without Normalization

The training of deep Spiking Neural Networks (SNNs) has traditionally relied on Batch Normalization (BN), which stabilizes input currents and gradients during training. However, BN is not a universal solution. It is unsuitable for variable-length tasks and scenarios with reduced batch size, constraining the development of deep SNNs, where removing BN typically causes the training to fail to converge. This dependence stems not from a fundamental necessity of BN but from the current lack of reasonable initialization methods for SNNs. This paper addresses this core limitation by proposing SpikeInit, a novel initialization framework for SNNs. By modeling the response curve and gradient of spiking layers, SpikeInit initializes the weights and shape parameters of surrogate gradients to maintain stable firing rates during forward propagation and stable gradient magnitudes during backpropagation. Extensive experiments demonstrate that deep SNNs with SpikeInit can be trained stably without normalization and achieve superior performance compared to their normalized counterparts under identical settings. Furthermore, we demonstrate the scalability of SpikeInit by successfully training an ultra-deep, 1000-layer SNN without normalization. Our work provides a foundational step toward large-scale normalization-free SNN, liberating SNN design from the constraints of normalization.