UniSparse: Combining Weight Pruning and Spike Sparsification in Spiking Neural Networks

Spiking Neural Networks (SNNs) offer a notable energy-saving advantage compared to Artificial Neural Networks (ANNs) when deployed on neuromorphic hardware. While recent SNNs achieve superior performance using larger and deeper backbones, this comes at a cost of diminishing their energy-saving benefits. In this paper, we propose UniSparse, a unified sparsification framework for enhancing the energy efficiency of SNNs. We demonstrate that the affine parameters in batch normalization also serve as the learnable threshold of its subsequent spiking neurons. Based on this, we propose a novel spike sparsification method that reduces firing rate by constraining the affine parameters. As a complement to spike sparsification, we propose a weight pruning method based on the same energy constraint, which can be naturally integrated with spike sparsification. Experimental results demonstrate that UniSparse achieves a state-of-the-art trade-off between accuracy and energy efficiency across models and datasets. The sparsified ResNet-18 model requires only 7.04M SOPs for inference to achieve 92.38\% accuracy on the CIFAR-10 dataset. Our work highlights the great potential of deep SNNs in improving energy efficiency.

Spiking Neural Networks are a type of AI model inspired by how the brain communicates using sparse, discrete signals, or “spikes.” However, as these models become larger and deeper to handle difficult tasks, their computational demands increase, which diminishes their natural energy-saving benefits. We developed UniSparse, a unified framework that reduces unnecessary computation by simultaneously pruning unnecessary network connections and filtering out redundant spikes. The key idea is that normalization parameters can act like adjustable firing thresholds, helping the network avoid weak or unimportant spikes while preserving important ones. UniSparse achieves a state-of-the-art balance between maintaining high task accuracy and drastically lowering energy consumption across various models and datasets. This framework highlights the potential for building highly capable, deep AI systems without sacrificing their energy efficiency.