SURGE: Surrogate Gradient Adaptation in Binary Neural Networks

The training of Binary Neural Networks (BNNs) is fundamentally based on gradient approximation for non-differentiable binarization operations (e.g., sign function). However, prevailing methods including the Straight-Through Estimator (STE) and its improved variants, rely on hand-crafted designs that suffer from gradient mismatch problem and information loss induced by fixed-range gradient clipping. To address this, we propose SURrogate GradiEnt Adaptation (SURGE), a novel learnable gradient compensation framework with theoretical grounding. SURGE mitigates gradient mismatch through auxiliary backpropagation. Specifically, we design a Dual-Path Gradient Compensator (DPGC) that constructs a parallel full-precision auxiliary branch for each binarized layer, decoupling gradient flow via output decomposition during backpropagation. DPGC enables bias-reduced gradient estimation by leveraging the full-precision branch to estimate components beyond STE's first-order approximation. To further enhance training stability, we introduce an Adaptive Gradient Scaler (AGS) based on an optimal scale factor to dynamically balance inter-branch gradient contributions via norm-based scaling. Experiments on image classification, object detection, and language understanding tasks demonstrate that SURGE performs best over state-of-the-art methods.

Binary neural networks can greatly reduce the memory and computation needed to run AI models, but they are difficult to train because binarization blocks accurate gradient learning. This work introduces SURGE, a training-time method that adds a temporary full-precision auxiliary branch to provide more reliable gradient guidance for binary layers. SURGE also adaptively controls the strength of this auxiliary signal so that training remains stable. After training, the auxiliary branch is removed, so the final model keeps the same efficient binary structure with no extra inference cost. Experiments show that SURGE improves binary models across image classification, object detection, and language understanding, making efficient AI deployment more practical on resource-limited devices.