FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction

Differentially private (DP) training protects individual examples by adding noise to gradients, but the injected noise interacts nontrivially with adaptive optimizers. Recent DP methods temporally filter privatized gradients to reduce variance; however, filtering also changes the DP noise statistics seen by AdamW’s second-moment accumulator. As a result, bias corrections derived for unfiltered DP noise (e.g., subtracting $\sigma_w^2$) can become miscalibrated when filtering is present. We propose FIBER, a DP optimizer designed for temporally filtered privatized gradients. FIBER (i) performs denoising in innovation space by filtering the residual stream and integrating it to form the filtered gradient estimate, (ii) decouples the two-point observation geometry from the innovation gain to enable independent tuning, and (iii) introduces a filter-aware second-moment calibration that subtracts the attenuated DP noise contribution $A(\omega)\sigma_w^2$, where $A(\omega)$ is derived in closed form for the innovation filter and can be computed for general stable linear filters. Across vision and language benchmarks, FIBER consistently demonstrates substantial improvements in the performance of DP optimizers, surpassing state-of-the-art results under equivalent privacy constraints on multiple tasks.

Modern AI models learn by example. However, when training data includes sensitive sources such as medical records, personal messages, or financial data, models can inadvertently memorize and leak individual details. Differential privacy, a standard safeguard, addresses this risk by introducing carefully calibrated random noise during training. However, this noise disrupts widely used adaptive training algorithms that adjust parameters dynamically, resulting in noticeably reduced model accuracy. FIBER is a new training method designed specifically for private learning. Rather than smoothing the noisy learning signal directly, FIBER tracks changes between training steps, similar to a navigation system that emphasizes course corrections instead of redrawing the entire route at each moment. Additionally, FIBER recalibrates the training algorithm to account for how this cleaning process alters the residual noise, a subtle effect overlooked by previous methods that contributed to degraded performance. On standard image-recognition and language-understanding benchmarks, FIBER produced more accurate models than existing private training methods, with the largest improvements observed under the strictest privacy constraints. These results enhance the practicality of private machine learning for real-world applications that depend on sensitive data.