DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning

Differentially private optimization suffers from a fundamental geometric mismatch: deep networks have highly anisotropic loss landscapes, yet DP-SGD injects isotropic noise. Second-order preconditioning can resolve this, but estimating curvature typically requires private data (consuming privacy budget) or public data (introducing distribution shift). We show that the Fisher Information Matrix decouples into *architectural sensitivity*, recoverable via synthetic noise, and *input correlations*, approximable from modality-specific frequency statistics. We propose DP-KFC, which constructs KFAC preconditioners by probing networks with structured synthetic noise, requiring neither private nor public data. Empirically, DP-KFC consistently outperforms DP-SGD and adaptive baselines across diverse modalities in strong privacy regimes ($\varepsilon \leq 3$). DP-KFC matches private-data preconditioners while public-data variants degrade by up to $4.8$ %, showing that curvature can be estimated without consuming privacy budget or introducing distribution shift. This enables privacy-preserving learning in specialized domains (e.g., medical applications) where regulatory constraints make data scarce.

Modern AI models are increasingly trained on sensitive personal data, such as medical scans or patient records. To protect the people behind that data, a widely used technique adds a controlled amount of random "noise" during training, so that what the model learned cannot be traced back to any single individual. This protection has a cost. The noise is added evenly across the model, but the useful learning signal is not, because some parts of a model matter far more than others. The noise can swamp the signal, leaving the protected model noticeably less accurate. The mismatch can be corrected by "reshaping" the training so the noise fits the model better. Until now, finding that reshaping required either using up part of the limited privacy protection, or a separate public dataset resembling the private one, which often does not exist for specialized problems. We show that the right reshaping depends mainly on the design of the model itself, not on the specific data it is trained on. We can therefore work it out by probing the model with structured random patterns instead of real data, at no privacy cost and with no extra dataset. This yields more accurate privacy-preserving AI, especially in sensitive areas such as medical imaging where suitable public data is unavailable.