Decoupling Regularization and Privacy in Differentially Private Ridge Regression and ERM

We study ridge regression and ridge-regularized empirical risk minimization (ERM) under $(\varepsilon,\delta)$-differential privacy via output perturbation. In classical private ERM, the ridge parameter simultaneously controls statistical regularization and the estimator’s global sensitivity. Larger regularization reduces the DP noise scale but increases bias. So choosing the tuning parameter becomes a privacy--accuracy bottleneck. We propose a framework that makes these two roles explicit by decoupling regularization into (i) a statistical penalty $\alpha$, defining the target ridge/ERM solution, and (ii) a privacy-stabilization parameter $c$, used only to enforce a curvature floor and hence a tight sensitivity bound. We apply this framework to ridge regression, where $c$ is used to boost the minimum eigenvalue of the empirical Gram matrix. We derive an explicit bias--variance--DP-variance risk decomposition and characterize optimal $(\alpha,c)$ in several regimes, yielding sharp tuning guidance and improved accuracy relative to single-parameter regularization. Finally, we extend the same decoupling principle to general ridge-regularized ERM. We support the theory with simulations.

Modern data analysis often relies on sensitive individual-level data, making privacy protection an important requirement when releasing statistical models. Differential privacy addresses this by adding a random noise to model outputs, where the noise is calibrated by global sensitivity. In ridge regression, this creates a tuning challenge: the usual ridge parameter must both control statistical regularization and reduce the global sensitivity that determines noise calibration. These two roles can conflict, since stronger regularization improves privacy stability but increases statistical bias. We propose a two-parameter framework that separates these roles. One parameter controls statistical regularization, while the other acts as a privacy guardrail by stabilizing ill-conditioned datasets and reducing sensitivity. For ridge regression, this guardrail boosts the smallest eigenvalue of the empirical Gram matrix. We derive an explicit decomposition of the error into statistical bias, sampling variance, and privacy-induced variance, which leads to theory-guided parameter choices. We also extend the same idea to ridge-regularized ERM. Simulations and real-data experiments show that this decoupling improves accuracy over standard differentially private regression methods while maintaining rigorous privacy guarantees.