Privacy Amplification in Differentially Private Zeroth-Order Optimization with Hidden States

Zeroth-order optimization has emerged as a promising approach for fine-tuning large language models on domain-specific data, particularly under differential privacy (DP) and memory constraints. While first-order methods have been extensively studied from a privacy perspective, the privacy analysis and algorithmic design for zeroth-order methods remain significantly underexplored. A critical open question concerns hidden-state DP analysis: although convergent privacy bounds are known for first-order methods, it has remained unclear whether similar guarantees can be established for zeroth-order methods. In this work, we provide an affirmative answer by proving a convergent DP bound for zeroth-order optimization. Our analysis generalizes the celebrated privacy amplification-by-iteration framework to the setting of smooth loss functions in zeroth-order optimization. Furthermore, it induces better DP zeroth-order algorithmic designs that are previously unknown to the literature.

Fine-tuning large AI models privately often requires a memory-saving technique called "zeroth-order optimization." However, this method has been missing a critical mathematical aspect known as a "hidden-state" privacy analysis. While standard (first-order) methods guarantee that privacy limits remain stable, or "convergent", as training progresses by keeping internal updates hidden, it was entirely unknown if zeroth-order methods could achieve these same stable bounds. In this paper, we answer this open question by proving that zeroth-order methods do indeed possess convergent privacy guarantees. We achieved this by adapting a specific framework where privacy mathematically amplifies with each training step compared to standard "public-state" privacy analysis. Our result secures the missing mathematical proofs and paves the way to entirely new, highly effective algorithmic designs for memory-efficient, privacy-preserving AI training.