Understanding Clipping in Zeroth-order Optimization

Zeroth-order optimization has emerged as a resource-efficient alternative to SGD when fine-tuning LLMs, thanks to the inherent low dimensionality of the loss landscape. It relies on two-point estimate of the gradient that only accesses the function value and not the gradient, bypassing resource heavy backpropagation. Recently, it was accidentally discovered that per-sample clipping can improve performance of the trained model, while running differentially private version of zeroth-order optimization \cite{zhang2024dpzero} with very large privacy parameter $\epsilon$ (corresponding to very little privacy). In this paper, we systematically investigate this phenomenon and demonstrate that ($i$) the optimal choice of learning rate $\eta$ for a given clipping threshold $c$ satisfy $\eta c =$ constant; ($ii$) clipping drives the zeroth-order optimization to a different solution that generalizes better; and ($iii$) this phenomenon is only observed with per-sample clipped zeroth-order optimization and not other zeroth-order alternatives that also provide robust estimate of the gradient.