Gradient Descent as a Perceptron Algorithm: Understanding Dynamics and Implicit Acceleration

Even for the gradient descent (GD) method applied to neural network training, understanding its optimization dynamics, including convergence rate, iterate trajectories, function value oscillations, and especially its implicit acceleration, remains a challenging problem. We analyze nonlinear models with the logistic loss and show that the steps of GD reduce to those of generalized perceptron algorithms (Rosenblatt, 1958), providing a new perspective on the dynamics. This reduction yields significantly simpler algorithmic steps, which we analyze using classical linear algebra tools. Using these tools, we demonstrate on a minimalistic example that the nonlinearity in a two-layer model can provably yield a faster iteration complexity $\tilde{\mathcal{O}}(\sqrt{d})$ compared to $\Omega(d)$ achieved by linear models, where $d$ is the number of features. This helps explain the optimization dynamics and the implicit acceleration phenomenon observed in neural networks. The theoretical results are supported by extensive numerical experiments. We believe that this alternative view will further advance research on the optimization of neural networks.

This work gives a new way to understand why gradient descent can train neural networks faster than simpler models. It shows that, for nonlinear models with logistic loss, gradient descent behaves like a generalized version of the classical perceptron algorithm. This connection makes the training dynamics easier to analyze and helps explain implicit acceleration: neural network structure itself can speed up optimization. The theory is illustrated on a simple example and supported by numerical experiments.