Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos

We develop a mean-field theory of dropout as a perturbation of critical signal propagation at the edge of chaos, and show that it predicts a simple, no-cost change to standard practice: front-loaded dropout schedules cut test loss by 18–35% over constant dropout in MLPs and Vision Transformers at fixed budget. The theoretical mechanism is that dropout shifts the perfect-alignment fixed point, making the depth scale for information propagation finite even at critical initialization. We derive critical and crossover scaling laws for correlation decay and establish that smooth activations and kinked, ReLU-like activations constitute distinct universality classes, with different critical exponents and a universal two-parameter scaling collapse in detuning and dropout strength. The distinction traces to the analytic structure of the correlation map: smooth activations admit a Taylor expansion near perfect alignment, while kinked activations develop a branch point with universal non-analyticity. As a corollary, the framework yields saturated dropout profiles under fixed budget; a regularization-reach argument then selects front-loaded schedules, with accuracy gains as a consistent secondary effect. We also discuss how the same Gaussian-kernel structure extends the theory beyond MLPs toward CNNs and residual architectures.

Neural networks work best when information can pass through many layers without fading away or becoming unstable. This paper studies how dropout, a common training method that randomly turns off parts of a network to reduce overfitting, changes that flow of information. We show that dropout has a predictable effect on deep networks, and that different activation functions respond to it in different ways. The practical takeaway is that, for MLPs suffering from overfitting or poor information propagation, using stronger dropout in earlier layers and weaker dropout later can reduce loss and improve accuracy. Experiments on standard neural networks and vision models show that this depth-aware approach can improve test performance without adding extra computation.