Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets

Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free architectures. While prior work has primarily examined scaling effects from an optimization perspective, this paper investigates their role in residual architectures through the lens of generalization theory. Specifically, we establish that wide residual networks (ResNets) with constant scaling factors become asymptotically unlearnable as depth increases. In contrast, when the scaling factor exhibits rapid depth-wise decay combined with early stopping, over-parameterized ResNets achieve minimax-optimal generalization rates. To establish this, we demonstrate that the generalization capability of wide ResNets can be approximated by the kernel regression associated with a specific kernel. Our theoretical findings are validated through experiments on synthetic data and real-world classification tasks, including MNIST and CIFAR-100.

Modern AI systems are often made very deep by stacking many layers, but more depth can make them unstable and worse at handling new examples. Many successful designs use shortcut connections: each layer mostly keeps the information it receives, while adding a smaller update. Engineers often shrink that update with a multiplier, but it has not been clear whether this only makes training easier or also improves how well the model learns. We show that this multiplier can act as a built-in form of protection against memorizing the training data. If the multiplier stays fixed while the network becomes very deep, the model can eventually lose the ability to learn useful patterns. If the multiplier becomes smaller quickly enough as depth grows, and training is stopped before the model starts fitting noise, the model can keep learning in a near-best possible way. Experiments on artificial data and image recognition tasks support these findings. This helps explain why simple scaling rules work and can guide the design of deep AI models that perform better on new data.