Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway

Recent analyses of multi-pathway Deep Linear Networks use Gradient Flow to predict a "winner-takes-all" specialization in which path symmetry breaks and each feature concentrates in a single pathway. In this work, we show that discrete Gradient Descent (GD) with a large step size tells a different story. We prove that single-path solutions are sharp minima, whereas distributing signals across pathways reduces sharpness by a factor that decreases with both the number of pathways and depth. Consequently, while early training reproduces the depth-driven symmetry breaking predicted by GF, oscillations at the Edge of Stability subsequently override this tendency and drive the network into a re-balancing phase, where signals redistribute across pathways. Together, these results clarify how depth shapes pathway competition and explain why large-step GD favors shared representations rather than persistent single-pathway dominance.

Many neural networks contain parallel routes through which information can flow. A basic question is whether training uses several of these routes together, or whether it tends to rely on only one. Previous theory, based on an idealized view where training changes the network in infinitesimally small steps, predicted that one route can dominate while the others become almost unused. We revisit this question in a simplified mathematical model: deep linear networks with multiple parallel pathways. In this setting, we show that putting all of a learned feature into one pathway creates a sharper loss landscape, meaning that small parameter changes can affect the training loss more strongly. Spreading the same feature across several pathways gives a flatter configuration. With small learning steps, training follows the earlier prediction and one pathway tends to win. With appropriately large learning steps, however, training first moves toward this one-pathway solution, then becomes unstable and shifts part of the signal back into other pathways. Thus, in this model, the finite steps used by gradient descent can change which solution training selects. The result suggests that theories based only on infinitesimally small training steps may miss important behavior of practical optimization, and it raises future questions for neural architectures with parallel components.