Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignment

Understanding why gradient-based training in deep networks exhibits strong implicit bias remains challenging, in part because tractable singular-value dynamics are typically available only for balanced deep linear models. We propose an alternative route based on two theoretically grounded and empirically testable signatures of deep Jacobians: depth-induced exponential scaling of ordered singular values and strong spectral separation. Adopting a fixed-gates view of piecewise-linear networks, where Jacobians reduce to products of masked linear maps within a single activation region, we prove the existence of Lyapunov exponents governing the top singular values at initialization, give closed-form expressions in a tractable masked model, and quantify finite-depth corrections. We further show that sufficiently strong separation forces singular-vector alignment in matrix products, yielding an approximately shared singular basis for intermediate Jacobians. Together, these results motivate an approximation regime in which singular-value dynamics become effectively decoupled, mirroring classical balanced deep-linear analyses without requiring balancing. Experiments in fixed-gates settings validate the predicted scaling, alignment, and resulting dynamics, supporting a mechanistic account of emergent low-rank Jacobian structure as a driver of implicit bias.

When a deep neural network is trained, it is systematically observed that a few directions in the input space become overwhelmingly more influential than the rest: we call this the low rank bias. The importance of a given relevant direction is quantified by a number called the singular value, so low rank bias means that certain singular values end up dominating all the others. This phenomenon has been linked to why deep networks tend to learn simple and well-generalizing solutions rather than memorizing training data. This paper asks why this structure emerges, and suggests an answer grounded in the architecture of the network, namely its depth. We show that multiplying many random matrices together, similarly to how parts of deep networks operate, naturally causes singular values to spread exponentially apart: deeper products yield stronger separations. We then show that strong spectral separation has a further consequence: the dominant input and output directions of different parts of the network become aligned with one another. This alignment, in turn, causes the singular values to progressively decouple during training, simplifying their dynamics and allowing for approximations which explain low rank bias. These approximations had been previously derived only for simpler linear networks under restrictive assumptions at initialization.