New Evidence of the Two-Phase Learning Dynamics of Neural Networks

Extensive evidence suggests that training dynamics undergo a distinct phase transition, yet our understanding of this transition still lags behind. In this paper, we introduce an interval-wise perspective that compares network states across a time window, revealing two new phenomena that illuminate the two-phase nature of deep learning. i) The Chaos Effect. By injecting an imperceptibly small parameter perturbation at various stages, we show that the response of the network to the perturbation exhibits a transition from chaotic to stable, suggesting there is an early critical period where the network is highly sensitive to initial conditions; ii) The Cone Effect. Tracking the evolution of the empirical Neural Tangent Kernel (eNTK), we find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset: while the kernel continues to change, it gets trapped in a tight angular region. Together, these effects provide a dynamical view of how deep networks transition from sensitive exploration to stable refinement during training.