On the Learning Dynamics of Label-Noise Memorization in ReLU MLPs

Understanding the mechanisms of memorization in neural networks remains an open and challeng- ing problem. In this work, we study label-noise memorization in two-layer ReLU MLPs through the learning dynamics of the first-layer weights1. Our analysis suggests that label noise initially attenuates first-layer magnitude evolution while largely preserving weight directions, before in- ducing competing magnitude and directional dy- namics between clean and noisy samples. Experi- ments on MNIST further suggest that these com- peting dynamics can reach a dynamical equilib- rium prior to memorization, indicating that memo- rization can emerge without significant distortion of the first-layer weights.