On the Geometry of Memorization: Interpolation and Second-Order Representation Irregularity

We study memorization through the second-order geometry of learned representations. Using the pullback metric, we show that interpolating rapidly varying labels forces large second-order variation, linking interpolation to higher-order complexity. Empirically, both structured and random labels exhibit large magnitude, but differ in their spatial organization. Structured targets yield smooth, globally distributed variation, while random targets produce sparse, high-magnitude spikes. This distinction is captured by a curvature localization measure, showing that memorization is not characterized by the magnitude of second-order variation, but by its organization in representation space.