Geometry of Rank Constraints in Shallow Polynomial Neural Networks
Param Mody · Maksym Zubkov
Abstract
We study shallow quadratic polynomial networks of width 2. Across 500 random initialisations, we observe a universal two-phase dynamic: rapid projection onto the rank-2 variety followed by slow diffusion along its flat directions. Rank-mismatch traps optimisation on a 2-D ellipse and imposes a non-zero loss floor but when the target rank matches capacity, the manifold collapses to three minima clusters where one reaches zero loss.
Chat is not available.
Successful Page Load