Geometry of Rank Constraints in Shallow Polynomial Neural Networks

We study shallow quadratic and cubic polynomial neural networks of width 2. In this setting, the ambient space is the space of symmetric polynomials, which is finite-dimensional. We consider four target functions that correspond to rank-2 and rank-3 symmetric matrices, and rank-2 and rank-3 symmetric tensors. We compare the learning dynamics when the target function lies within versus outside the function space (neuromanifold), and we analyze the patterns of critical points in both the parameter space and the corresponding functional space.