Low-Rank Networks Recover Weight and Functional Symmetry Better

We study an empirical phenomenon in low-rank random-feature networks: even when stochastic training sees one non-symmetrized sample at a time, the learned model can recover a globally symmetric function and, in successful regimes, symmetric internal partial functions. The effect is not explained by output invariance alone. Low-loss counterexamples exist where the output is nearly symmetric but the learned partial functions remain asymmetric. We therefore analyze symmetry simultaneously in function space, representation space, and weight space. Across one-dimensional sum of cosines experiments and new multidimensional batch-size-one experiments, low-rank bottleneck networks exhibit a distinctive regime where symmetry is visible in late partial functions and in mirror-paired first-layer atoms with matched outgoing weights.