KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

The success of Hyper-Connections (HC) in neural networks (NN) has also highlighted issues related to training instability and restricted scalability. The Manifold-Constrained Hyper-Connections (mHC) mitigate these challenges by projecting the residual connection space onto a Birkhoff polytope, however, it faces two issues: 1) its iterative Sinkhorn-Knopp (SK) algorithm does not always yield exactly doubly stochastic residual matrices; 2) mHC incurs a prohibitive $\mathcal{O}(n^3C)$ parameter complexity with $n$ as the width of the residual stream and $C$ as the feature dimension. The recently proposed mHC-lite reparametrizes the residual matrix via the Birkhoff-von-Neumann theorem to guarantee double stochasticity, but also faces a factorial explosion in its parameter complexity, $\mathcal{O} \left( nC \cdot n! \right)$. To address both challenges, we propose **KromHC**, which uses the $\underline{\text{Kro}}$necker products of smaller doubly stochastic matrices to parametrize the residual matrix in $\underline{\text{mHC}}$. By enforcing manifold constraints across the factor residual matrices along each mode of the tensorized residual stream, KromHC guarantees exact double stochasticity of the residual matrices while reducing parameter complexity to only $\mathcal{O}(n^2C)$. Experiments show that KromHC matches or even outperforms other state-of-the-art (SOTA) mHC variants, while requiring significantly fewer trainable parameters. The code is at https://github.com/wz1119/KromHC.

Deep AI models pass information between their internal layers using direct connections. Hyper-connections expand these pathways into multiple streams to improve the model's expressiveness, but they suffer from training instability. Existing solutions attempt to fix this using a mathematical constraint, but they either fail to enforce it exactly or cause an explosion in the number of parameters in the AI model. To solve this trade-off, we introduce KromHC. Instead of learning one massive matrix to route information, our method uses a mathematical operation, the Kronecker product, to assemble the matrix from much smaller matrices. This approach guarantees that the constraint is enforced exactly while significantly decreasing the number of parameters in the AI model. As a result, KromHC matches or outperforms state-of-the-art methods, allowing for training scalable deep AI models with significantly lower parameter overhead.