The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?

Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures and consistently outperform their non-gated counterparts, yet the underlying reasons for this advantage remain unclear. In this work, we study GLU by analyzing two-layer networks in the neural tangent kernel (NTK) regime. Our analysis reveals that the GLU structure reshapes the NTK spectrum, leading to a smaller condition number and a more compact eigenvalue distribution. Building on this finding, we further analyze the resulting training dynamics and show how the reshaped spectrum leads to faster convergence of GLU models, including a characteristic loss-crossing phenomenon observed between GLU and non-GLU models. Finally, we empirically observe that GLU has limited impact in reducing the generalization gap on various models, including ViT and GPT-2, suggesting that its primary benefit lies in accelerating optimization rather than reducing the generalization gap. The code is available at: https://github.com/Zemdalk/GLU-NTK.

Modern large language models are built from many components. One widely used component is called a Gated Linear Unit (GLU). GLU often improves model training in practice, but it has been unclear why. We study this question by looking at a simplified setting where the learning process can be described mathematically. Our main finding is that GLU improves a quantity called the condition number of a standard matrix used to analyze neural network training, which measures how uneven the learning process is across different directions. A smaller condition number means that the model can reduce different parts of its training error at more balanced speeds, rather than being held back by a few slow directions. Therefore, models with GLU can learn from training data faster than similar models without GLU. We further find that GLU does not substantially change how much better a model performs on training data than on new data. This suggests that GLU’s main benefit is not that it makes models inherently better at generalizing to new data, but that it makes them easier to learn from the given training data. Understanding this mechanism may help researchers design future models that learn more efficiently.