Linear Loss Classification: Efficient Training Through Neural Collapse

Logistic loss is the de facto standard for classification. This success is commonly attributed to inductive biases, such as the directional convergence of the last-layer classifier toward the max-margin solution for a given feature representation, and neural collapse (NC), a terminal-phase phenomenon characterized by structural simplification of last-layer features and classifiers. However, both emerge slowly because the gradients of logistic loss decay exponentially. In this work, we introduce a linear loss $\ell(u)=-u$, which avoids exponentially decaying gradients and leads to faster training dynamics. Under this loss, the last-layer classifier trained by GD no longer directionally converges to the max-margin solution, but instead aligns with the difference between class means. Although this direction differs from the max-margin solution and may therefore appear suboptimal, once NC occurs, the two directions become closely aligned, reducing the discrepancy. Empirically, we demonstrate that GD with linear loss, combined with weight normalization, exhibits NC, and that both NC and generalization occur faster than with logistic loss. On the theoretical side, we prove that the unconstrained feature model (UFM) exhibits NC, while shallow ReLU networks exhibit a weaker directional collapse of features and classifiers. Together, these results demonstrate that linear loss, despite being unbounded below, provides a simple yet effective alternative to logistic loss by accelerating both neural collapse and generalization.