Linear Loss Classification: Efficient Training Through Neural Collapse

Logistic loss is widely used for classification, as neural networks trained by gradient descent (GD) on this loss exhibit strong generalization. This success is commonly attributed to inductive biases, such as the directional convergence of the last-layer classifier toward the max-margin solution for a given feature representation, and neural collapse (NC), a terminal-phase phenomenon characterized by structural simplification of last-layer features and classifiers. However, they can emerge slowly due to exponentially decaying gradients. In this work, we introduce linear loss $l(u)=-u$, which eliminates gradient decay and leads to faster training dynamics. Under this loss, GD no longer directionally converges to the max-margin solution, but instead aligns with the difference between class means. While this may appear suboptimal, once NC occurs, the two directions become closely aligned, mitigating the discrepancy. Empirically, we demonstrate that GD with the linear loss, combined with appropriate normalizations, induces NC, and that both NC and generalization occur faster than with logistic loss. On the theoretical side, we prove that two-layer neural networks with ReLU activation trained with GD on linear loss exhibit directional NC for orthogonally separable data. Together, these results suggest that the linear loss, despite its simplicity and deviation from standard classification losses, can be sufficient to induce NC and thereby achieve strong generalization faster than other losses.