Expand Neurons, Not Parameters

This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance. We show that this gain corresponds with a decrease in interference between multiple features that would otherwise share the same neurons. On symbolic Boolean tasks, splitting each neuron into sparser sub-neurons with knowledge of the clauses systematically reduces polysemanticity metrics and yields higher task accuracy. Notably, even random splits of neuron weights approximate these gains, indicating that reduced collisions, not precise assignment, are a primary driver. Consistent with the superposition hypothesis, the benefits of this framework grow with increasing interference: when polysemantic load is high, accuracy improvements are the largest. Transferring these insights to more realistic models, including classifiers over CLIP embeddings, convolutional neural networks, and deeper multilayer networks, we find that widening networks while maintaining a constant non-zero parameter count consistently increases accuracy. These results identify an interpretability-grounded mechanism to leverage width against superposition, improving performance without increasing the number of non-zero parameters. Such a direction is well matched to modern accelerators, where memory movement of non-zero parameters, rather than raw compute, is often a dominant bottleneck.

Modern machine learning models often improve by becoming larger, but this also makes them more expensive to store and use. We ask whether a model can perform better without increasing the number of non-zero parameters it has. Our work is inspired by the idea that individual neurons often have to represent multiple useful features at once, which can cause these features to interfere with each other. We study a simple way to reduce this interference: split neurons into smaller, more specialized pieces while keeping the total number of non-zero parameters fixed. Across controlled reasoning tasks and image classification settings, we find that this can improve accuracy, especially when many features are competing for too few neurons. These results suggest that model design should not only focus on how many parameters a model has, but also on how those parameters are arranged across neurons. In the long term, this could help guide the development of more efficient machine learning models.