Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in smaller or insufficiently trained models such expert solutions occupy a negligible fraction of the volume of this distribution, making their discovery reliant on structured optimization methods such as gradient descent. In contrast, in large, well-pretrained models the density of task-experts increases dramatically, so that diverse specialists populate a substantial fraction of the neighborhood around the pretrained weights. Motivated by this perspective, we explore a simple, fully parallel post-training method that samples $N$ parameter vectors at random, selects the top $K$, and ensembles them via majority vote to combine complementary expertise. Despite its simplicity, this approach is competitive with standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.

Large AI models are usually improved by training them further on new data. In this paper, we find that large AI models are less like a single expert and more like a large team of hidden specialists. Some are better at solving math problems, while others are better at different tasks. As models become larger and receive more pretraining, more of these specialists appear to emerge. By making random perturbation to the model, we can uncover different specialists. Surprisingly, random guessing followed by ensembling can compete with much more complex state-of-the-art methods.