IsoPLM: Isolating the Impacts of Architecture on Protein Language Models

Since the release of ESM, protein language model (PLM) architectures have proliferated rapidly, with new primitives released regularly. But because these releases vary simultaneously in dataset, training recipe, and scale, broader claims about which architectural primitives actually drive performance are difficult to isolate. We therefore pre-train a family of PLMs at three scales while holding data and training recipe constant, isolating the contributions of hybrid and mixture-of-experts (MoE) primitives. Each model is evaluated on a suite of benchmarks spanning protein structure understanding and both in-domain and out-of-domain variant effect prediction, with internal representations probed through linear probes, and expert co-activation and specialization analysis. We find that the apparent benefits of hybrid and MoE priors erode with increasing scale. Across the full sweep, both dense and sparse hybrid variants generalize most strongly, and at all but the largest scales sparse hybrids dominate on local structural tasks while dense hybrids dominate on global structural tasks. The representational analysis shows that hybrids encode a strong sequence-level prior, while MoEs encode a sharply localized structural prior. More broadly, this work outlines a controlled-pretraining framework for rigorously evaluating new architectural primitives in biological foundation models.