Separating Representation from Reconstruction Enables Scalable Text Encoders

While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly *unexploitable* by frozen probes, despite improved perplexity. The misalignment originates in BERT's flat design, which couples representation learning to the token reconstruction loss. We propose **CrossBERT**, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\geq 50$%) and gradient collection over all tokens via a *Complementary Masking Strategy*, respectively increasing throughput by $1.5$ to $2\times$ and sample efficiency by $2\times$. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.

Today's AI language models come in two types. Decoders, which generate text and Encoders, which read a passage and turn it into a compact numerical summary of its meaning, powering search, recommendations, and the retrieval tools chatbots rely on. Encoders' core design has barely changed since the original BERT model appeared in 2018. When scaling up these models we found a surprising problem: as standard encoders grow, the summaries they produce become harder to use. Our solution, CrossBERT borrows an idea from computer vision to separate text understanding from text generation into two separate networks. One component focuses on understanding; a small separate one handles reconstruction. The result is higher-quality summaries, faster training, and steady improvement with scale.