LS-Merge: Merging Language Models in Latent Space

Model merging in weight space efficiently reuses pretrained models, but existing methods assume matching architectures, making heterogeneous merges brittle or infeasible. We address this by encoding weights into a smooth latent space where cross-architecture merging becomes natural, then decoding the merged representation back to weights. Two challenges arise: the billions of parameters in LLMs make latent encoding costly, and high compression ratios can hinder generalization to unseen weights. We tackle both with a transformer-based VAE trained via a two-stage compression curriculum with layer-aware chunking, first learning a high-capacity representation and then distilling it to a compact code for stability and out-of-distribution generalization. A dimensionality-matching projection and distributional alignment via optimal transport further enables interpolation across different model sizes. Empirically, latent-space interpolation is more robust than direct weight averaging and yields stronger downstream performance across heterogeneous merges. Together, these components provide a scalable recipe for model merging across heterogeneous architectures.