Transport and Merge: Cross-Architecture Merging for Large Language Models

Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data, yet many real-world deployments rely on smaller models trained or adapted from low-resource data. This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets. While model merging provides an effective transfer mechanism, most existing approaches assume architecture-compatible models and therefore cannot directly transfer knowledge from large high-resource LLMs to heterogeneous low-resource targets. In this work, we propose a cross-architecture merging framework based on optimal transport (OT) that aligns activations to infer cross-neuron correspondences between heterogeneous models. The resulting transport plans are then used to guide direct weight-space fusion, enabling effective high-resource to low-resource transfer using only a small set of inputs. Extensive experiments across low-resource languages and specialized domains demonstrate consistent improvements over target models.

Modern AI language systems are often very large and powerful, but they are expensive to run and difficult to adapt for languages or fields where only limited data is available. Smaller models are easier to deploy, yet they usually lack the knowledge learned by larger models. This paper introduces a way to transfer useful knowledge from large models into smaller models, even when the models are built differently. Instead of training the smaller model from scratch or requiring a long teaching process, our method compares how different models respond to the same examples, finds which parts of the models play similar roles, and then uses this information to combine their internal knowledge. Experiments show that this approach can improve smaller models for low-resource languages and specialized domains using only a small amount of unlabeled text. This makes it easier to build efficient language models for settings where data, computing resources, or deployment budgets are limited.