Orthogonal Model Merging

Merging finetuned Large Language Models (LLMs) has become increasingly important for integrating diverse capabilities into a single unified model. However, prevailing model merging methods rely on linear arithmetic in Euclidean space, which often destroys the intrinsic geometric properties of pretrained weights, such as hyperspherical energy. To address this, we propose Orthogonal Model Merging (OrthoMerge), a method that performs merging operations on the Riemannian manifold formed by the orthogonal group to preserve the geometric structure of the model’s weights. By mapping task-specific orthogonal matrices learned by Orthogonal Finetuning (OFT) to the Lie algebra, OrthoMerge enables a principled yet efficient integration that takes into account both the direction and intensity of adaptations. In addition to directly leveraging orthogonal matrices obtained by OFT, we further extend this approach to general models finetuned with non-OFT methods (e.g., low-rank finetuning, full finetuning) via an Orthogonal-Residual Decoupling strategy. This technique extracts the orthogonal components of expert models by solving the orthogonal Procrustes problem, which are then merged on the manifold of the orthogonal group, while the remaining linear residuals are processed through standard additive merging. Extensive empirical results demonstrate the effectiveness of OrthoMerge in mitigating catastrophic forgetting and maintaining model performance across diverse tasks.

Modern language models are often specialized by further training them on different tasks, such as coding, reasoning, dialogue, or domain-specific knowledge. A natural goal is to combine several specialized models into one model that keeps all of these abilities. However, many existing model merging methods combine model parameters using simple arithmetic, such as adding or averaging weights. This can damage important geometric patterns inside the model and may cause the merged model to forget skills or perform worse. We propose OrthoMerge, a new way to merge fine-tuned language models while better preserving the structure of their learned weights. Instead of treating model updates as ordinary vectors, OrthoMerge represents important parts of these updates as rotations and combines them on the curved geometric space where rotations naturally live. For models trained with other common fine-tuning methods, we also introduce a way to separate each model’s update into a rotation-like part and a remaining correction, then merge each part appropriately. Our experiments show that OrthoMerge helps combine different expert models more reliably, reducing forgetting and maintaining strong performance across diverse tasks.