Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization. We ask whether these two bottlenecks can be addressed jointly by training parts of the mixture independently and reconciling them once in parameter space. We develop a local quadratic theory inside a shared flat basin that yields three results: weight merging produces a curvature-weighted variance reduction; PCA-aligned conflict splitting maximizes this gain along high-curvature directions; and merging additionally acts as spectral filtering with implicit norm regularization. These results directly motivate MERIT, a decentralized merge-ready instruction-tuning pipeline that estimates dataset-level gradient conflicts, partitions the mixture along the top PCA conflict axes, fine-tunes each partition independently with no inter-partition communication, and merges once via token-weighted averaging. On Qwen2.5-VL-3B with 136 Vision-FLAN tasks, MERIT improves the 8-benchmark average from 54.3 (joint training) to 57.0. The same recipe scales to a 7B model on a 1.6M-example, 176-source mixture—matching or exceeding centralized joint training with minimal cost overhead—and transfers to text-only FLAN. Our code is available at https://github.com/naver-ai/merit.

Modern multimodal AI models, which handle both text and images, are trained on huge collections of examples spanning many different tasks. As these collections grow, two problems emerge. Different tasks pull the model in conflicting directions: learning to read text in documents can clash with learning to solve math or describe photographs. And standard training requires expensive GPU clusters whose machines constantly stay in sync, an arrangement out of reach for many researchers. We propose a simple alternative: group the tasks so conflicting ones are separated, train a separate model on each group with no cross-group communication, then combine them by averaging at the end. Our analysis shows that averaging provably cancels interference along the directions where tasks disagree most, so the right way to split the data is along exactly those axes. In experiments, MERIT generally outperforms standard joint training across our 8-benchmark suite, raising the average by 2.7 points, and the same recipe scales to a 7B model trained on 1.6 million examples from 176 sources. By eliminating constant synchronization during training, MERIT lets a much wider range of labs and organizations train powerful AI, broadening who can shape its future.