Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning

Recently, the prominent performance of large language models (LLMs) has been largely driven by multi-task instruct-tuning. Unfortunately, this training paradigm suffers from a key issue, named cross-task interference, due to conflicting gradients over shared parameters among different tasks. Some previous methods mitigate this issue by isolating task-specific parameters, e.g., task-specific neuron selection and mixture-of-experts. In this paper, we empirically reveal that the cross-task interference still exists for the existing solutions because of many parameters also shared by different tasks, and accordingly, we propose a novel solution, namely Basic Abilities Decomposition for multi-task Instruct-Tuning (BADIT). Specifically, we empirically find that certain parameters are consistently co-activated, and that co-activated parameters naturally organize into base groups. This motivates us to analogize that LLMs encode several orthogonal basic abilities, and that any task can be represented as a linear combination of these abilities. Accordingly, we propose BADIT that decomposes LLM parameters into orthogonal high-singular-value LoRA experts representing basic abilities, and dynamically enforces their orthogonality during training via spherical clustering of rank-1 components. We conduct extensive experiments on the SuperNI benchmark with 6 LLMs, and empirical results demonstrate that BADIT can outperform SOTA methods and mitigate the degree of cross-task interference.

Imagine training a single AI assistant to handle hundreds of different tasks, like translating languages, and writing code; often, learning one new skill accidentally degrades what the AI already learned about another because all these tasks fight over the same internal parameters. This common issue is known as cross-task interference. While previous solutions tried to give each task its own dedicated corner in the AI's network, our research reveals that these corners still overlap too much, and the confusion persists. To solve this, we introduce BADIT, a novel training approach inspired by our discovery that the AI’s internal network naturally organizes itself into tight-knit groups, much like a set of "basic universal abilities." Think of these as foundational Lego bricks, such as logic, creativity, or grammar, where any complex task is just a unique combination of these distinct blocks. Instead of letting tasks clash, BADIT identifies these core abilities, keeps them strictly separate and independent from one another during training, and builds new skills by cleanly mixing and matching them. We tested BADIT across six different major AI models, and the results demonstrate that our method successfully prevents the AI from getting confused, allowing it to master multiple skills simultaneously much more effectively than current state-of-the-art methods.