Multiplicative Mixture of Experts for Rank-Efficient LLM Finetuning

Large language models (LLMs) have achieved impressive results in many general-purpose domains, but their performance on specific tasks can still be improved through finetuning. Parameter-efficient finetuning (PEFT) tailors an LLM to one or more tasks through a small amount of trainable parameters, requiring reduced computational resources. On one hand, techniques like low-rank adaptation (LoRA) provide the required parameter efficiency with additive adapters of low, and fixed, rank, which limits their flexibility. On the other hand, mixture of experts (MoEs) enhance the flexibility of a model at the cost of parameter count and memory budget. The combination of the two approaches, parameter-efficient MoEfication, has shown promise in addressing the issues of both. In this work, we show that replacing additive with multiplicative interactions improves the rank-efficiency of PEFT adapters, increasing the flexibility and reducing the number of parameters involved in MoEfication. In particular, our multiplicative method is especially effective when memory becomes the bottleneck, an increasingly common scenario with large-scale MoEs.We find that our quantum-inspired multiplicative method, OperA, is optimal given the same parameter budget for 7/8 models considered, using fewer or equal parameters than the baseline. In a multi-task setting, OperA fully surpasses all baselines on all models. We also propose an algorithm to optimize the use of OperA in large-scale deployments by alleviating its computational complexity. Finally, we provide evidence that OperA surpasses the effective rank of competing solutions by more than two orders of magnitude.