Orthogonal Mixture-of-Expert Low-Rank Adapter for Continual Learning

Continual Learning (CL) aims to prevent catastrophic forgetting during downstream finetuning. While Parameter-Efficient Fine-Tuning (PEFT) methods mitigate this by shielding pre-trained weights, they still suffer from severe cross-task interference. Existing solutions either use independent routers, causing structural misalignment, or rigid orthogonal constraints, severely limiting model plasticity. We propose the Orthogonal Mixture-of-Expert Low-Rank Adapter (OMoE-LoRA), which integrates an end-to-end contrastive soft router within the down-projection matrix to avoid misalignment, and an orthogonal constraint exclusively on the up-projection matrix to suppress cross-talk without sacrificing plasticity. Experiments on the MTIL benchmark demonstrate OMoE-LoRA achieves comparable accuracy with state-of-the-art method while effectively reducing trainable parameters.