LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning

MoE-PEFT methods combine Mixture of Experts with parameter-efficient fine-tuning for multi-task adaptation, but require separate adapters per expert—causing trainable parameters to scale linearly with expert count and limiting applicability to adapter-based architectures. We propose LiME (Lightweight Mixture of Experts), which achieves expert specialization through lightweight modulation rather than adapter replication. Instead of separate adapters, LiME uses a single shared PEFT module and modulates its output with lightweight expert vectors, reducing expert parameters while generalizing to any PEFT method. Notably, LiME introduces zero-parameter routing by leveraging existing frozen and adapted representations—eliminating learned router parameters typically required per layer. Theoretically, we prove that (i) more experts preserve more task-relevant information and (ii) modulation approximates full expert-specific PEFT with bounded error. LiME further incorporates n-gram windowed routing and adaptive expert selection (Auto Top-K) based on routing confidence. Experiments on MMT-47, a multimodal multi-task benchmark with 47 tasks spanning text, image, and video, demonstrate that LiME achieves competitive or superior performance while using up to 4× fewer trainable parameters and up to 29% faster training compared to corresponding MoE-PEFT baselines.

Modern AI systems — the kind behind chatbots, image search, and video understanding tools — are enormous, often containing billions of internal settings. Adapting one of these systems to a new task, like answering medical questions or reading charts, normally requires expensive retraining. A common shortcut is to leave the original system untouched and instead train a small "helper" attached to it. To handle many different tasks at once, researchers build a whole team of these helpers, with a traffic controller deciding which helper to use for each input. The catch is that this team grows quickly: more helpers means more pieces to train, more traffic-control machinery, and less practice data for each helper to learn from. We introduce LiME, a method that gets the same benefits with far less overhead. Instead of training a separate helper for every task, LiME trains just one shared helper and gives each specialist a small set of adjustment dials — roughly the size of a single row in a spreadsheet — that fine-tune the helper's output for their particular job. We also eliminate the traffic controller entirely. LiME decides which specialist to call by listening to signals the AI system is already producing as it works, so no additional machinery is needed. We tested LiME on 47 different tasks covering text, images, and video, all handled by one system at the same time. It matched or beat existing methods while needing up to four times fewer adjustable pieces and finishing training nearly a third faster. That difference matters in practice: it means smaller research groups, hospitals, schools, and businesses can adapt powerful AI to their own needs without access to massive computing resources — making advanced AI more affordable and more widely available.