MoLoRA: Composable Specialization via Per-Token Adapter Routing

Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like ``write code to solve this equation,'' which need expertise from multiple specialized adapters. We introduce \emph{per-token routing}, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal for mixed-adapter requests: $N$ work for $N$ tokens, versus $K \cdot N$ for per-sequence systems that must replay one adapter per pass. Our key contribution is MoLoRA (Mixture of LoRA), which enables \emph{composable specialization}: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7$\times$ smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.

Many AI systems can be customized by adding small skill modules, but current serving systems usually choose only one module for an entire user request. This is limiting when a request mixes different needs, such as writing code to solve a math problem or producing text and images in the same response. MoLoRA lets the system choose the right skill module for each part of a request, so multiple specialists can work together within one answer. This makes it possible for a smaller model with focused skills to outperform a much larger general model on several reasoning tasks. The paper also shows how to run this approach efficiently, reducing delay and making response times more predictable.