ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocation

Large language models allocate uniform computation across all tokens, ignoring that some sequences are trivially predictable while others require deep reasoning. We introduce ConceptMoE, which dynamically merges semantically similar tokens into concepts through learnable chunking at target compression ratio $R$. The MoE architecture enables controlled evaluation: reallocating saved computation to match baseline FLOPs and parameters isolates genuine architectural benefits. ConceptMoE consistently outperforms standard MoE, achieving +0.9 points on language pretraining, +2.3 on long context, and +0.6 on multimodal tasks. Continual training conversion with layer looping gains +5.5 points. Beyond performance, at $R=2$, ConceptMoE reduces attention computation by $R^2\times$ and KV cache by $R\times$, delivering prefill speedups up to 175\% and decoding speedups up to 117\%. The minimal architectural changes enable straightforward integration, demonstrating that adaptive concept-level processing fundamentally improves LLM effectiveness and efficiency.

ConceptMoE shifts language model processing from uniform token-level to adaptive concept-level computation. By learning to merge semantically similar tokens into unified concepts while preserving fine-grained granularity for complex tokens, it performs implicit compute allocation—automatically investing computation where needed. Key results: (1) Fair comparison under identical parameters and FLOPs shows consistent gains across language (+0.9), vision-language (+0.6, +2.3 on long context), and continual training (+5.5 with layer loops, +6.4 from scratch). (2) Inherent efficiency: at compression ratio R=2, attention computation reduces by R²× and KV cache by R×, achieving prefill speedups up to 175% and decoding speedups up to 117%. (3) Minimal architectural changes (chunk module + decoder QKV projectors) enable straightforward deployment in existing MoE systems. Represents a paradigm shift toward hierarchical semantic processing in LLMs.