cMoLLM at Scale: Horizontal Scaling Laws for Convolutionally-Gated Mixture-of-LLMs

Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size—a critical bottleneck as models approach trillion-parameter regimes. We aim to scale capacity through MoE-style mixture throughout the LLM pipeline rather than only the FFN. Prior pipeline-level approaches include ParaScale, which introduces virtual tokens and parallel streams but incurs substantial overhead and suffers from homogenized routing and gradient collapse, and AltUp, which uses an auxiliary prediction branch but offers limited adaptivity and slow convergence. We establish that MoE-style mixture layers can be reformulated as variable-kernel dynamic convolutions, where each expert corresponds to a $1{\times}1$ convolutional kernel and routing implements input-conditioned kernel aggregation. Building on this equivalence, we introduce cMoLLM: a convolutionally gated mixture-of-LLMs that routes over end-to-end streams through fully differentiable dynamic convolution. In GPT-2-style models trained on FineWeb, cMoLLM improves language modeling perplexity and downstream GLUE and SQuAD accuracy under matched compute, with better stream utilization, more stable optimization, and favorable scaling compared to ParaScale- and AltUp-style baselines.

Large language models grow more capable as they scale, but the standard Transformer architecture activates every parameter for every input token, meaning that adding capacity inevitably increases computation. Mixture-of-Experts (MoE) methods address this by selectively activating subsets of parameters, yet existing pipeline-level approaches either confine mixing to isolated components or suffer from instability and poor stream utilization. This paper introduces cMoLLM, which extends MoE-style routing across the entire model pipeline by running multiple parallel LLM streams and fusing their outputs through fully differentiable dynamic convolution. A key theoretical contribution is showing that mixture layers are mathematically equivalent to variable-kernel dynamic convolutions, where each stream acts as a convolutional expert and routing corresponds to input-conditioned kernel aggregation. This framing enables end-to-end optimization without the gradient collapse or homogenized routing that plague prior methods. Empirically, cMoLLM consistently improves language modeling perplexity and performance on GLUE and SQuAD benchmarks in GPT-2-style models trained on FineWeb, while exhibiting more balanced stream utilization and favorable scaling behavior relative to ParaScale- and AltUp-style baselines under identical compute budgets.