A Recipe for an Elastic Mixture: One Mixture-of-Experts for Every Resource Budget

We present a single Mixture-of-Experts check- point that exposes a continuous memory–quality knob at serving time, letting researchers trade resident expert memory for quality without retraining or maintaining a family of distilled models. Starting from Qwen3-30B-A3B, we continue-pretraining for a small fraction of the original compute budget under a fixed Bernoulli expert mask. The resulting model matches a no-mask control trained for the same compute at full inference, while degrading far more gracefully at mask rates it was never trained on — substan- tially lower loss across a wide range of held-out masking ratios, where the base model collapses. This means a single deployed checkpoint can be served at a range of expert-memory footprints, shrinking HBM residency on memory-pressured nodes, fitting larger batches or longer contexts in the freed capacity, and allowing the same weights to be co-located with other workloads on a shared accelerator. Looking at the router itself, the mechanism is not redundancy. Per-token routing actually becomes more peaky under masking; what changes is that different tokens prefer different fallbacks, so the batch-level load flattens, yielding a meaningful reduction in max-expert load and a corresponding theoretical throughput gain on top of the memory savings.