SpecMD: A Comprehensive Study On Speculative Expert Prefetching

Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model’s parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required. Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop \textbf{SpecMD}, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose \textbf{Least-Stale}, a novel eviction policy that exploits MoE's predictable expert access patterns to reduce collision misses by up to $85\times$ over LRU. With such gains, we achieve over $88\%$ hit rates with up to $34.7\%$ Time-to-first-token (TTFT) reduction on OLMoE at only $5\%$ or $0.6GB$ of VRAM cache capacity.

Modern Language models are huge, and models that are Mixture-of-Experts only use a small piece of itself at a time, akin to opening a few drawers in a giant filing cabinet. To make storing this giant filing cabinet feasible on today's computers, the right drawers need to be kept close at hand. In this paper, we asked: which strategy is best to swapping in and out pieces of this large model such that it still enables normal usage. To this end, we built SpecMD, a testbed that compares strategies. Testing many existing approaches, we found that the usual assumption of keep whatever was used most recently does not match how these models actually reach for their parts. Based on this, we designed Least-Stale, a new strategy that exploits the model's predictable access patterns. With this approach, the model finds what it needs in fast memory 88% of the time and responds up to 34.7% faster using only 0.6GB of memory.