ShadowSpec: Towards Zero Speculation Overhead for Substitute Speculative Decoding

Deploying large language models (LLMs) on memory constrained consumer GPUs requires offloading model parameters to CPU memory, where PCIe bandwidth becomes the dominat in- ference bottleneck. While speculative decoding (SD) reduces the number of target model invo- cations, its sequential draft-then-verify pipeline leaves the GPU idle during parameter loading, limiting throughput in offloading environments. We identify that draft model computation and tar- get model parameter loading are heterogeneous, non-contending operations that can be overlapped without resource conflict. Building on this obser- vation, we propose two complementary system- level strategies to overlap draft tree extension with target model parameter loading, converting idle GPU cycles into productive draft computation. Experiments on consumer GPU demonstrate a 17–54% end-to-end speedup over the SubSpec baseline.