ARC-Decode: Accelerated Decoding with Risk-Bounded Acceptance

As larger language models deliver stronger capabilities, their autoregressive inference becomes increasingly expensive. *Speculative decoding* accelerates generation by letting a fast draft propose tokens that the target model verifies in parallel. Yet under sampling ($T>0$), observed speedups consistently lag behind those under greedy decoding, as the classical lossless verification rule tends to over-reject low-risk drafts, leading to lower acceptance rates and limited acceleration. To address this gap, we propose **ARC-Decode** (**A**cceptance with **R**isk **C**ontrol), a training-free method that augments speculative decoding without extra forward passes. ARC-Decode enables \textbf{relaxed} acceptance by identifying drafts whose acceptance preserves the output distribution of the target model, under a risk-controlled criterion based on Jensen--Shannon divergence. It combines confidence-based pre-verification filtering with a risk-bounded acceptance criterion derived from an analytic upper bound on the potential distributional deviation. Integrated into the state-of-the-art EAGLE-3 pipeline, ARC-Decode increases accept length per cycle and reduces verification compute, achieving up to **1.6**$\times$ end-to-end speedup over EAGLE-3 under sampling with negligible quality change across benchmarks.

Large language models generate text one token at a time, which makes them slow to run. Speculative decoding speeds this up by letting a smaller or faster model draft several tokens, while the large model checks whether those tokens can be accepted. In sampling-based generation, this checking step often rejects draft tokens that are different from the large model’s exact choice but would still lead to similar and valid continuations. These unnecessary rejections reduce the speedup. This paper introduces ARC-Decode, a method for faster speculative decoding with controlled risk. ARC-Decode uses a Local Tolerance Score to estimate whether accepting a rejected draft token would cause a meaningful change in the large model’s following behavior. Tokens with low estimated risk can be softly accepted, while higher-risk cases fall back to standard verification. ARC-Decode also removes draft branches that are unlikely to be useful before verification, reducing wasted computation. ARC-Decode does not require retraining the language model or adding extra model calls. Across several models and benchmarks, it accepts longer draft spans and improves generation speed while maintaining comparable output quality.