SafeSpec: Fast and Safe LLM via Dynamic Reflective Sampling

Speculative inference accelerates large language model (LLM) decoding but provides no inherent safety guarantees. Existing safety defenses are largely incompatible with speculative inference: they either introduce additional computation or disrupt the draft–verify mechanism, negating acceleration benefits. This reveals a fundamental incompatibility between current safety methods and speculative decoding. We propose SafeSpec, a safety-aware speculative inference framework that integrates risk estimation directly into the verification process. SafeSpec attaches a lightweight latent safety head to the target model to jointly evaluate semantic validity and safety in a single forward pass. When unsafe generations are detected, SafeSpec applies rollback and safety-guided reflective multi-sampling to recover safe continuations rather than terminating generation. We model jailbreak attacks as distributional shifts over generative trajectories, where adversarial prompts increase the probability of harmful continuations without eliminating safe ones. Under this model, SafeSpec performs risk-aware trajectory recovery within the speculative decoding process. Across multiple models and adversarial benchmarks, SafeSpec achieves a substantially improved safety–efficiency trade-off. On Qwen3-32B, SafeSpec reduces attack success rates by 15\% while preserving a 2.06× inference speedup on benign workloads, demonstrating that speculative acceleration and inference-time safety can be jointly optimized.

Large language models (LLMs) like ChatGPT and Claude can take noticeable time to respond because they generate text one word at a time. A popular speedup trick called "speculative decoding" uses a small, fast AI to draft upcoming words, which a larger, more capable AI then quickly verifies—much like a junior writer drafting sentences for a senior editor to approve. This can double the response speed. However, this shortcut comes with a hidden risk: it has no built-in defense against malicious prompts designed to trick the AI into producing harmful content (so-called "jailbreak attacks"). Worse, the safety techniques people normally use don't work well here—they either slow the AI back down or break the speedup mechanism entirely. We introduce SafeSpec, a method that bakes safety-checking directly into the verification step, so the AI checks both whether a draft is plausible and whether it is safe—at the same time, with no extra cost. When an unsafe response is caught, SafeSpec rewinds and tries a different, safer continuation instead of giving up. Tested against a range of attacks on widely-used open-source models, SafeSpec blocks substantially more harmful outputs while keeping the speed advantage almost intact. This shows that faster AI and safer AI don't have to be a trade-off.