SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification

Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent "judge'' decoding boosts this process by relaxing verification criteria by accepting draft tokens that may exhibit minor discrepancies from target model output, but existing methods are restricted by their reliance on human annotations or tasks with verifiable ground truths, limiting generalizability across diverse NLP tasks. We propose SelfJudge, which trains judge verifiers via self-supervision of the target model. Our method measures semantic preservation by assessing whether token-substituted responses preserve the meaning of original responses, enabling automatic verifier training across diverse NLP tasks. Our experiments show SelfJudge achieves superior inference-accuracy trade-offs than judge decoding baselines, offering a broadly applicable solution for faster LLM inference.

AI chatbots generate text one word at a time, and each step requires loading the entire model into fast memory — a memory bottleneck that slows down responses. A popular speed-up trick addresses this by having a small, fast model guess several words ahead, which a larger, more accurate model then verifies in a single step. We introduce SelfJudge, which allows the large model to teach itself when a guessed word is good enough by checking whether substituting it preserves the original meaning. Because this process requires no human labels, SelfJudge generalizes across a wide range of tasks and delivers faster responses without compromising answer quality.