KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

Self-speculative decoding (SSD) accelerates LLM inference by skipping layers to create an efficient draft model, yet existing methods often rely on static heuristics that ignore the dynamic computational overhead of attention in long-context scenarios. We propose KnapSpec, a training-free framework that reformulates draft model selection as a knapsack problem to maximize tokens-per-time throughput. By decoupling Attention and MLP layers and modeling their hardware-specific latencies as functions of context length, KnapSpec adaptively identifies optimal draft configurations on the fly via a parallel dynamic programming algorithm. Furthermore, we provide the first rigorous theoretical analysis establishing cosine similarity between hidden states as a mathematically sound proxy for the token acceptance rate. This foundation allows our method to maintain high drafting faithfulness while navigating the shifting bottlenecks of real-world hardware. Our experiments on Qwen3 and Llama3 demonstrate that KnapSpec consistently outperforms state-of-the-art SSD baselines, achieving up to 1.47$\times$ wall-clock speedup across various benchmarks. Our plug-and-play approach ensures high-speed inference for long sequences without requiring additional training or compromising the target model's output distribution.

Large Language Models (LLMs) have become remarkably powerful, but their expanding size and long conversational histories make them slow and expensive to run. A popular acceleration technique called Self-Speculative Decoding speeds this up by skipping some neural network layers to generate a quick "draft" text, which is then verified by the full model. However, existing methods rely on fixed, static rules that completely ignore how computer hardware bottlenecks when processing extra-long texts. To solve this, we introduce KnapSpec, a training-free framework that treats draft model building as a classic mathematical optimization puzzle—similar to a knapsack problem where one must pack the most valuable items under a strict weight limit. KnapSpec dynamically measures the hardware speed of different sub-components on the fly, and automatically selects the absolute fastest layer configuration tailored to the current text length. By successfully navigating the shifting bottlenecks of real-world hardware, KnapSpec accelerates text generation by up to 1.47$\times$ without requiring any costly retraining. This plug-and-play approach makes running advanced AI models significantly faster and more resource-efficient for long sequences, while guaranteeing that the final output quality remains completely uncompromised.