TokenDrop: Token-Level Importance-Aware Backward Propagation Skipping for Efficient LLM Fine-Tuning

Despite the success of parameter-efficient fine-tuning (PEFT) methods in reducing parameter-related overhead, fine-tuning large language models (LLMs) is still bottlenecked by significant memory and computational demands. In this paper, we propose **TokenDrop**, a token-level importance-aware backpropagation skipping method that reduces activation memory and accelerates LLM fine-tuning by skipping backward computations for less informative tokens. TokenDrop evaluates token importance based on the magnitude of residual updates during the forward pass, enabling lightweight, gradient-free importance estimation. Furthermore, we introduce cumulative token selection to preserve gradient continuity across layers and lazy selection scheduling that defers token selection to facilitate globally informed importance scoring under memory constraints. Across a range of experiments, TokenDrop achieves up to **42.9**\% reduction in memory usage and up to **1.50**$\times$ training speedup, while preserving accuracy and outperforming existing backpropagation-skipping baselines. The code is available at https://github.com/kimbss470/tokendrop_official.

Large language models are expensive to adapt to new tasks since training requires large amounts of memory and computation for every token in a sentence, even though many tokens contribute little to learning. This makes fine-tuning difficult for researchers and organizations with limited computing resources. We introduce TokenDrop, a method that makes LLM fine-tuning more efficient by focusing training on the most informative tokens while skipping unnecessary computations for less useful ones. Instead of processing every token equally, TokenDrop estimates which parts of the input are likely to matter most for learning and prioritizes them during training. Across language understanding, reasoning, mathematics, and code generation tasks, TokenDrop reduced memory usage by up to 42.9\% and accelerated training by up to 1.5$\times$ while maintaining model quality. By lowering the cost of fine-tuning, our work can help make large language model research and deployment more accessible.