Automatic Pruning Discovery for Large Language Models

Large language models (LLMs) have achieved remarkable performance on a wide range of tasks, hindering real-world deployment due to their massive size. Existing pruning methods (e.g., Wanda) tailored for LLMs rely heavily on manual design pruning algorithms, thereby leading to $\textit{huge labor costs}$ and $\textit{requires expert knowledge}$. Furthermore, we are the first to identify the serious \textit{outlier value issue} behind dramatic performance degradation under high pruning ratios that are caused by uniform sparsity, raising an additional concern about how to design adaptive pruning sparsity ideal for LLMs. Can LLMs prune by themselves? In this work, we introduce an affirmative answer by proposing a novel pruning method called $\textbf{AutoPrune}$, which first overcomes expert knowledge limits by leveraging LLMs to design optimal pruning algorithms for themselves automatically without any expert knowledge. Specifically, to mitigate the black-box nature of LLMs, we propose a Graph-driven Chain-of-Thought (GCoT) to optimize prompts, significantly enhancing the reasoning process in learning the pruning algorithm and enabling us to generate pruning algorithms with superior performance and interpretability in the next generation. Finally, grounded in insights of outlier value issue, we introduce Skew-aware Dynamic Sparsity Allocation (SDSA) to overcome the outlier value issue, mitigating performance degradation under high pruning ratios. We conduct extensive experiments on mainstream LLMs benchmarks, demonstrating the superiority of AutoPrune, which consistently excels state-of-the-art competitors. The code is available at: \url{https://anonymous.4open.science/r/AutoPrune}.

Large language models, such as systems used for chatbots and writing assistants, are powerful but expensive to run because they contain enormous numbers of internal components. A common way to make them smaller is to remove parts that appear less important, but deciding what can be safely removed usually requires substantial human expertise and trial-and-error. This work asks whether large language models can help design better ways to shrink themselves. We introduce AutoPrune, a method that uses a language model to propose and test strategies for identifying which parts of another language model can be removed with limited damage. We also find that some parts of a model contain unusually important values, so removing the same fraction of components everywhere can hurt performance badly. To address this, AutoPrune gives extra protection to these sensitive parts while allowing less important parts to be reduced more aggressively. Across several widely used language models and evaluation tasks, AutoPrune produces smaller models that retain better performance than strong existing pruning methods. This can make powerful AI systems easier to deploy on devices or services with limited computing resources.