Less Is More: Elevating RAG via Performance-Driven Context Compression

Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm for improving the timeliness of knowledge updates and the factual accuracy of large language models. However, incorporating a large volume of retrieved documents significantly increases input length, leading to prohibitive computational costs. Existing compression approaches often compromise task performance, primarily due to their reliance on predefined heuristics. These heuristics fail to ensure that the compressed context is conducive to the generation tasks. To address these limitations, we propose CORE-RAG, a novel framework for context compression in RAG systems. CORE eliminates reliance on proxy heuristics through a performance-driven learning framework, which directy utilizes task performance as a feedback signal to iteratively refine the compressor policy. Prior to this optimization process, we incorporate a knowledge distillation phase to initialize the compressor with a robust policy. Extensive experiments demonstrate the superiority of our approach. At a high compression ratio of 3\%, CORE not only avoids performance degradation but also improves the average Exact Match (EM) score by 3.3 points compared to using full documents. Our code is available at https://github.com/ziqiangcui/CORE-RAG-ICML26.

Artificial intelligence models often need to read large amounts of external documents to provide accurate and up-to-date answers. However, processing all this text makes them slow and expensive to run. Current methods try to shrink this reading material, but they often rely on rigid rules that accidentally remove important information, leading to worse answers. To fix this, we created a system that learns how to compress text without losing crucial details. Instead of guessing what is important, it uses a trial-and-error approach, getting rewarded only if the AI can still answer questions correctly using the shortened text. Remarkably, our method can shrink the provided documents down to just 3% of their original size while actually helping the AI give more accurate answers than if it had read the full texts. This makes advanced AI systems significantly faster, cheaper, and more reliable for everyday applications.