LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning

Long-context understanding remains challenging for LLMs due to limited context windows. This paper introduces Long Input Fine-Tuning (LIFT), a framework that improves the long-context performance of arbitrary short-context LLMs by dynamically adapting their parameters to each long input. Instead of endlessly extending context windows to fit longer inputs in context, LIFT stores and absorbs the input in parameters. By fine-tuning long inputs into parameters, LIFT enables short-context LLMs to answer questions even when required information is absent from the inference context, avoiding the quadratic input-length complexity of standard long-context models. Rather than simple continued pretraining on new long contexts, LIFT uses carefully designed LLM-generated synthetic tasks to enhance comprehension beyond memorization. To offset fine-tuning overhead, we design a highly optimized pipeline that reduces Time to First Token (TTFT) to under 10 seconds for 8k context. We further analyze LIFT's strengths and limitations, discuss large-scale deployment feasibility, and highlight future research directions. Implementation is open-sourced at https://github.com/MuLabPKU/LIFT.

Large language models (LLMs) struggle to process very long documents—such as entire books or legal contracts—because they can only process a limited number of words at once, and the computational cost increases sharply as texts grow longer. We developed LIFT, a method that teaches short-context models to understand long documents by converting the document's knowledge into the model's internal parameters, rather than requiring the model to read the entire text every time. LIFT automatically generates practice questions and answers from the document to train the model, ensuring it truly comprehends the content instead of simply memorizing words. This approach allows any LLM to handle documents of unlimited length while remaining computationally efficient. Experiments show that LIFT achieves perfect accuracy on needle-in-haystack tests and significantly outperforms existing methods on complex long-context question-answering benchmarks, making advanced long-document analysis practical and affordable.