SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass

We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLM). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://anonymous.4open.science/r/metalora-734E

Large language models often need new information, such as a document, article, or user-provided context, to answer specific questions. Today, this usually requires either adding the context to every prompt, which can be slow and expensive, or fine-tuning the model, which requires extra training data, time, memory, and storage. We introduce SHINE, a method that quickly turns a piece of context into a small add-on for the language model. Instead of retraining the model, SHINE reads the context once and produces lightweight model updates, called LoRA adapters, in a single pass. These adapters allow the model to answer questions about the context later without needing to repeatedly process the original context. SHINE requires no additional training for each new context, making it fast, efficient, and effective. This makes language models more practical to adapt to new information. In our experiments, SHINE performs well on question-answering tasks while reducing the time, computation, and memory costs associated with standard fine-tuning or directly inputting long contexts. More broadly, SHINE points toward a practical way to personalize and update language models for many different documents, tasks, or users, and opens up new possibilities for continual learning.