Understanding LoRA as Knowledge Memory: An Empirical Analysis

Continuous knowledge updating for pre-trained large language models (LLMs) is increasingly necessary yet remains challenging. Although inference-time methods like In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) are popular, they face constraints in context budgets, costs, and retrieval fragmentation. Departing from these context-dependent paradigms, this work investigates a parametric approach using Low-Rank Adaptation (LoRA) as a modular knowledge memory. Although few recent works examine this concept, the fundamental mechanics governing its capacity and composability remain largely unexplored. We bridge this gap through the first systematic empirical study mapping the design space of LoRA-based memory, ranging from characterizing storage capacity and optimizing internalization to scaling multi-module systems and evaluating long-context reasoning. Rather than proposing a single architecture, we provide practical guidance on the operational boundaries of LoRA memory. Overall, our findings position LoRA as the complementary axis of memory alongside RAG and ICL, offering distinct advantages.

Large language models are trained once and then their knowledge is essentially frozen, so keeping them up to date with new facts, documents, or personal information is surprisingly hard. The popular fixes work by stuffing the new information into the model's prompt at the moment you ask a question, but this runs into limits on how much text fits, how expensive it gets, and how reliably the right pieces are pulled up. We explored a different idea: instead of feeding knowledge in at question time, store it inside the model using small, swappable add-on modules called LoRA — like attachable memory cards for an AI. We ran the first careful, systematic study of how well these modules actually work as memory, asking how much they can hold, how best to train them, whether many can be combined, and how they handle long passages of reasoning. We also built two new tests, PhoneBook and PaperQA, to probe these properties. We found that LoRA memory is rarely a complete solution on its own, but it is a genuinely useful complement to existing methods — when set up and combined carefully. Our results give practitioners clear guidance on when and how to use it.