Panini: Continual Learning in Token Space via Structured Memory

Language models are increasingly used to reason over content they were not trained on, such as new documents, evolving knowledge, and user-specific data. A common approach is retrieval-augmented generation (RAG), which stores verbatim documents externally (as chunks) and retrieves only a relevant subset at inference time for an LLM to reason over. However, this results in inefficient usage of test-time compute (LLM repeatedly reasons over the same documents); moreover, chunk retrieval can inject irrelevant context that increases unsupported generation. We propose a human-like non-parametric continual learning framework, where the base model remains fixed, and learning occurs by integrating each new experience into an external semantic memory state that accumulates and consolidates itself continually. We present PANINI, which realizes this by representing documents as Generative Semantic Workspaces (GSW)—an entity- and event-aware network of question–answer (QA) pairs, sufficient for an LLM to reconstruct the experienced situations and mine latent knowledge via reasoning-grounded inference chains on the network. Given a query, PANINI only traverses the continually-updated GSW (not the verbatim documents or chunks), and retrieves the most likely inference chains. Across six QA benchmarks, PANINI achieves the highest average performance, 5%–7% higher than other competitive baselines, while using 2–30× fewer answer-context tokens, supports fully open-source pipelines, and reduces unsupported answers on curated unanswerable queries. The results show that efficient and accurate structuring of experiences at write time—as achieved by the GSW framework—yields both efficiency and reliability gains at read time. Code is available at https://github.com/roychowdhuryresearch/gsw-memory.

Large language models are powerful, but they do not automatically learn new facts after training. Updating their weights each time new documents arrive can be expensive, slow, and risky, so many systems instead place relevant information in the model’s input along with the question to be answered. This works, but it often makes the model reread long text repeatedly and can include irrelevant information that leads to unsupported answers. In our work, we present a framework called PANINI which is a way to give a fixed language model an external memory that is updated as new documents arrive. Instead of storing documents only as raw text chunks, PANINI converts each document into a structured memory of people, events, and simple question-answer facts, then follows chains through that memory when answering a question. This lets the system retrieve small pieces of evidence that connect across documents, rather than sending large passages to the model. Across several question-answering benchmarks, PANINI answers more accurately than strong retrieval-based systems while using far fewer input tokens. It also better recognizes when the stored evidence is not enough to answer a question and hence reduces hallucinations, or unsupported answers. This suggests that organizing information when it is first stored can make language-model systems cheaper, more reliable, and easier to update without retraining the model.