Integrated Episodic and Semantic Memory via Modulating Transformer FeedForward Layers

It is widely recognized that, after generative pre-training, Transformer FeedForward layers implicitly function as semantic memory, encoding linguistic and factual knowledge, while the contexts in key–value (KV) cache contain raw events, serving as the source of models' episodic memory. In this work, we show that a same group of Transformer FeedForward-layer parameters can both be semantic and episodic memory, which is retrievable without explicitly attending to the related KV cache. To realize this idea, we introduce Hypermem, a hypernetwork that recurrently maps contexts into targeted updates of FeedForward parameters. We post-train the hypernetwork using continuation and random-access associative memory objectives, eliminating the need for test-time training. Extensive experiments demonstrate that our approach outperforms related methods, including MemoryLLM and generative adapter, on memory retrieval, long-context question answering, and personalization benchmarks, establishing a new state of the art for hypernetwork-based memory mechanisms. Our results suggest that directly bridging data and parameters provides a viable direction for exploring next-generation foundation models with more flexible and persistent memory capabilities.

Large language models can answer questions using two broad kinds of memory. Some knowledge is stored inside the model itself, such as grammar, word meanings, and factual associations learned during training. Other information comes from the text currently given to the model, such as details from a conversation or document. These two forms of memory are usually treated as separate. In this paper, we explore whether the model’s internal memory can also store new, event-like information from recent contexts. We introduce Hypermem, a method that learns how to turn pieces of context into small updates to the model’s internal components. Once stored this way, the information can later be recalled without needing the original context to be present. This allows the model to remember useful details more persistently, while avoiding additional training at test time. We evaluate Hypermem on tasks that require remembering associations, answering questions over long contexts, and adapting to personal information. Across these settings, Hypermem performs better than several related memory methods. Our results suggest that directly connecting new experiences to a model’s internal knowledge may be a promising path toward AI systems with more flexible, long-lasting, and useful memory.