PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents

Long-term memory is essential for large language model (LLM) agents operating in complex environments, yet existing memory designs are either task-specific and non-transferable, or task-agnostic but less effective due to low task-relevance and context explosion from raw memory retrieval. We propose PlugMem, a task-agnostic plugin memory module that can be attached to arbitrary LLM agents without task-specific redesign. Motivated by the fact that decision-relevant information is concentrated as abstract knowledge rather than raw experience, we draw on cognitive science to structure episodic memories into a compact, extensible knowledge-centric memory graph that explicitly represents propositional and prescriptive knowledge. This representation enables efficient memory retrieval and reasoning over task-relevant knowledge, rather than verbose raw trajectories, and departs from other graph-based methods like GraphRAG by treating knowledge as the unit of memory access and organization instead of entities or text chunks. We evaluate PlugMem unchanged across three heterogeneous benchmarks (long-horizon conversational question answering, multi-hop knowledge retrieval, and web agent tasks). The results show that PlugMem consistently outperforms task-agnostic baselines and exceeds task-specific memory designs, while also achieving the highest information density under a unified information-theoretic analysis. Code and data are available at https://github.com/TIMAN-group/PlugMem.

AI assistants are increasingly asked to help over long stretches of time, across thousands of past messages, dozens of webpages, or many documents. To stay useful, they need a memory. But today's memory systems force an awkward choice: either store everything the assistant has ever seen, which is bloated and noisy, or hand-craft a custom memory for one narrow task, which breaks the moment the assistant is used somewhere else. We took inspiration from how human memory works. People don't replay every past conversation to recall a friend's food allergy or to remember how to book a flight; we distill experience into two cleaner forms: facts ("knowing that") and procedures ("knowing how"). We built a memory module, PlugMem, that does the same for AI assistants. It reads through raw experience, extracts the facts and procedures worth keeping, and organizes them so the assistant can find what it needs quickly. The same module, without any changes, works across three very different settings: long conversations, fact-finding across many documents, and clicking around websites. In each, it helps the assistant make better decisions while using a tiny fraction of the memory other methods require.