PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?

Conversational assistants are increasingly integrating long-term memory with large language models (LLMs). This persistence of memories, e.g., the user is vegetarian, can enhance personalization in future conversations. However, the same persistence can also introduce safety risks that have been largely overlooked. Hence, we introduce \textbf{PersistBench} to measure the extent of these safety risks. We identify two long-term memory-specific risks: \textit{cross-domain leakage}, where LLMs inappropriately inject context from the long-term memories; and \textit{memory-induced sycophancy}, where stored long-term memories insidiously reinforce user biases. We evaluate 18 frontier and open-source LLMs on our benchmark. Our results reveal a surprisingly high failure rate across these LLMs - a median failure rate of $53\%$ on cross-domain samples and $97\%$ on sycophancy samples. To address this, our benchmark encourages the development of more robust and safer long-term memory usage in frontier conversational systems.

AI assistants are able to form long-term memories about users across conversations, ranging from personal preferences to sensitive medical information. These memories can be useful for personalizing responses, such as providing better recipe suggestions by remembering the user is vegetarian. However, as these memories build up over time, they also introduce risks. We study two such risks. First, AI assistants may incorporate memories in the wrong situation, making responses feel intrusive by bringing up irrelevant details unprompted. Second, they can remember user beliefs and reinforce these beliefs even in situations where neutral answers would be more appropriate. We introduce PersistBench, a benchmark to measure when AI assistants should use memories and when they should ignore them. We evaluate 18 leading AI models and find that many struggle with this distinction. Our results show that AI systems need better ways to judge when memories should be used in responses.