SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safety

Recent advances in foundation models have transformed LLMs from passive conversational systems into autonomous agents capable of reasoning and tool execution. While these capabilities unlock substantial practical value, they also introduce new security risks, as adversaries can manipulate agents into performing harmful actions in real-world environments. Existing defense strategies mitigate such threats but frequently struggle to balance safety and utility, resulting in over-refusal of benign user requests. To mitigate this trade-off, we propose SafeHarbor, a novel framework designed to establish precise decision boundaries for LLM agents. Unlike static guidelines, SafeHarbor extracts context-aware defense rules through enhanced adversarial generation. We design a local hierarchical memory system for dynamic rule injection, offering a training-free, efficient, and plug-and-play solution. Furthermore, we introduce an information entropy-based self-evolution mechanism that continuously optimizes the memory structure through dynamic node splitting and merging. Extensive experiments demonstrate that SafeHarbor achieves state-of-the-art performance on both ambiguous benign tasks and explicit malicious attacks, notably attaining a peak benign utility of 63.6\% on GPT-4o while maintaining a robust refusal rate exceeding 93\% against harmful requests. The source code is publicly available at https://github.com/ljj-cyber/SafeHarbor.

Large language model agents can now use tools such as browsers, file systems, email, and APIs to complete real-world tasks. This makes them useful, but also risky: a malicious user may try to trick an agent into deleting files, leaking private data, or sending harmful messages. Existing safety systems often respond by blocking too much, which can prevent harmless tasks such as backups, debugging, or data analysis. We propose SafeHarbor, a safety framework that helps agents make more precise decisions. Instead of relying only on broad fixed rules, SafeHarbor builds a structured memory of harmful patterns and matching safe exceptions. When a new request arrives, it compares the request with this memory, checks whether it looks clearly safe, clearly harmful, or ambiguous, and only uses deeper reasoning when needed. This allows the agent to block dangerous requests while still allowing legitimate work. Experiments show that SafeHarbor improves protection against harmful agent actions while reducing unnecessary refusals of benign user requests. This makes LLM agents safer and more practical for real-world use.