MINIM: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization

Modern LLM-powered autonomous agents increasingly rely on rich user interface (UI) state observations to achieve reliable action grounding in complex digital environments. However, many deployments transmit the full UI state to remote inference servers even when most elements are irrelevant to the current task, which can leak sensitive but unnecessary context such as authentication codes, private notifications, and background application states. We propose MINIM, a trusted local broker that performs privacy-aware minimization on the client side before any observation leaves the device. Grounded in Contextual Integrity (CI), MINIM learns a dual-score representation for each UI element by predicting an inherent sensitivity score ($s$) and a task-conditioned necessity score ($n$). These scores drive a ternary disclosure policy that keeps essential elements, abstracts sensitive attributes when needed, and removes task-irrelevant content. We optimize a CI-aware objective that penalizes necessity errors more strongly on high-risk content, enabling aggressive pruning while preserving task-critical information. Experiments on real-world UI observations derived from WebArena show that MINIM substantially reduces task-irrelevant sensitive leakage while preserving task-critical semantic context and the interactive affordances required for reliable agent actions.

Modern AI agents increasingly interact with users by observing rich interface states, such as the elements on a webpage or desktop. However, current systems often send the entire interface to remote servers, even when most of the information is irrelevant to the task. This can unintentionally expose sensitive data, such as verification codes, private messages, or background activity. In this paper, we introduce MINIM, a system that protects user privacy by filtering interface information directly on the user’s device before it is shared. Inspired by the principle of Contextual Integrity (CI), MINIM learns to evaluate each piece of information along two dimensions: how sensitive it is and how necessary it is for completing the current task. It then selectively keeps, removes, or obscures information based on this assessment. Experiments show that MINIM significantly reduces unnecessary exposure of sensitive information while preserving the critical context needed for AI agents to function effectively. This work takes an important step toward making autonomous AI systems more privacy-aware and trustworthy.