Visual Persuasion: What Influences Decisions of Vision-Language Models?

The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent's decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents.

AI agents are starting to browse the web for us, potentially deciding which product to buy, which job applicant to shortlist, or which house or hotel to recommend. Many of these are visual judgments based on images. But we know very little about what actually catches an agent's visual attention, and whether it can be swayed by how something is presented rather than what it really is. To find out, we built a system that repeatedly tweaks an image—changing its lighting, background, or styling, but never the actual object—and checks whether these edits make an agent more likely to choose it. By optimizing for the edits that work best, we can map out the hidden visual preferences driving the agents' decisions. Across nine leading models and four tasks, we found these preferences are strong and consistent: simple cosmetic changes can dramatically shift which option an agent picks, and the same edits sway human choices too. Our findings reveal a real vulnerability that allows AI agents to be visually manipulated. Surfacing these blind spots is a first step toward building and governing them safely.