Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

Autonomous code agents built on large language models are reshaping software and AI development through tool use, long-horizon reasoning, and self-directed interaction. However, this autonomy introduces a previously unrecognized security risk: agentic interaction fundamentally expands the LLM attack surface, enabling systematic probing and recovery of hidden system prompts that guide model behavior. We identify system prompt extraction as an emergent vulnerability intrinsic to code agents and present JustAsk, a self-evolving framework that autonomously discovers effective extraction strategies through interaction alone. Unlike prior prompt-engineering or dataset-based attacks, JustAsk requires no handcrafted prompts, labeled supervision, or privileged access beyond standard user interaction. It formulates extraction as an online exploration problem, using Upper Confidence Bound-based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration. These skills exploit imperfect system-instruction generalization and inherent tensions between helpfulness and safety. Evaluated on 41 black-box commercial models across multiple providers, JustAsk consistently achieves full or near-complete system prompt recovery, revealing recurring design- and architecture-level vulnerabilities. Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.

Modern AI assistants, from chatbots to coding agents, are steered by a hidden "system prompt": a block of written instructions that sets the assistant's identity, behavior, and safety rules. Companies guard these instructions as trade secrets. Our work shows that this secrecy is far weaker than assumed. We built JustAsk, an automated agent that recovers a target assistant's hidden instructions through ordinary conversation alone, with no special access or training data. It tries different conversational strategies, learns from each attempt which ones work, and adapts to each target on the fly. Tested against 41 commercial AI models, including widely deployed products, JustAsk recovered the hidden instructions nearly every time. This demonstrates that keeping system prompts confidential might not be achievable with today's technology. The risk goes beyond leaked intellectual property: an exposed prompt reveals an assistant's safety rules, which an attacker can study to bypass them. Our findings argue that developers should treat system prompts as visible to determined users, and should build automated defenses that keep pace with these automated attacks.