CrossAnchor: Image-Anchored Text Optimization Exposes Blind Spots in Multi-line Defenses of Agentic Systems

Input-side defenses for agentic LLM and VLM deployments are typically evaluated against discrete prompt attacks that trip guardrails or require white-box access to deployed model; multimodal jailbreaks bypass text classifiers but deliver payload as an image, limiting them to vision-enabled models. Neither family optimizes text tokens directly against a non-text embedding anchor. Our method swaps target-model log-likelihood loss of the GCG family for a cross-modal anchor: cosine distance to a non-text representation of the same intent, namely, a typographic image, averaged across an ensemble of independently trained dual-encoders. This helps remove dependence of the optimization signal on any specific target model and producing transfer across architectures and modalities. The optimizer never sees the target model. Search is greedy coordinate-gradient descent over a sparse, randomly selected subset of the prompt's interior tokens that are sparse enough to disrupt guardrail pattern-matching, while the cross-modal loss recovers the steering signal that masking destroys. On prompts drawn from SALAD-Bench, evaluated across 8 sparsity levels and 5 frontier models, Qwen3Guard detection drops from 30\% for unedited prompts to 3.3\% for optimized prompts at 80\% sparsity while optimized ASR at the same sparsity recovers from $\leq 7$\% for randomly initialized prompts to 20–45\% across models. Played as a sparsity cascade, the method succeeds on 86.7–98.3\% of the cohort end-to-end, including 88.3\% on the held-out model. The optimized prompts and end-to-end success envelope form an adversarial corpus and residual-risk benchmark to harden input-side guardrails of agentic systems.