Image Triaging for Budget-Aware Universal Attacks on Vision-Language Models

Vision-Language Models (VLMs) are typically evaluated under adversarial settings where perturbations can be applied to all inputs. In practice, attackers often face budget constraints, making which inputs to attack as critical as how the perturbation is constructed. We study budgeted deployment of targeted universal adversarial perturbations (UAPs), where a fixed perturbation must be selectively applied to maximize attack success under limited access. We propose an image-triaging framework that predicts transfer vulnerability from clean, image-level features, trained using surrogate attack outcomes, and requires no model queries at deployment time. On a road accident monitoring task, across six VLMs and two UAP methods (XTransfer and AnyAttack), our approach substantially improves attack efficiency over random selection, achieving up to 73–97\% ASR at 1\% attack budget. Gains are largest in the low-budget regime, where indiscriminate deployment is highly inefficient. Beyond performance, we find that transfer vulnerability is largely input-intrinsic: simple low-level image statistics capture most of the triaging gains, and models trained with one UAP method generalize to another with minimal degradation. These results suggest that vulnerability signals are shared across attack constructions and can be exploited without access to the victim model.