Attacking Medical Vision-Language Models with Query-Based Zero-Order Optimization

Medical vision--language models (VLMs) are increasingly integrated into clinical workflows, yet their robustness to adversarial image manipulation remains poorly understood. Unlike text-based attacks that can often be detected through input logging, image-based perturbations introduce a distinct threat surface: bounded manipulations in the model's image input space that remain clinically plausible and resemble benign preprocessing or encoding artifacts can alter model predictions with direct implications for patient care. We show that adversarial attacks on medical VLMs are feasible under a black-box threat model with query access to next-token scores. Using zero-order optimization with a low-frequency prior on the MedXpertQA benchmark, we induce answer flips in restricted multiple-choice settings on two medical VLMs (\texttt{MedGemma 4B} and \texttt{MedGemma 27B}) and a generic VLM (\texttt{Qwen3-VL-Instruct 4B}). Under fixed perturbation budgets (), we achieve attack success rates exceeding 40% on initially correct examples across diverse imaging modalities, including radiology, pathology, and electrocardiography. Attacks are consistently more effective on color images than under luminance-constrained perturbations, reflecting reduced degrees of freedom in the latter setting. These results indicate that restricting access to model weights alone is insufficient to ensure robustness in clinical deployments and motivate defenses beyond access control, such as monitoring exposed inference signals and incorporating safeguards for high-stakes decision support.