Steal the Patch Size: Adversarially Manipulate Vision Language Models

We present a black-box model-stealing attack that recovers private vision-tokenizer configurations of deployed vision-language models (VLMs), including the visual patch size and input preprocessing pipeline. The key idea is a task-level side channel induced by ViT-style patchification: when a synthetic grid image is aligned with the hidden patch grid, boundary cues are erased at tokenization, causing periodic accuracy drop. By sweeping the grid cell size and measuring these collapses, we infer the patch size; by introducing padding and a consistency-check test, we further identify whether preprocessing is dynamic- or fixed-resolution and recover the target resize resolution. Across open-source Qwen-VL variants and proprietary models including GPT and Claude, we reliably recover tokenizer-related parameters. Finally, we show that such leakage enables preprocessing-aware transfer attacks and model-targeted adversarial manipulation.

Modern vision-language models first chop images into small patches before analyzing them, but this hidden preprocessing can create predictable blind spots. We show that simple colored grid images can reveal private details such as a model’s patch size and image resizing strategy by causing failures whenever grid lines align with hidden patch boundaries. This works on open-source and commercial models, including GPT and Claude variants. Because these leaked details can help build stronger model-specific adversarial attacks, image preprocessing should be treated as a security-relevant part of deployed AI systems.