Robust Vision-Language Models via Manifold-Adversarial Adapters

Vision-language models (VLMs) have progressed rapidly with large-scale high-quality data and adaptation strategies, yet remain brittle under real-world corruptions, where both visual recognition and language-grounded reasoning degrade. Beyond cascaded image restoration, a natural alternative is parameter-efficient adaptation that aligns corrupted features with clean references; however, Euclidean alignment alone is not semantics-preserving and can even harm downstream reasoning. We attribute this to a semantic misalignment gap, where features become geometrically closer while drifting off the in-distribution support on which multimodal reasoning is calibrated. To address this, we propose Manifold-Adversarial Adapters (MAA), parameter-efficient layer-wise modules for a frozen vision encoder that explicitly steer corrupted features back onto the clean in-distribution manifold rather than merely shrinking feature-space distance. MAA combines paired feature self-distillation with a token-level adversarial manifold constraint to prevent off-manifold semantic shortcuts. At inference, only the adapters are retained, enabling single-stage robustness with negligible overhead and avoiding the latency and semantic drift of restoration pipelines. Across benchmarks and corruption settings, MAA consistently improves performance over strong baselines.

AI systems that answer questions about images often work well on clear photos, but can make surprising mistakes when photos are blurry, noisy, compressed, or poorly lit. Simply cleaning the image first, or forcing the model’s internal image signals to match those from clean photos, can sometimes introduce new errors. We propose Manifold-Adversarial Adapters, small add-on modules placed inside the image-processing part of a vision-language model. During training, these modules learn from pairs of clean and degraded images. They not only move degraded images closer to their clean versions inside the model, but also keep the corrected signals in the kind of range the model already knows how to use. At test time, only these small add-ons are used, so the model does not need a separate image-restoration system. In our experiments, this makes the model more reliable on low-quality images—for example, describing details, understanding actions, and reading text—while adding little extra computation.