Do-Prompt: Causal Interventions Meet Variational Prompt Bottlenecks

Multi-modal prompt learning is a parameter-efficient approach to adapting large vision-language models to downstream classification tasks. However, prompts can inadvertently evolve into a high-capacity pathway that encodes environment-dependent spurious correlations, which are predictive only in the source domain and thereby undermine transferability. To address this issue, this paper introduces Do-Prompt, a compress-and-intervene framework that brings together variational bottlenecks and causal interventions for robust prompt tuning. We model prompts as stochastic latent variables and impose a variational prompt bottleneck to explicitly regulate the information transmitted through prompts, effectively mitigating their tendency to memorize spurious nuisance cues. Building on this capacity constraint, we propose lightweight prompt-level interventions by perturbing the environment-related prompt components and enforcing prediction consistency under these do-style perturbations. This synergistic integration encourages reliance on task-stable, invariant semantics rather than spurious prompt content. Notably, Do-Prompt is plug-and-play compatible with existing multi-modal prompt tuning pipelines and introduces negligible computational overhead. Extensive experiments on base-to-novel generalization, cross-dataset transfer, and ImageNet distribution shifts demonstrate consistent performance gains, with particularly notable improvements on datasets exhibiting pronounced domain or texture biases.

Large vision-language models can interpret images using visual and textual cues, but adapting them to new tasks is challenging. Prompt learning provides small guiding signals instead of retraining the full model, yet these prompts can pick up shortcuts from backgrounds, textures, or dataset-specific styles, which fail on new environments. We propose Do-Prompt, which limits the information prompts can carry and perturbs shortcut-related parts during training, encouraging the model to rely on stable visual concepts. Experiments show Do-Prompt improves performance on new classes, datasets, and shifted ImageNet variants, making vision-language models more robust and practical for real-world applications.