Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision–Language Models

Contrastive vision-language models like CLIP exhibit remarkable zero-shot generalization. However, prompt tuning remains highly sensitive to label noise, as mislabeled samples generate disproportionately large gradients that can overwhelm pre-trained priors. We argue that because CLIP already provides a near-optimal initialization, adaptation should be inherently conservative, particularly against the extreme gradient updates common in noisy settings. To this end, we propose Double-Softmax Prompt Tuning (DSPT), a hyperparameter-free method for intrinsic gradient suppression. By applying a sequential probabilistic normalization, DSPT induces a self-adaptive saturation zone that suppresses gradients from high-error noisy samples while maintaining informative updates. We also provide both theoretical analysis and empirical evidence about how this mechanism achieves adaptive suppression. This design transforms ``gradient vanishing'', traditionally a training bottleneck, into a principled noise-filtering shield for label-noise prompt tuning. Extensive experiments confirm that this simple, drop-in design achieves state-of-the-art robustness across various noisy benchmarks, outperforming methods with complex architectures and handcrafted hyperparameters.

Modern artificial intelligence models that understand both images and text already possess an excellent baseline knowledge of the world. To customize these models for specific tasks, scientists use a quick adjustment process called "prompt tuning." However, this process easily breaks down if the training data contains mistakes, such as an image of a car incorrectly labeled as a dog. Because the AI already knows what a car looks like, these glaring errors cause an internal conflict, generating massive, disruptive learning signals that overwrite and ruin the AI's pre-existing knowledge. To solve this, we developed a simple method called Double-Softmax Prompt Tuning (DSPT). Instead of building complex and slow systems to manually find and delete bad data, our method acts as a built-in mathematical shield. By slightly altering how the AI calculates its updates, it naturally creates a "saturation zone" that automatically ignores overwhelmingly massive learning signals caused by incorrect labels. At the same time, it allows normal, correct information to pass through safely.