Prompt Tuning for CLIP on the Pretrained Manifold

Prompt tuning introduces learnable prompt vectors that adapt pretrained vision-language models to downstream tasks in a parameter-efficient manner. However, under limited supervision, prompt tuning alters pretrained representations and drives downstream features away from the pretrained manifold toward directions that are unfavorable for transfer. This drift degrades generalization. To address this limitation, we propose ManiPT, a framework that performs prompt tuning on the pretrained manifold. ManiPT introduces cosine consistency constraints in both the text and image modalities to confine the learned representations within the pretrained geometric neighborhood. Furthermore, we introduce a structural bias that enforces incremental corrections, guiding the adaptation along transferable directions to mitigate reliance on shortcut learning. From a theoretical perspective, ManiPT alleviates overfitting tendencies under limited data. Our experiments cover four downstream settings: unseen-class generalization, few-shot classification, cross-dataset transfer, and domain generalization. Across these settings, ManiPT achieves higher average performance than baseline methods. Notably, ManiPT provides an explicit perspective on how prompt tuning overfits under limited supervision.

Modern AI models that connect images and text can often learn new image recognition tasks from only a few labeled examples. However, with so little data, they may learn patterns that work well on the small training set but do not carry over well to new classes or new datasets. This paper introduces ManiPT, a method that helps the model adapt to new tasks without moving too far away from the broad visual and language knowledge it learned during its original training. ManiPT keeps the original knowledge as a guide and allows only small useful changes. Tests on many image recognition tasks show that this helps the model perform better when training examples are scarce, when new classes appear, or when images differ from the training examples.