AlignedNorm: Prompting Vision–Language Models via Coupled Prompt Field

Prompt learning for vision-language models (VLMs) primarily follows end-to-end or decoupled routes to balance base and new task performance, but suffers a fundamental bottleneck: sample-wise optimization within task-specific feature spaces traps models in local optima, hindering global optimality. To address this, we identify a key insight that VLMs can be prompted within a Coupled Prompt Field-a shared space where base and new tasks are mutually constrained-and present AlignedNorm, which enforces the field coupling. By dynamically aligning the norms of prompts to VLMs' native scale, our method enables joint optimization of both tasks. Without complex designs, our method matches leading decoupled approaches on 15 datasets across 4 experimental settings, offering both a new perspective and a practical solution to the local-optima dilemma in prompt learning.

When adapting vision-language models to downstream tasks, a common problem is that improving performance on seen classes often harms performance on unseen classes. Existing methods mainly address this in two ways: they either regularize the adaptation to prevent the model from changing too much, or use a separate module for seen-class adaptation. The first strategy still faces an unavoidable trade-off, while the second is difficult to use in real-world settings where we usually do not know whether a test sample belongs to a seen or unseen class. This paper introduces AlignedNorm, a simple method for improving this adaptation process. Instead of forcing the fine-tuned features to stay close to the original features, AlignedNorm uses feature norms as a weaker constraint. This helps preserve the relative structure of the model’s feature space while allowing the fine-tuned features more freedom to move. Across 15 datasets and 4 evaluation settings, AlignedNorm achieves strong performance without extra teacher models, separate inference rules, or additional inference cost. These results suggest that controlling feature norms is an effective and efficient way to make vision-language model adaptation more reliable and generalizable.