Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models

Vision-language models can perform new tasks without parameter updates through in-context learning (ICL), whose core mechanism is utilizing the support set for task induction. In the standard ICL setting, once the task is induced, its decision criterion remains fixed. However, in real-world applications, many tasks exhibit a stable high-level intent, while their decision criteria shift according to specific requirements. Thus, we introduce a new setting, denoted as Criterion-Conditional In-Context Learning (CC-ICL), where models must infer the latent criterion from context and adjust predictions accordingly under fixed task semantics. To evaluate this capability, we propose two complementary metrics, Criterion Invariance and Criterion Sensitivity, capturing the model's robustness and adaptability under criterion shifts. We further construct CC-Bench, a multi-domain benchmark that supports evaluation under the CC-ICL setting. By employing a dual-level data hierarchy, CC-Bench enables legitimate ground-truth variation conditioned on the active criterion even when the task remains fixed. Experiments on CC-Bench reveal that most models exhibit a rigid boundary bias, struggling to align their decisions with the latent criterion. We also find that even a simple multi-criterion training strategy can significantly reduce this bias, improving Criterion Sensitivity and enabling 7B-scale models to surpass proprietary models without degrading general multimodal performance.

When using visual-language models (VLMs) today, a popular trick known as in-context learning is to show them a few examples of what you want. However, for certain tasks, different sets of examples may imply very different standards. We want to study whether models can pick up on these shifting standards from examples alone. We introduce a new way to test this ability, called Criterion-Conditional In-Context Learning, where the task stays the same but the criterion shifts depending on the examples shown. We build a benchmark called CC-Bench, covering multiple real-world domains where the same image and question can have different correct answers depending on the active criterion. Our experiments show that current models tend to apply a fixed decision boundary even when the provided examples clearly suggest a different standard. Encouragingly, a simple training strategy that exposes models to multiple criteria during training can largely fix this rigidity. Our work opens up a new angle on VLM's in-context learning research. By introducing a dedicated evaluation setting and benchmark, we lay the groundwork for future research on building AI models that can truly adapt to diverse user needs.