Dynamic Decision Learning: Test-Time Evolution for Abnormality Grounding in Rare Diseases

Clinical abnormality grounding for rare diseases is often hindered by data scarcity, rendering supervised fine-tuning infeasible and single-pass inference highly unstable. Thus, we propose Dynamic Decision Learning (DDL), a framework that enables frozen LVLMs to refine their decisions across language and visual spaces by optimizing instructions and consolidating predictions under visual perturbations, thereby improving localization quality and producing a consensus-based reliability score that quantifies the model’s confidence. Results on brain-imaging benchmarks, including a rare-disease dataset with 281 pathology types across 3B–72B models, show that DDL improves mAP@75 by up to 105% on rare-disease cases and surpasses adaptation baselines and supervised fine-tuning. Moreover, we show that DDL yields stronger calibration between consensus-based reliability scores and localization accuracy under severe distribution shifts and increasing task difficulty. The code is available at https://github.com/compai-lab/2026-ICML-DDL .

Many rare-disease cases involve clinical abnormalities that are difficult for AI systems to locate in medical images, because there are often too few labeled examples for each type of pathology. This makes it impractical to train a specialized model, and existing models may give unstable answers when the question or image changes slightly. In this paper, we introduce Dynamic Decision Learning (DDL), a method that helps existing vision-language models make more reliable localization decisions without additional training. Instead of changing the model itself, DDL improves the instructions given to the model and checks whether the model’s answers remain consistent when the image is slightly changed. It then combines multiple answers into a final localization result and uses their agreement as a reliability score. Experiments on brain-imaging datasets, including a rare-disease dataset with 281 pathology types, show that DDL improves the localization of clinical abnormalities and provides reliability scores that better reflect whether the model’s answer is likely to be correct.