[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation

Vision-Language Models such as CLIP exhibit strong zero-shot recognition capability by aligning images with textual concepts, yet they often underperform on multi-label recognition where multiple objects co-exist. A key bottleneck is that the CLS token, as a single global visual representation, is insufficient to faithfully encode diverse targets with varying scales, contexts, and co-occurrence patterns. To address this limitation, we present a new multi-label image recognition framework, termed PIAA, which formulates prediction as **Patch-level Inference followed by Adaptive Aggregation. Specifically, we first enhance patch-wise predictions from two complementary perspectives: (i) mitigating semantic entanglement in the visual encoder to obtain more discriminative patch representations, and (ii) learning an unsupervised visual classifier to narrow the vision–language modality gap. We then introduce an adaptive aggregation module that consolidates patch-level scores into the final multi-label prediction. Notably, the entire pipeline is fully Training-free, requiring no gradient updates or parameter fine-tuning. Experiments show that our method achieves strong improvements with minimal extra computation, exceeding a 6\% mAP gain on the challenging NUS-WIDE benchmark over representative baselines.

Modern AI systems are excellent at recognizing images, but they typically summarize a picture into one overarching concept. This works perfectly for a simple portrait of a dog, but struggles in complex, real-world scenes where a dog, a frisbee, a child, and a park bench all appear together. The AI’s "big picture" view simply glosses over the smaller or less obvious details. To solve this, we developed a new approach that asks the AI to act more like a human scanning a messy room. Instead of just looking at the whole image at once, our method breaks the image down into smaller puzzle pieces, or "patches," to evaluate each region independently. Because AI can sometimes get confused by background clutter in these small pieces, we created an automated mathematical filter that reliably identifies the objects in these patches without requiring any time-consuming retraining. Finally, we intelligently combine these close-up details with the original big-picture view. Our approach significantly improves the AI's ability to accurately spot multiple objects of varying sizes in cluttered, everyday images, all while remaining extremely fast, efficient, and easy to deploy.