Granularity-Aware Adaptive Classifier Expansion via Zero-Shot Learning

Zero-shot classifier expansion aims to recognize unseen classes by learning a shared mechanism to map semantics of all classes to classifier weights without access to images. However, existing methods rely on a shared mapping, which is difficult to classify in scenarios containing a mixture of distinct and similar classes, especially with the continuous expansion of classes. Since this mapping prioritizes general attributes for distinct classes while neglecting subtle attributes for similar ones, this granularity mismatch, compounded by sensitivity to noise, induces optimization interference where gradients from distinct classes dominate the learning process. To overcome this limitation, a granularity-aware adaptive framework with interventions is introduced to balance them. Specifically, this method first generates multi-source semantics by intervening on non-causal noise, then discovers latent class structure to separate distinct classes, and finally refine similar classes to synthesize weights with invariance to non-causal noise. The effectiveness is demonstrated through theoretical and empirical analysis in multiple aspects.

People can often recognize a new object from a description, even if they have never seen it before. For example, after hearing that a certain bird has a red crown and black wings, a person may know what visual details to look for. We want machine learning models to have a similar ability: learning to recognize new categories from descriptions, without requiring new images for every category. Our paper studies image-free zero-shot classifier expansion, where classifier weights for unseen categories are synthesized directly from class descriptions. Existing methods usually rely on one shared rule to map all descriptions into classifier weights. However, this shared rule struggles when the expanded categories contain both clearly different classes and highly similar ones. Broad attributes may be enough to distinguish a cat from a cow, but they may fail to separate two bird species that differ only in subtle visual details. We propose a granularity-aware method that generates diverse descriptions, identifies likely confusions among classes, and uses adaptive modules to synthesize more suitable classifiers for different class groups. This work makes visual recognition systems easier to expand, less dependent on annotated images, and better suited to open-world applications.