Identifying Latent Concepts and Structures for Generalized Category Discovery

Generalized Category Discovery (GCD) aims to recognize known classes while autonomously discovering novel ones in open-world settings. However, current approaches primarily focus on designing clustering objectives, often overlooking a critical bottleneck: standard vision backbones yield high-rank, entangled token representations that are ill-suited for unsupervised discovery of latent concepts and structures. In this paper, we propose Compositional Primitive Fields (CPF-GCD), a novel representation learning framework that reshapes the feature space to make such latent structure identifiable by enforcing a low-rank compositional organization. Our core hypothesis is that all categories, whether known or novel, can be expressed as compositions and spatial arrangements of a finite set of learnable visual primitives that capture reusable concepts. CPF instantiates this geometric constraint via a spatial field mechanism. Inserted between the backbone and the head, it rewrites noisy patch tokens through low-rank primitive mixtures, effectively decomposing images into reusable atomic parts and their spatial layouts. By explicitly modeling the spatial distribution of primitives, CPF enables novel categories to emerge naturally as new activation patterns over a shared vocabulary. This shifts the focus of representation from merely partitioning global embeddings to constructing a structured and separable primitive field. Extensive experiments demonstrate that CPF serves as a generic, plug-and-play module that consistently boosts performance across diverse GCD baselines, validating that identifying and leveraging low-rank compositional structure is a crucial inductive bias for open-world recognition.

Modern image recognition systems usually work well when they are asked to classify objects they have seen before, but real-world images often contain new categories that were not labeled during training. The goal of Generalized Category Discovery is to help a model keep recognizing known categories while also grouping previously unseen ones in a meaningful way. Many existing methods focus on improving the final clustering or classification step. Our work instead looks earlier in the pipeline, at how visual information is represented before that step. We introduce Compositional Primitive Fields, a lightweight module that organizes image patches using a small set of reusable visual building blocks, such as parts, textures, and local structures. Instead of treating every new category as requiring a completely new representation, our method lets new categories emerge as new combinations and spatial arrangements of these shared building blocks. This makes the visual representation cleaner and easier for existing discovery methods to use. Across several image benchmarks, our module consistently improves different Generalized Category Discovery systems, suggesting that better organizing visual evidence is an important step toward more adaptable open-world recognition.