Learning Sparse Visual Representations via Spatial-Semantic Factorization

Self-supervised learning (SSL) faces a tension between semantic understanding and image reconstruction. High-level semantic SSL methods such as DINO encourage transformation-invariant global representations for augmentation alignment, while reconstruction-oriented methods such as MAE preserve dense feature grids for spatial grounding but often yield weaker semantic abstractions. We introduce STELLAR, a framework that alleviates this tension by factorizing visual features into a low-rank product of semantic concepts and their spatial distributions. This disentanglement allows us to perform augmentation alignment on semantic tokens while maintaining spatial localization through a localization matrix for reconstruction. We demonstrate that as few as 16 sparse tokens under this factorized form can simultaneously support high-quality reconstruction (2.60 FID) and strong semantic transfer (79.10% ImageNet accuracy). Our results highlight STELLAR as a versatile sparse representation that bridges discriminative and generative vision by separating semantic identity from spatial geometry. Code is available at https://github.com/microsoft/STELLAR.

Many AI systems learn from images by turning each image into a large grid of visual features, similar to breaking the image into many small patches. This works well for tasks that need precise location information, such as reconstructing the image, but it is less efficient for tasks that need higher-level understanding, such as recognizing objects or scenes. On the other hand, methods that focus on high-level understanding often lose some of the spatial detail needed for reconstruction. This paper proposes STELLAR, a new way to represent images using a small number of visual concepts together with information about where those concepts appear in the image. For example, instead of representing every patch separately, the model can learn concepts such as object parts, textures, or regions, and then describe how they are arranged spatially. This makes the image representation more compact while still preserving both meaning and location. We show that STELLAR can represent an image using as few as 16 visual tokens, while still supporting both accurate image reconstruction and strong performance on image understanding tasks. The results suggest that separating “what is in the image” from “where it appears” is a promising direction for building more efficient, interpretable, and general-purpose visual AI systems.