Deep Ensemble Clustering for Visual Representation Learning

Recent advances in visual representation learning have seen the rise of clustering-based vision backbones, which adopt clustering as a core paradigm for feature extraction. However, existing clustering-based backbones typically rely on a single clustering algorithm, whose inherent inductive bias limits their representational capacity. To address this, we propose EnFormer, which embeds ensemble clustering as a core component of feature extraction. EnFormer structures feature extraction around two steps: (i) Ensemble Generation, where several differentiable base clustering methods are introduced to capture diverse semantic structures; and (ii) Consensus Aggregation, which employs a differentiable mechanism to fuse the results of all base clusterings to reconstruct refined visual features. Extensive experiments show that EnFormer consistently outperforms existing clustering-based backbones across core vision tasks, with higher performance and significantly improved throughput.

Modern computer vision systems learn to understand images by organizing visual information into useful patterns. In this process, clustering algorithms can group similar parts of an image together. For example, in a picture of a boat, one grouping may separate the boat, the sea, and the sky. However, different clustering algorithms may prefer different ways of grouping the same image. Current clustering-based vision backbone models, which serve as the feature extractors of computer vision systems, usually rely on one fixed clustering algorithm. This limits the diversity of visual patterns they can capture. We propose EnFormer, a vision backbone model that combines multiple clustering algorithms. Each algorithm provides a different perspective on the image, and EnFormer learns how to merge these perspectives into a stronger and more reliable visual representation. EnFormer is a general framework for combining different ways of grouping visual information while remaining computationally efficient. It offers a more flexible way to build clustering-based vision backbone models, where different grouping strategies can work together rather than forcing the model to depend on only one.