Evaluating Object-Centric Models beyond Object Discovery

Object-centric learning (OCL) aims to learn structured scene representations that support compositional generalization and robustness to out-of-distribution (OOD) data. However, OCL models are often not evaluated regarding these goals. Instead, most prior work focuses on evaluating OCL models solely through object discovery and simple reasoning tasks, such as probing the representation via image classification. We identify two limitations in existing benchmarks: (1) They provide limited insights on the representation usefulness of OCL models, and (2) localization and representation usefulness are assessed using disjoint metrics. To address (1), we use instruction-tuned VLMs as evaluators, enabling scalable benchmarking across diverse VQA datasets to measure how well VLMs leverage OCL representations for complex reasoning tasks. To address (2), we introduce a unified evaluation task and metric that jointly assess localization (where) and representation usefulness (what), thereby eliminating inconsistencies introduced by disjoint evaluation. Finally, we include a simple multi-feature reconstruction baseline as a reference point.

Modern deep learning systems often treat an image as a single whole, whereas humans understand scenes as collections of separate objects and their relationships. For example, a deep learning system should recognize not only that a dog is present in an image, but also where the dog is and how it relates to other objects of the scene. Object-centric learning aims to build models with this kind of structured understanding. However, current evaluations mostly test whether these models can separate objects, without asking whether the separated parts actually help the model understand the image. In this paper, we propose a broader evaluation of object-centric learning methods. We use an image-question-answering testing protocol to test whether object-based representations help these models better understand and reason about images. We also introduce a new metric that evaluates not only whether the model gives the correct answer, but also whether it uses the correct object to produce the answer. Our results show that existing object-centric models are useful, but still have important limitations. We hope this evaluation will help researchers build models that understand images more reliably and in a more human-like way.