Timezone: »
Learning representations that can decompose a multi-object scene into its constituent objects and recompose them flexibly is desirable for object-oriented reasoning and planning. Built upon object masks in the pixel space, existing metricsfor objectness can only evaluate generative models with an object-specific “slot” structure. We propose to directly measure compositionality in the representation space as a form of objections, making such evaluations tractable for a widerclass of models. Our metric, COAT (Compositional Object Algebra Test), evaluates if a generic representation exhibits certain geometric properties that underpin object compositionality beyond what is already captured by the raw pixel space. Our experiments on the popular CLEVR (Johnson et.al., 2018) domain reveal that existing disentanglement-based generative models are not as compositional as one might expect, suggesting room for further modeling improvements. We hope our work allows for a unified evaluation of object-centric representations, spanning generative as well as discriminative, self-supervised models.
Author Information
Sirui Xie (UCLA)
Ari Morcos (Facebook AI Research (FAIR))
Song-Chun Zhu (UCLA)
Shanmukha Ramakrishna Vedantam (Facebook AI Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: COAT: Measuring Object Compositionality in Emergent Representations »
Wed. Jul 20th through Thu the 21st Room Hall E #632
More from the Same Authors
-
2022 Poster: Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time »
Mitchell Wortsman · Gabriel Ilharco · Samir Gadre · Rebecca Roelofs · Raphael Gontijo Lopes · Ari Morcos · Hongseok Namkoong · Ali Farhadi · Yair Carmon · Simon Kornblith · Ludwig Schmidt -
2022 Spotlight: Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time »
Mitchell Wortsman · Gabriel Ilharco · Samir Gadre · Rebecca Roelofs · Raphael Gontijo Lopes · Ari Morcos · Hongseok Namkoong · Ali Farhadi · Yair Carmon · Simon Kornblith · Ludwig Schmidt -
2022 Poster: Latent Diffusion Energy-Based Model for Interpretable Text Modelling »
Peiyu Yu · Sirui Xie · Xiaojian Ma · Baoxiong Jia · Bo Pang · Ruiqi Gao · Yixin Zhu · Song-Chun Zhu · Ying Nian Wu -
2022 Spotlight: Latent Diffusion Energy-Based Model for Interpretable Text Modelling »
Peiyu Yu · Sirui Xie · Xiaojian Ma · Baoxiong Jia · Bo Pang · Ruiqi Gao · Yixin Zhu · Song-Chun Zhu · Ying Nian Wu -
2021 : [12:02 - 12:47 PM UTC] Invited Talk 1: Explainable AI: How Machines Gain Justified Trust from Humans »
Song-Chun Zhu -
2021 Poster: CURI: A Benchmark for Productive Concept Learning Under Uncertainty »
Shanmukha Ramakrishna Vedantam · Arthur Szlam · Maximilian Nickel · Ari Morcos · Brenden Lake -
2021 Spotlight: CURI: A Benchmark for Productive Concept Learning Under Uncertainty »
Shanmukha Ramakrishna Vedantam · Arthur Szlam · Maximilian Nickel · Ari Morcos · Brenden Lake -
2021 Poster: ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases »
Stéphane d'Ascoli · Hugo Touvron · Matthew Leavitt · Ari Morcos · Giulio Biroli · Levent Sagun -
2021 Spotlight: ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases »
Stéphane d'Ascoli · Hugo Touvron · Matthew Leavitt · Ari Morcos · Giulio Biroli · Levent Sagun -
2020 Poster: Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning »
Qing Li · Siyuan Huang · Yining Hong · Yixin Chen · Ying Nian Wu · Song-Chun Zhu -
2019 Workshop: Identifying and Understanding Deep Learning Phenomena »
Hanie Sedghi · Samy Bengio · Kenji Hata · Aleksander Madry · Ari Morcos · Behnam Neyshabur · Maithra Raghu · Ali Rahimi · Ludwig Schmidt · Ying Xiao -
2019 Poster: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh -
2019 Oral: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh -
2018 Poster: Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction »
Siyuan Qi · Baoxiong Jia · Song-Chun Zhu -
2018 Oral: Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction »
Siyuan Qi · Baoxiong Jia · Song-Chun Zhu -
2018 Poster: Measuring abstract reasoning in neural networks »
Adam Santoro · Feilx Hill · David GT Barrett · Ari S Morcos · Timothy Lillicrap -
2018 Oral: Measuring abstract reasoning in neural networks »
Adam Santoro · Feilx Hill · David GT Barrett · Ari S Morcos · Timothy Lillicrap