Timezone: »
We propose a new class of probabilistic neural-symbolic models, that has symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring lesser number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the questions or programs that could lead to a specified answer given an image. Our results on a dataset of compositional questions about SHAPES verify our hypotheses, showing that the model gets better program (and answer) prediction accuracy even in the low data regime, and allows one to probe the coherence and consistency of reasoning performed.
Author Information
Shanmukha Ramakrishna Vedantam (Facebook AI Research)
Karan Desai (Georgia Tech)
Stefan Lee (Georgia Institute of Technology)
Marcus Rohrbach (Facebook AI Research)
Dhruv Batra (Georgia Institute of Technology / Facebook AI Research)
Devi Parikh (Georgia Tech & Facebook AI Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Fri. Jun 14th 01:30 -- 04:00 AM Room Pacific Ballroom #81
More from the Same Authors
-
2023 Poster: Text-To-4D Dynamic Scene Generation »
Uriel Singer · Shelly Sheynin · Adam Polyak · Oron Ashual · Iurii Makarov · Filippos Kokkinos · Naman Goyal · Andrea Vedaldi · Devi Parikh · Justin Johnson · Yaniv Taigman -
2023 Poster: Adaptive Coordination in Social Embodied Rearrangement »
Andrew Szot · Unnat Jain · Dhruv Batra · Zsolt Kira · Ruta Desai · Akshara Rai -
2022 Poster: COAT: Measuring Object Compositionality in Emergent Representations »
Sirui Xie · Ari Morcos · Song-Chun Zhu · Shanmukha Ramakrishna Vedantam -
2022 Spotlight: COAT: Measuring Object Compositionality in Emergent Representations »
Sirui Xie · Ari Morcos · Song-Chun Zhu · Shanmukha Ramakrishna Vedantam -
2021 Poster: CURI: A Benchmark for Productive Concept Learning Under Uncertainty »
Shanmukha Ramakrishna Vedantam · Arthur Szlam · Maximilian Nickel · Ari Morcos · Brenden Lake -
2021 Spotlight: CURI: A Benchmark for Productive Concept Learning Under Uncertainty »
Shanmukha Ramakrishna Vedantam · Arthur Szlam · Maximilian Nickel · Ari Morcos · Brenden Lake -
2019 : Forcing Vision + Language Models To Actually See, Not Just Talk »
Devi Parikh -
2019 Poster: TarMAC: Targeted Multi-Agent Communication »
Abhishek Das · Theophile Gervet · Joshua Romoff · Dhruv Batra · Devi Parikh · Michael Rabbat · Joelle Pineau -
2019 Poster: Trainable Decoding of Sets of Sequences for Neural Sequence Models »
Ashwin Kalyan · Peter Anderson · Stefan Lee · Dhruv Batra -
2019 Oral: TarMAC: Targeted Multi-Agent Communication »
Abhishek Das · Theophile Gervet · Joshua Romoff · Dhruv Batra · Devi Parikh · Michael Rabbat · Joelle Pineau -
2019 Oral: Trainable Decoding of Sets of Sequences for Neural Sequence Models »
Ashwin Kalyan · Peter Anderson · Stefan Lee · Dhruv Batra -
2019 Poster: Counterfactual Visual Explanations »
Yash Goyal · Ziyan Wu · Jan Ernst · Dhruv Batra · Devi Parikh · Stefan Lee -
2019 Oral: Counterfactual Visual Explanations »
Yash Goyal · Ziyan Wu · Jan Ernst · Dhruv Batra · Devi Parikh · Stefan Lee -
2018 Poster: Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations »
Ashwin Kalyan · Stefan Lee · Anitha Kannan · Dhruv Batra -
2018 Oral: Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations »
Ashwin Kalyan · Stefan Lee · Anitha Kannan · Dhruv Batra