Timezone: »

 
Oral
Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh

Thu Jun 13 04:00 PM -- 04:20 PM (PDT) @ Grand Ballroom
in Bayesian Deep Learning »

We propose a new class of probabilistic neural-symbolic models, that has symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring lesser number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the questions or programs that could lead to a specified answer given an image. Our results on a dataset of compositional questions about SHAPES verify our hypotheses, showing that the model gets better program (and answer) prediction accuracy even in the low data regime, and allows one to probe the coherence and consistency of reasoning performed.

Author Information

Shanmukha Ramakrishna Vedantam (Facebook AI Research)
Karan Desai (Georgia Tech)
Stefan Lee (Georgia Institute of Technology)
Marcus Rohrbach (Facebook AI Research)
Dhruv Batra (Georgia Institute of Technology / Facebook AI Research)
Devi Parikh (Georgia Tech & Facebook AI Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors