Timezone: »

 
Poster
Generating Programmatic Referring Expressions via Program Synthesis
Jiani Huang · Calvin Smith · Osbert Bastani · Rishabh Singh · Aws Albarghouthi · Mayur Naik

Tue Jul 14 07:00 AM -- 07:45 AM & Tue Jul 14 06:00 PM -- 06:45 PM (PDT) @
in Poster Session 1 »

Incorporating symbolic reasoning into machine learning algorithms is a promising approach to improve performance on learning tasks that require logical reasoning. We study the problem of generating a programmatic variant of referring expressions that we call referring relational programs. In particular, given a symbolic representation of an image and a target object in that image, the goal is to generate a relational program that uniquely identifies the target object in terms of its attributes and its relations to other objects in the image. We propose a neurosymbolic program synthesis algorithm that combines a policy neural network with enumerative search to generate such relational programs. The policy neural network employs a program interpreter that provides immediate feedback on the consequences of the decisions made by the policy, and also takes into account the uncertainty in the symbolic representation of the image. We evaluate our algorithm on challenging benchmarks based on the CLEVR dataset, and demonstrate that our approach significantly outperforms several baselines.

Keywords: Algorithms · Deep Learning - Algorithms

Author Information

Jiani Huang (University of Pennsylvania)
Calvin Smith (University of Wisconsin at Madison)
Osbert Bastani (University of Pennsylvania)
Rishabh Singh (Google Brain)
Aws Albarghouthi (University of Wisconsin-Madison)
Mayur Naik (University of Pennsylvania)

More from the Same Authors