Timezone: »
Oral
Trainable Decoding of Sets of Sequences for Neural Sequence Models
Ashwin Kalyan · Peter Anderson · Stefan Lee · Dhruv Batra
Many structured prediction tasks admit multiple correct outputs and so, it is often useful to decode a set of outputs that maximize some task-specific set-level metric. However, retooling standard sequence prediction procedures tailored towards predicting the single best output leads to the decoding of sets containing very similar sequences; failing to capture the variation in the output space. To address this, we propose $\nabla$BS, a trainable decoding procedure that outputs a set of sequences, highly valued according to the metric. Our method tightly integrates the training and decoding phases and further allows for the optimization of the task-specific metric addressing the shortcomings of standard sequence prediction. Further, we discuss the trade-offs of commonly used set-level metrics and motivate a new set-level metric that naturally evaluates the notion of ``capturing the variation in the output space''. Finally, we show results on the image captioning task and find that our model outperforms standard techniques and natural ablations.
Author Information
Ashwin Kalyan (Georgia Tech)
Peter Anderson (Georgia Tech)
Stefan Lee (Georgia Institute of Technology)
Dhruv Batra (Georgia Institute of Technology / Facebook AI Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Trainable Decoding of Sets of Sequences for Neural Sequence Models »
Fri. Jun 14th 01:30 -- 04:00 AM Room Pacific Ballroom #48
More from the Same Authors
-
2020 : Bridging Worlds in Reinforcement Learning with Model-Advantage »
Ashwin Kalyan · Nirbhay Modhe -
2023 Poster: Adaptive Coordination in Social Embodied Rearrangement »
Andrew Szot · Unnat Jain · Dhruv Batra · Zsolt Kira · Ruta Desai · Akshara Rai -
2019 Poster: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh -
2019 Poster: TarMAC: Targeted Multi-Agent Communication »
Abhishek Das · Theophile Gervet · Joshua Romoff · Dhruv Batra · Devi Parikh · Michael Rabbat · Joelle Pineau -
2019 Oral: TarMAC: Targeted Multi-Agent Communication »
Abhishek Das · Theophile Gervet · Joshua Romoff · Dhruv Batra · Devi Parikh · Michael Rabbat · Joelle Pineau -
2019 Oral: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh -
2019 Poster: Counterfactual Visual Explanations »
Yash Goyal · Ziyan Wu · Jan Ernst · Dhruv Batra · Devi Parikh · Stefan Lee -
2019 Oral: Counterfactual Visual Explanations »
Yash Goyal · Ziyan Wu · Jan Ernst · Dhruv Batra · Devi Parikh · Stefan Lee -
2018 Poster: Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations »
Ashwin Kalyan · Stefan Lee · Anitha Kannan · Dhruv Batra -
2018 Oral: Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations »
Ashwin Kalyan · Stefan Lee · Anitha Kannan · Dhruv Batra