Timezone: »
There is a recent surge of interest in designing deep architectures based on the update steps in traditional algorithms, or learning neural networks to improve and replace traditional algorithms. While traditional algorithms have certain stopping criteria for outputting results at different iterations, many algorithm-inspired deep models are restricted to a fixed-depth'' for all inputs. Similar to algorithms, the optimal depth of a deep architecture may be different for different input instances, either to avoid
over-thinking'', or because we want to compute less for operations converged already. In this paper, we tackle this varying depth problem using a steerable architecture, where a feed-forward deep model and a variational stopping policy are learned together to sequentially determine the optimal number of layers for each input instance. Training such architecture is very challenging. We provide a variational Bayes perspective and design a novel and effective training procedure which decomposes the task into an oracle model learning stage and an imitation stage. Experimentally, we show that the learned deep model along with the stopping policy improves the performances on a diverse set of tasks, including learning sparse recovery, few-shot meta learning, and computer vision tasks.
Author Information
Xinshi Chen (Georgia Institution of Technology)
Hanjun Dai (Google Brain)
Yu Li (King Abdullah University of Science and Technology)
Xin Gao (Kaust)
Le Song (Georgia Institute of Technology)
More from the Same Authors
-
2023 : DISCS: A Benchmark for Discrete Sampling »
Katayoon Goshvadi · Haoran Sun · Xingchao Liu · Azade Nova · Ruqi Zhang · Will Grathwohl · Dale Schuurmans · Hanjun Dai -
2023 Workshop: Sampling and Optimization in Discrete Space »
Haoran Sun · Hanjun Dai · Priyank Jaini · Ruqi Zhang · Ellen Vitercik -
2023 Poster: Revisiting Sampling for Combinatorial Optimization »
Haoran Sun · Katayoon Goshvadi · Azade Nova · Dale Schuurmans · Hanjun Dai -
2023 Poster: Gradient-Free Structured Pruning with Unlabeled Data »
Azade Nova · Hanjun Dai · Dale Schuurmans -
2022 : Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions »
Yu Li -
2022 Poster: Prototype-Anchored Learning for Learning with Imperfect Annotations »
Xiong Zhou · Xianming Liu · Deming Zhai · Junjun Jiang · Xin Gao · Xiangyang Ji -
2022 Spotlight: Prototype-Anchored Learning for Learning with Imperfect Annotations »
Xiong Zhou · Xianming Liu · Deming Zhai · Junjun Jiang · Xin Gao · Xiangyang Ji -
2022 Poster: Marginal Distribution Adaptation for Discrete Sets via Module-Oriented Divergence Minimization »
Hanjun Dai · Mengjiao Yang · Yuan Xue · Dale Schuurmans · Bo Dai -
2022 Spotlight: Marginal Distribution Adaptation for Discrete Sets via Module-Oriented Divergence Minimization »
Hanjun Dai · Mengjiao Yang · Yuan Xue · Dale Schuurmans · Bo Dai -
2021 Poster: Asymmetric Loss Functions for Learning with Noisy Labels »
Xiong Zhou · Xianming Liu · Junjun Jiang · Xin Gao · Xiangyang Ji -
2021 Poster: SpreadsheetCoder: Formula Prediction from Semi-structured Context »
Xinyun Chen · Petros Maniatis · Rishabh Singh · Charles Sutton · Hanjun Dai · Max Lin · Denny Zhou -
2021 Poster: LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs »
Hongyu Ren · Hanjun Dai · Bo Dai · Xinyun Chen · Michihiro Yasunaga · Haitian Sun · Dale Schuurmans · Jure Leskovec · Denny Zhou -
2021 Spotlight: SpreadsheetCoder: Formula Prediction from Semi-structured Context »
Xinyun Chen · Petros Maniatis · Rishabh Singh · Charles Sutton · Hanjun Dai · Max Lin · Denny Zhou -
2021 Spotlight: LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs »
Hongyu Ren · Hanjun Dai · Bo Dai · Xinyun Chen · Michihiro Yasunaga · Haitian Sun · Dale Schuurmans · Jure Leskovec · Denny Zhou -
2021 Spotlight: Asymmetric Loss Functions for Learning with Noisy Labels »
Xiong Zhou · Xianming Liu · Junjun Jiang · Xin Gao · Xiangyang Ji -
2021 Town Hall: Town Hall »
John Langford · Marina Meila · Tong Zhang · Le Song · Stefanie Jegelka · Csaba Szepesvari -
2020 Workshop: Bridge Between Perception and Reasoning: Graph Neural Networks & Beyond »
Jian Tang · Le Song · Jure Leskovec · Renjie Liao · Yujia Li · Sanja Fidler · Richard Zemel · Ruslan Salakhutdinov -
2020 : Opening Remarks: Jian Tang & Le Song »
Jian Tang · Le Song -
2020 Poster: Energy-Based Processes for Exchangeable Data »
Mengjiao Yang · Bo Dai · Hanjun Dai · Dale Schuurmans -
2020 Poster: Retro*: Learning Retrosynthetic Planning with Neural Guided A* Search »
Binghong Chen · Chengtao Li · Hanjun Dai · Le Song -
2020 Poster: Temporal Logic Point Processes »
Shuang Li · Lu Wang · Ruizhi Zhang · xiaofu Chang · Xuqin Liu · Yao Xie · Yuan Qi · Le Song -
2020 Poster: Scalable Deep Generative Modeling for Sparse Graphs »
Hanjun Dai · Azade Nova · Yujia Li · Bo Dai · Dale Schuurmans -
2019 : posters »
Zhengxing Chen · Juan Jose Garau Luis · Ignacio Albert Smet · Aditya Modi · Sabina Tomkins · Riley Simmons-Edler · Hongzi Mao · Alexander Irpan · Hao Lu · Rose Wang · Subhojyoti Mukherjee · Aniruddh Raghu · Syed Arbab Mohd Shihab · Byung Hoon Ahn · Rasool Fakoor · Pratik Chaudhari · Elena Smirnova · Min-hwan Oh · Xiaocheng Tang · Tony Qin · Qingyang Li · Marc Brittain · Ian Fox · Supratik Paul · Xiaofeng Gao · Yinlam Chow · Gabriel Dulac-Arnold · Ofir Nachum · Nikos Karampatziakis · Bharathan Balaji · Supratik Paul · Ali Davody · Djallel Bouneffouf · Himanshu Sahni · Soo Kim · Andrey Kolobov · Alexander Amini · Yao Liu · Xinshi Chen · · Craig Boutilier -
2019 Poster: Particle Flow Bayes' Rule »
Xinshi Chen · Hanjun Dai · Le Song -
2019 Poster: Generative Adversarial User Model for Reinforcement Learning Based Recommendation System »
Xinshi Chen · Shuang Li · Hui Li · Shaohua Jiang · Yuan Qi · Le Song -
2019 Oral: Generative Adversarial User Model for Reinforcement Learning Based Recommendation System »
Xinshi Chen · Shuang Li · Hui Li · Shaohua Jiang · Yuan Qi · Le Song -
2019 Oral: Particle Flow Bayes' Rule »
Xinshi Chen · Hanjun Dai · Le Song -
2018 Poster: Adversarial Attack on Graph Structured Data »
Hanjun Dai · Hui Li · Tian Tian · Xin Huang · Lin Wang · Jun Zhu · Le Song -
2018 Poster: Towards Black-box Iterative Machine Teaching »
Weiyang Liu · Bo Dai · Xingguo Li · Zhen Liu · James Rehg · Le Song -
2018 Poster: SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation »
Bo Dai · Albert Shaw · Lihong Li · Lin Xiao · Niao He · Zhen Liu · Jianshu Chen · Le Song -
2018 Oral: Towards Black-box Iterative Machine Teaching »
Weiyang Liu · Bo Dai · Xingguo Li · Zhen Liu · James Rehg · Le Song -
2018 Oral: Adversarial Attack on Graph Structured Data »
Hanjun Dai · Hui Li · Tian Tian · Xin Huang · Lin Wang · Jun Zhu · Le Song -
2018 Oral: SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation »
Bo Dai · Albert Shaw · Lihong Li · Lin Xiao · Niao He · Zhen Liu · Jianshu Chen · Le Song -
2018 Poster: Learning to Explain: An Information-Theoretic Perspective on Model Interpretation »
Jianbo Chen · Le Song · Martin Wainwright · Michael Jordan -
2018 Poster: Stochastic Training of Graph Convolutional Networks with Variance Reduction »
Jianfei Chen · Jun Zhu · Le Song -
2018 Poster: Learning Steady-States of Iterative Algorithms over Graphs »
Hanjun Dai · Zornitsa Kozareva · Bo Dai · Alex Smola · Le Song -
2018 Oral: Stochastic Training of Graph Convolutional Networks with Variance Reduction »
Jianfei Chen · Jun Zhu · Le Song -
2018 Oral: Learning Steady-States of Iterative Algorithms over Graphs »
Hanjun Dai · Zornitsa Kozareva · Bo Dai · Alex Smola · Le Song -
2018 Oral: Learning to Explain: An Information-Theoretic Perspective on Model Interpretation »
Jianbo Chen · Le Song · Martin Wainwright · Michael Jordan -
2017 Poster: Stochastic Generative Hashing »
Bo Dai · Ruiqi Guo · Sanjiv Kumar · Niao He · Le Song -
2017 Poster: Variational Policy for Guiding Point Processes »
Yichen Wang · Grady Williams · Evangelos Theodorou · Le Song -
2017 Talk: Stochastic Generative Hashing »
Bo Dai · Ruiqi Guo · Sanjiv Kumar · Niao He · Le Song -
2017 Talk: Variational Policy for Guiding Point Processes »
Yichen Wang · Grady Williams · Evangelos Theodorou · Le Song -
2017 Poster: Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs »
Rakshit Trivedi · Hanjun Dai · Yichen Wang · Le Song -
2017 Talk: Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs »
Rakshit Trivedi · Hanjun Dai · Yichen Wang · Le Song -
2017 Poster: Fake News Mitigation via Point Process Based Intervention »
Mehrdad Farajtabar · Jiachen Yang · Xiaojing Ye · Huan Xu · Rakshit Trivedi · Elias Khalil · Shuang Li · Le Song · Hongyuan Zha -
2017 Poster: Iterative Machine Teaching »
Weiyang Liu · Bo Dai · Ahmad Humayun · Charlene Tay · Chen Yu · Linda Smith · James Rehg · Le Song -
2017 Talk: Iterative Machine Teaching »
Weiyang Liu · Bo Dai · Ahmad Humayun · Charlene Tay · Chen Yu · Linda Smith · James Rehg · Le Song -
2017 Talk: Fake News Mitigation via Point Process Based Intervention »
Mehrdad Farajtabar · Jiachen Yang · Xiaojing Ye · Huan Xu · Rakshit Trivedi · Elias Khalil · Shuang Li · Le Song · Hongyuan Zha