Timezone: »

Learning GFlowNets From Partial Episodes For Improved Convergence And Stability
Kanika Madan · Jarrid Rector-Brooks · Maksym Korablyov · Emmanuel Bengio · Moksh Jain · Andrei-Cristian Nica · Tom Bosc · Yoshua Bengio · Nikolay Malkin

Tue Jul 25 08:30 PM -- 08:38 PM (PDT) @ Ballroom C
Generative flow networks (GFlowNets) are a family of algorithms for training a sequential sampler of discrete objects under an unnormalized target density and have been successfully used for various probabilistic modeling tasks. Existing training objectives for GFlowNets are either local to states or transitions, or propagate a reward signal over an entire sampling trajectory. We argue that these alternatives represent opposite ends of a gradient bias-variance tradeoff and propose a way to exploit this tradeoff to mitigate its harmful effects. Inspired by the TD($\lambda$) algorithm in reinforcement learning, we introduce *subtrajectory balance* or SubTB($\lambda$), a GFlowNet training objective that can learn from partial action subsequences of varying lengths. We show that SubTB($\lambda$) accelerates sampler convergence in previously studied and new environments and enables training GFlowNets in environments with longer action sequences and sparser reward landscapes than what was possible before. We also perform a comparative analysis of stochastic gradient dynamics, shedding light on the bias-variance tradeoff in GFlowNet training and the advantages of subtrajectory balance.

Author Information

Kanika Madan (Mil)
Jarrid Rector-Brooks (Mila, Universite de Montreal)
Maksym Korablyov (MILA)
Emmanuel Bengio (McGill University)
Moksh Jain (Mila / Université de Montréal)
Andrei-Cristian Nica (The Faculty of Automatic Control and Computers, University POLITEHNICA of Bucharest)
Tom Bosc (Montreal Institute for Learning Algorithms, University of Montreal, University of Montreal)
Yoshua Bengio (Mila - Quebec AI Institute)
Nikolay Malkin (Mila / Université de Montréal)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors