Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning

Seyed Kamyar Seyed Ghasemipour · Satoshi Kataoka · Byron David · Daniel Freeman · Shixiang Gu · Igor Mordatch

Hall E #123

Keywords: [ MISC: Transfer, Multitask and Meta-learning ] [ DL: Graph Neural Networks ] [ RL: Deep RL ] [ RL: Online ] [ APP: Robotics ]

[ Abstract ]
[ Paper PDF
Thu 21 Jul 3 p.m. PDT — 5 p.m. PDT
Spotlight presentation: Applications
Thu 21 Jul 10:30 a.m. PDT — noon PDT


Assembly of multi-part physical structures is both a valuable end product for autonomous robotics, as well as a valuable diagnostic task for open-ended training of embodied intelligent agents. We introduce a naturalistic physics-based environment with a set of connectable magnet blocks inspired by children’s toy kits. The objective is to assemble blocks into a succession of target blueprints. Despite the simplicity of this objective, the compositional nature of building diverse blueprints from a set of blocks leads to an explosion of complexity in structures that agents encounter. Furthermore, assembly stresses agents' multi-step planning, physical reasoning, and bimanual coordination. We find that the combination of large-scale reinforcement learning and graph-based policies-- surprisingly without any additional complexity -- is an effective recipe for training agents that not only generalize to complex unseen blueprints in a zero-shot manner, but even operate in a reset-free setting without being trained to do so. Through extensive experiments, we highlight the importance of large-scale training, structured representations, contributions of multi-task vs. single-task learning, as well as the effects of curriculums, and discuss qualitative behaviors of trained agents. Our accompanying project webpage can be found at:

Chat is not available.