Timezone: »
Poster
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning
Aleksei Petrenko · Zhehui Huang · Tushar Kumar · Gaurav Sukhatme · Vladlen Koltun
Increasing the scale of reinforcement learning experiments has allowed researchers to achieve unprecedented results in both training sophisticated agents for video games, and in sim-to-real transfer for robotics. Typically such experiments rely on large distributed systems and require expensive hardware setups, limiting wider access to this exciting area of research. In this work we aim to solve this problem by optimizing the efficiency and resource utilization of reinforcement learning algorithms instead of relying on distributed computation. We present the "Sample Factory", a high-throughput training system optimized for a single-machine setting. Our architecture combines a highly efficient, asynchronous, GPU-based sampler with off-policy correction techniques, allowing us to achieve throughput higher than $10^5$ environment frames/second on non-trivial control problems in 3D without sacrificing sample efficiency. We extend Sample Factory to support self-play and population-based training and apply these techniques to train highly capable agents for a multiplayer first-person shooter game. Github: https://github.com/alex-petrenko/sample-factory
Author Information
Aleksei Petrenko (University of Southern California)
Zhehui Huang (University of Southern California)
Tushar Kumar (University of Southern California)
Gaurav Sukhatme (University of Southern California)
Vladlen Koltun (Intel Labs)
More from the Same Authors
-
2020 : Efficient Adaptation for End-to-End Vision-Based Robotic Manipulation »
Gaurav Sukhatme -
2021 Poster: Megaverse: Simulating Embodied Agents at One Million Experiences per Second »
Aleksei Petrenko · Erik Wijmans · Brennan Shacklett · Vladlen Koltun -
2021 Spotlight: Megaverse: Simulating Embodied Agents at One Million Experiences per Second »
Aleksei Petrenko · Erik Wijmans · Brennan Shacklett · Vladlen Koltun -
2021 Poster: Training Graph Neural Networks with 1000 Layers »
Guohao Li · Matthias Müller · Bernard Ghanem · Vladlen Koltun -
2021 Poster: Stabilizing Equilibrium Models by Jacobian Regularization »
Shaojie Bai · Vladlen Koltun · Zico Kolter -
2021 Spotlight: Training Graph Neural Networks with 1000 Layers »
Guohao Li · Matthias Müller · Bernard Ghanem · Vladlen Koltun -
2021 Spotlight: Stabilizing Equilibrium Models by Jacobian Regularization »
Shaojie Bai · Vladlen Koltun · Zico Kolter -
2021 Poster: Efficient Differentiable Simulation of Articulated Bodies »
Yi-Ling Qiao · Junbang Liang · Vladlen Koltun · Ming Lin -
2021 Spotlight: Efficient Differentiable Simulation of Articulated Bodies »
Yi-Ling Qiao · Junbang Liang · Vladlen Koltun · Ming Lin -
2020 Poster: Scalable Differentiable Physics for Learning and Control »
Yi-Ling Qiao · Junbang Liang · Vladlen Koltun · Ming Lin -
2017 Poster: Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning »
Yevgen Chebotar · Karol Hausman · Marvin Zhang · Gaurav Sukhatme · Stefan Schaal · Sergey Levine -
2017 Talk: Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning »
Yevgen Chebotar · Karol Hausman · Marvin Zhang · Gaurav Sukhatme · Stefan Schaal · Sergey Levine