Timezone: »
In recent years, researchers in ML and systems have been working together to bring big models -- such as GPT-3 with 175B parameters -- into research and production. It has been revealed that increasing model sizes can significantly boost ML performance, and even lead to fundamentally new capabilities.
However, experimenting and adopting big models call for new techniques and systems to support their training and inference on big data and large clusters. This tutorial identifies research and practical pain points in model-parallel training and serving. In particular, this tutorial introduces new algorithmic techniques and system architectures for addressing the training and serving of popular big models, such as GPT-3, PaLM, and vision transformers. The tutorial also consists of a session on how to use the latest open-source system toolsets to support the training and serving of big models. Through this tutorial, we hope to lower the technical barrier of using big models in ML research and bring the big models to the masses.
Mon 8:00 a.m. - 8:15 a.m.
|
Trends on using big models with big data
(Talk)
|
Ion Stoica 🔗 |
Mon 8:15 a.m. - 8:30 a.m.
|
New views of ML parallelism: Intra- and Inter-op parallelisms
(Talk)
|
Hao Zhang 🔗 |
Mon 8:30 a.m. - 9:00 a.m.
|
Inter-op parallelism: algorithms and systems
(Talk)
|
Zhuohan Li 🔗 |
Mon 9:00 a.m. - 9:15 a.m.
|
Break and Q&A
(Break)
|
🔗 |
Mon 9:15 a.m. - 9:45 a.m.
|
Intra-op parallelism: algorithms and implementations
(Talk)
|
Lianmin Zheng 🔗 |
Mon 9:45 a.m. - 9:50 a.m.
|
Q&A
|
🔗 |
Mon 9:50 a.m. - 10:05 a.m.
|
Auto parallelization of ML computation
(Talk)
|
Hao Zhang 🔗 |
Mon 10:05 a.m. - 10:20 a.m.
|
Tools for Big Model, Key Takeaways, and Q&A
(Talk)
|
Hao Zhang · Zhuohan Li · Lianmin Zheng 🔗 |
Author Information
Hao Zhang (UC Berkeley)
Lianmin Zheng (UC Berkeley)
Zhuohan Li (UC Berkeley)
Ion Stoica (UC Berkeley)
More from the Same Authors
-
2022 Poster: GACT: Activation Compressed Training for Generic Network Architectures »
Xiaoxuan Liu · Lianmin Zheng · Dequan Wang · Yukuo Cen · Weize Chen · Xu Han · Jianfei Chen · Zhiyuan Liu · Jie Tang · Joseph Gonzalez · Michael Mahoney · Alvin Cheung -
2022 Spotlight: GACT: Activation Compressed Training for Generic Network Architectures »
Xiaoxuan Liu · Lianmin Zheng · Dequan Wang · Yukuo Cen · Weize Chen · Xu Han · Jianfei Chen · Zhiyuan Liu · Jie Tang · Joseph Gonzalez · Michael Mahoney · Alvin Cheung -
2022 Poster: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging »
Shishir G. Patil · Paras Jain · Prabal Dutta · Ion Stoica · Joseph E Gonzalez -
2022 Spotlight: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging »
Shishir G. Patil · Paras Jain · Prabal Dutta · Ion Stoica · Joseph E Gonzalez -
2022 : Tools for Big Model, Key Takeaways, and Q&A »
Hao Zhang · Zhuohan Li · Lianmin Zheng -
2022 : Auto parallelization of ML computation »
Hao Zhang -
2022 : Intra-op parallelism: algorithms and implementations »
Lianmin Zheng -
2022 : Inter-op parallelism: algorithms and systems »
Zhuohan Li -
2022 : New views of ML parallelism: Intra- and Inter-op parallelisms »
Hao Zhang -
2022 : Trends on using big models with big data »
Ion Stoica -
2021 Poster: TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models »
Zhuohan Li · Siyuan Zhuang · Shiyuan Guo · Danyang Zhuo · Hao Zhang · Dawn Song · Ion Stoica -
2021 Poster: ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training »
Jianfei Chen · Lianmin Zheng · Zhewei Yao · Dequan Wang · Ion Stoica · Michael Mahoney · Joseph E Gonzalez -
2021 Oral: ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training »
Jianfei Chen · Lianmin Zheng · Zhewei Yao · Dequan Wang · Ion Stoica · Michael Mahoney · Joseph E Gonzalez -
2021 Spotlight: TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models »
Zhuohan Li · Siyuan Zhuang · Shiyuan Guo · Danyang Zhuo · Hao Zhang · Dawn Song · Ion Stoica -
2020 Poster: Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers »
Zhuohan Li · Eric Wallace · Sheng Shen · Kevin Lin · Kurt Keutzer · Dan Klein · Joseph Gonzalez -
2020 Poster: Variable Skipping for Autoregressive Range Density Estimation »
Eric Liang · Zongheng Yang · Ion Stoica · Pieter Abbeel · Yan Duan · Peter Chen -
2020 Poster: FetchSGD: Communication-Efficient Federated Learning with Sketching »
Daniel Rothchild · Ashwinee Panda · Enayat Ullah · Nikita Ivkin · Ion Stoica · Vladimir Braverman · Joseph E Gonzalez · Raman Arora -
2019 : Invited Talk 6: RLlib: A Platform for Finance Research »
Ion Stoica -
2019 Workshop: AI in Finance: Applications and Infrastructure for Multi-Agent Learning »
Prashant Reddy · Tucker Balch · Michael Wellman · Senthil Kumar · Ion Stoica · Edith Elkind -
2018 Poster: RLlib: Abstractions for Distributed Reinforcement Learning »
Eric Liang · Richard Liaw · Robert Nishihara · Philipp Moritz · Roy Fox · Ken Goldberg · Joseph E Gonzalez · Michael Jordan · Ion Stoica -
2018 Oral: RLlib: Abstractions for Distributed Reinforcement Learning »
Eric Liang · Richard Liaw · Robert Nishihara · Philipp Moritz · Roy Fox · Ken Goldberg · Joseph E Gonzalez · Michael Jordan · Ion Stoica