Workshop
Hardware-aware efficient training (HAET)
Gonçalo Mordido · Yoshua Bengio · Ghouthi BOUKLI HACENE · Vincent Gripon · François Leduc-Primeau · Vahid Partovi Nia · Julie Grollier
Room 327 - 329
Sat 23 Jul, 5:45 a.m. PDT
To reach top-tier performance, deep learning models usually require a large number of parameters and operations, using considerable power and memory. Several methods have been proposed to tackle this problem by leveraging quantization of parameters, pruning, clustering of parameters, decompositions of convolutions, or using distillation. However, most of these works focus mainly on improving efficiency at inference time, disregarding the training cost. In practice, however, most of the energy footprint of deep learning results from training. Hence, this workshop focuses on reducing the training complexity of deep neural networks. Our aim is to encourage submissions specifically concerning the reduction in energy, time, or memory usage at training time. Topics of interest include but are not limited to: (i) compression methods for memory and complexity reduction during training, (ii) energy-efficient hardware architectures, (iii) energy-efficient training algorithms, (iv) novel energy models or energy efficiency training benchmarks, (v) practical applications of low-energy training.
Schedule
Sat 5:45 a.m. - 6:00 a.m.
|
Opening welcome speech
(
Intro
)
>
SlidesLive Video |
🔗 |
Sat 6:00 a.m. - 6:30 a.m.
|
Melika Payvand: Brain-inspired hardware and algorithm co-design for low power online training on the edge
(
Keynote
)
>
SlidesLive Video |
🔗 |
Sat 6:30 a.m. - 7:00 a.m.
|
Alexander Keller: How Computer Graphics advances Hardware Aware Efficient Training
(
Keynote
)
>
SlidesLive Video |
🔗 |
Sat 7:00 a.m. -
|
Not All Lotteries Are Made Equal
(
Poster
)
>
|
Surya Kant Sahu · Sai Mitheran · Somya Suhans Mahapatra 🔗 |
Sat 7:00 a.m. -
|
Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search
(
Poster
)
>
|
Taehyeon Kim · Heesoo Myeong · Se-Young Yun 🔗 |
Sat 7:00 a.m. -
|
Efficient Fine-Tuning of Compressed Language Models with Learners
(
Poster
)
>
|
Danilo Vucetic · Mohammadreza Tayaranian · Maryam Zia · James J. Clark · Brett Meyer · Warren Gross 🔗 |
Sat 7:00 a.m. -
|
Cut Inner Layers: A Structured Pruning Strategy for Efficient U-Net GANs
(
Poster
)
>
|
Bo-Kyeong Kim · Shinkook Choi · Hancheol Park 🔗 |
Sat 7:00 a.m. -
|
A 28nm 8-bit Floating-Point CNN Training Processor with Hardware-Efficient Dynamic Sparsification and 4.7X Training Speedup
(
Poster
)
>
|
Shreyas Kolala Venkataramanaiah · Jian Meng · Han-Sok Suh · Injune Yeo · Jyotishman Saikia · Sai Kiran Cherupally · Yichi Zhang · Zhiru Zhang · Jae-sun Seo 🔗 |
Sat 7:00 a.m. -
|
MobileTL: On-device Transfer Learning with Inverted Residual Blocks
(
Poster
)
>
|
Hung-Yueh Chiang · Natalia Frumkin · Feng Liang · Diana Marculescu 🔗 |
Sat 7:00 a.m. -
|
Low-Bit DNN Training with Hardware-Efficient Stochastic Rounding Unit Design
(
Poster
)
>
|
11 presentersSung-En Chang · Geng Yuan · Alec Lu · Mengshu Sun · Yanyu Li · Xiaolong Ma · Yanyue Xie · Minghai Qin · Xue Lin · Zhenman Fang · Yanzhi Wang |
Sat 7:00 a.m. -
|
Investigating the Not-So-Obvious Effects of Structured Pruning
(
Poster
)
>
|
Hugo Tessier · Vincent Gripon · Mathieu Léonardon · Matthieu Arzel · David Bertrand · Thomas Hannagan 🔗 |
Sat 7:00 a.m. -
|
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
(
Poster
)
>
|
Youhe Jiang · Xupeng Miao · Xiaonan Nie · Bin Cui 🔗 |
Sat 7:00 a.m. -
|
Studying the impact of magnitude pruning on contrastive learning methods
(
Poster
)
>
|
Francesco Corti · Rahim Entezari · Sara Hooker · Davide Bacciu · Olga Saukh 🔗 |
Sat 7:00 a.m. -
|
RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network
(
Poster
)
>
|
Vitaliy Chiley · Vithursan Thangarasa · Abhay Gupta · Anshul Samar · Joel Hestness · Dennis DeCoste 🔗 |
Sat 7:00 a.m. -
|
Finding Structured Winning Tickets with Early Pruning
(
Poster
)
>
|
Udbhav Bamba · Devin Kwok · Gintare Karolina Dziugaite · David Rolnick 🔗 |
Sat 7:00 a.m. - 8:30 a.m.
|
Poster session I: open discussion and coffee break.
(
Poster session
)
>
|
🔗 |
Sat 8:30 a.m. - 9:00 a.m.
|
Damien Querlioz: Memory-Centric Machine Learning
(
Keynote
)
>
SlidesLive Video |
🔗 |
Sat 9:00 a.m. - 10:15 a.m.
|
Lunch
|
🔗 |
Sat 10:15 a.m. - 10:45 a.m.
|
Fabien Cardinaux: DNN Quantization with Mixed Precision and Trained Lookup Tables
(
Keynote
)
>
SlidesLive Video |
🔗 |
Sat 10:45 a.m. - 11:15 a.m.
|
Tien-Ju Yang: Neural Network Design and Training for Efficient On-Device Learning
(
Keynote
)
>
SlidesLive Video |
🔗 |
Sat 11:15 a.m. - 11:45 a.m.
|
Jian Tang: Neural Bellman-Ford Networks: An Efficient and General Path-based Method for Link Prediction based on GNNs
(
Keynote
)
>
SlidesLive Video |
🔗 |
Sat 11:45 a.m. - 12:00 p.m.
|
Best paper award presentation
(
Presentation
)
>
SlidesLive Video |
🔗 |
Sat 12:00 p.m. -
|
Rethinking Pareto Frontier for Performance Evaluation of Deep Neural Networks
(
Poster
)
>
|
Vahid Partovi Nia · Alireza Ghaffari · Mahdi Zolnouri · Yvon Savaria 🔗 |
Sat 12:00 p.m. -
|
GroupBERT: Enhanced Transformer Architecture with Efficient GroupedStructures
(
Poster
)
>
|
Ivan Chelombiev · Daniel Justus · Douglas Orr · Anastasia Dietrich · Frithjof Gressmann · Alexandros Koliousis · Carlo Luschi 🔗 |
Sat 12:00 p.m. -
|
Principal Component Networks: Parameter Reduction Early in Training
(
Poster
)
>
|
Roger Waleffe · Theodoros Rekatsinas 🔗 |
Sat 12:00 p.m. -
|
TT-PINN: A Tensor-Compressed Neural PDE Solver for Edge Computing
(
Poster
)
>
|
Ziyue Liu · Xinling Yu · Zheng Zhang 🔗 |
Sat 12:00 p.m. -
|
Get the Random Number on the fly: A Low-Precision DNN Training Framework using Stochastic Rounding without the Random Number Generator
(
Poster
)
>
|
13 presentersGeng Yuan · Sung-En Chang · Alec Lu · Jun Liu · Yanyu Li · Yushu Wu · Zhenglun Kong · Yanyue Xie · Peiyan Dong · Minghai Qin · Xiaolong Ma · Zhenman Fang · Yanzhi Wang |
Sat 12:00 p.m. -
|
Efficient Training of Deep Equilibrium Models
(
Poster
)
>
|
Bac Nguyen · Lukas Mauch 🔗 |
Sat 12:00 p.m. -
|
Locally Supervised Learning with Periodic Global Guidance
(
Poster
)
>
|
Hasnain Irshad Bhatti · Jaekyun Moon 🔗 |
Sat 12:00 p.m. -
|
Energy-aware Network Operator Search in Deep Neural Networks
(
Poster
)
>
|
Shamma Nasrin 🔗 |
Sat 12:00 p.m. -
|
TrimBERT: Tailoring BERT for Trade-offs
(
Poster
)
>
|
Sharath Nittur Sridhar · Anthony Sarah · Sairam Sundaresan 🔗 |
Sat 12:00 p.m. -
|
QReg: On Regularization Effects of Quantization
(
Poster
)
>
|
MohammadHossein AskariHemmat · Reyhane Askari Hemmat · Alexander Hoffman · Ivan Lazarevich · Ehsan Saboori · Olivier Mastropietro · Sudhakar Sah · Yvon Savaria · Jean-Pierre David 🔗 |
Sat 12:00 p.m. -
|
MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point
(
Poster
)
>
|
Tao Yu · Wentao Guo · Canal Li · Tiancheng Yuan · Christopher De Sa 🔗 |
Sat 12:00 p.m. -
|
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
(
Poster
)
>
|
Tri Dao · Daniel Y Fu · Stefano Ermon · Atri Rudra · Christopher Re 🔗 |
Sat 12:00 p.m. - 1:30 p.m.
|
Poster session II: open discussion and coffee break.
(
Poster session
)
>
|
🔗 |
Sat 1:30 p.m. - 2:15 p.m.
|
Panel
(
Discussion Panel
)
>
SlidesLive Video |
🔗 |
Sat 2:15 p.m. - 2:30 p.m.
|
Closing remarks.
(
Closing message
)
>
SlidesLive Video |
🔗 |