Workshop

2nd Workshop on Advancing Neural Network Training : Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024)

Julia Gusak · Jean Kossaifi · Alena Shilova · Rocco Sedona · Jan Kautz

Project Page

Abstract

Join HPC and AI experts to learn how to train neural networks at an unprecedented scale with your existing infrastructure

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles

12:00 AM

Coffee & Poster placement

12:00 AM

Welcome speech from organizers Talk

Julia Gusak

Video

12:10 AM

Online Training from Numerical Simulations Invited Talk

Bruno Raffin

Video

12:40 AM

Making device-agnostic ML training and inference easy at scale Invited Talk

Zach Mueller

Video

1:10 AM

Contributed Talks Contributed Talk

Video

1:30 AM

Poster session Poster session

2:30 AM

Enabling extremely fast inference and training performance using dataflow and custom chip Invited Talk

Urmish Thakker

Video

3:00 AM

Contributed Talks Contributed Talk

Video

3:30 AM

Lunch

4:30 AM

Poster session Poster session

5:00 AM

Structured matrices for memory-efficient training and finetuning Invited Talk

Beidi Chen

Video

5:30 AM

Architecting and deploying compute clusters for large language models Invited Talk

Adam DeConinck

Video

6:00 AM

Contributed Talks Contributed Talk

Video

6:20 AM

Best Paper Awards Talk

Jean Kossaifi

Video

6:30 AM

Coffee & Poster session Poster session

7:00 AM

Panel discussion Panel

Adam DeConinck · Zach Mueller · Bruno Raffin · Max Ryabinin · Julia Gusak

Video

7:50 AM

Closing remarks Talk

Video

Optimistic Asynchrony Control: Achieving Synchronous Convergence With Asynchronous Throughput for Embedding Model Training Poster

Roger Waleffe

Link

An Analytical Approach to Enhancing DNN Efficiency and Accuracy Using Approximate Multiplication Poster

Salar Shakibhamedan · Anice Jahanjoo · Amin Aminifar · Nima Amirafshar · Nima TaheriNejad · Axel Jantsch

Link

Memory and Bandwidth are All Your Need for Fully Sharded Data Parallel Poster

J. Wang · Jan Ebert · Oleg Filatov · Stefan Kesselheim

Link

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity Poster

Wentao Guo · Jikai Long · YIMENG ZENG · Zirui Liu · Xinyu Yang · Yide Ran · Jacob Gardner · Osbert Bastani · Chris De Sa · Xiaodong Yu · Beidi Chen · Zhaozhuo Xu

Link

DiLoCo: Distributed Low-Communication Training of Language Models Poster

Arthur Douillard · Qixuan Feng · Andrei Rusu · Rachita Chhaparia · Yani Donchev · Adhiguna Kuncoro · Marc'Aurelio Ranzato · Arthur Szlam · Jiajun Shen

Link

Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates Poster

Cristian Meo · Ksenia Sycheva · Anirudh Goyal · Justin Dauwels

Link

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones Poster

Zhengqing Yuan · Zhaoxu Li · Weiran Huang · Yanfang Ye · Lichao Sun

Link

Multi-objective Differentiable Neural Architecture Search Poster

Rhea Sukthanker · Arber Zela · Benedikt Staffler · Samuel Dooley · Josif Grabocka · Frank Hutter

Link

Resource-constrained Neural Architecture Search on Language Models: A Case Study Poster

Andreas Paraskeva · Joao Reis · Suzan Verberne · Jan Rijn

Link

SatDiffMoE: A Mixture of Estimation Method for Satellite Image Super-resolution with Latent Diffusion Models Poster

Zhaoxu Luo · Bowen Song · Liyue Shen

Link

Accelerating Best-of-N via Speculative Rejection Poster

Ruiqi Zhang · Momin Haider · Ming Yin · Jiahao Qiu · Mengdi Wang · Peter Bartlett · Andrea Zanette

Link

Adaptive Model Pruning in Federated Learning through Loss Exploration Poster

Christian Internò · Elena Raponi · Niki van Stein · Thomas Bäck · Markus Olhofer · Yaochu Jin · CITEC Barbara Hammer

Link

Single Train Multi Deploy on Topology Search Spaces using Kshot-Hypernet Poster

Jingyue Zhuge · Christian Mayr · Anand Subramoney · David Kappel

Link

Enhancing Fine-grained Multi-modal Alignment via Adapters: A Parameter-Efficient Training Framework for Referring Image Segmentation Poster

Zunnan Xu · Jiaqi Huang · Ting Liu · Yong Liu · Haonan Han · Kehong Yuan · Xiu Li

Link

Liouna: Biologically Plausible Learning for Efficient Pre-Training of Transferrable Deep Models Poster

Fady Rezk · Antreas Antoniou · Henry Gouk · Timothy Hospedales

Link

Boolean Logic for Low-Energy Deep Learning Poster

Van Minh NGUYEN · Cristian Ocampo · Aymen Askri · Ba-Hien Tran

Link

Class-aware Initialization of Early Exits for Pre-training Large Language Models Poster

Alperen Gormez · Erdem Koyuncu

Link

Communication Efficient Federated Learning with Differentiated Aggregation Poster

Peyman Gholami · Hulya Seferoglu

Link

Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough Poster

Konstantin Dobler · Gerard de Melo

Link

MoReDrop: Dropout without Dropping Poster

Li Jiang · Duo Li · Yichuan Ding · Xue Liu · Victor Chan

Link

ECO: Efficient Computational Optimization for Exact Machine Unlearning in Deep Neural Networks Poster

Yu-Ting Huang · Pei-Yuan Wu · Chuan-Ju Wang

Link

DrJAX: Scalable and Differentiable MapReduce Primitives in JAX Poster

J K Rush · Zachary Charles · Zachary Garrett · Sean Augenstein · Nicole Mitchell

Link

Variational Stochastic Gradient Descent for Deep Neural Networks Poster

Haotian Chen · Anna Kuzina · Babak Esmaeili · Jakub Tomczak

Link

Coarse-to-Fine Semi-Structured Pruning of Graph Convolutional Networks for Skeleton-based Recognition Poster

Hichem Sahbi

Link

Towards Efficient and Scalable Training of Differentially Private Deep Learning Poster

Sebastian Rodriguez Beltran · Marlon Tobaben · Niki Loppi · Antti Honkela

Link

Lowering PyTorch's Memory Consumption for Selective Differentiation Poster

Samarth Bhatia · Felix Dangel

Link

Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies Poster

Brian Bartoldson · James Diffenderfer · Konstantinos Parasyris · Bhavya Kailkhura

Link

DASH: Warm-Starting Neural Network Training Without Loss of Plasticity Under Stationarity Poster

Baekrok Shin · Junsoo Oh · Hanseul Cho · Chulhee Yun

Link

Efficient Adaptive Federated Optimization Poster

Su Hyeong Lee · Sidharth Sharma · Manzil Zaheer · Tian Li

Link

Fisher-aware Quantization for DETR Detectors with Critical-category Objectives Poster

Huanrui Yang · Yafeng Huang · Zhen Dong · Denis Gudovskiy · Tomoyuki Okuno · Yohei Nakata · Yuan Du · EECS Kurt Keutzer · Shanghang Zhang

Link

Efficient Document Ranking with Learnable Late Interactions Poster

Himanshu Jain · Ziwei Ji · Ankit Singh Rawat · Andreas Veit · Sadeep Jayasumana · Sashank J. Reddi · Aditya Menon · Felix Xinnan Yu

Link

Effective Layer Pruning Through Similarity Metric Perspective Poster

Ian Pons · Bruno L. Yamamoto · Anna Reali · Artur Jordao Lima Correia

Link

Model-Agnostic Graph Dataset Compression with the Tree Mover’s Distance Poster

Mika Jain · Stefanie Jegelka · Ishani Karmarkar · Luana Ruiz · Ellen Vitercik

Link

Scalify: scale propagation for efficient low-precision LLM training Poster

Paul Balanca · Sam Hosegood · Carlo Luschi · Andrew Fitzgibbon

Link

u-μP: The Unit-Scaled Maximal Update Parametrization Poster

Charlie Blake · Constantin Eichenberg · Josef Dean · Lukas Balles · Luke Prince · Björn Deiseroth · Andres Felipe Cruz Salinas · Carlo Luschi · Samuel Weinbach · Douglas Orr

Link

Resolving Discrepancies in Compute-Optimal Scaling of Language Models Contributed Talk & Poster

Tomer Porian · Mitchell Wortsman · Jenia Jitsev · Ludwig Schmidt · Yair Carmon

Link

Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs Contributed Talk & Poster

Ashwinee Panda · Berivan Isik · Xiangyu Qi · Sanmi Koyejo · Tsachy Weissman · Prateek Mittal

Link

Asynchronous Local-SGD Training for Language Modeling Contributed Talk & Poster

Bo Liu · Rachita Chhaparia · Arthur Douillard · Satyen Kale · Andrei Rusu · Jiajun Shen · Arthur Szlam · Marc'Aurelio Ranzato

Link

AdaMeM: Memory Efficient Momentum for Adafactor Contributed Talk & Poster

Nikhil Vyas · Depen Morwani · Sham Kakade

Link

Can LLMs Enhance Performance Prediction for Deep Learning Models? Contributed Talk & Poster

Karthick Panner Selvam · Phitchaya Phothilimthana · Sami Abu-El-Haija · Bryan Perozzi · Mats Brorsson

Link

LoQT: Low Rank Adapters for Quantized Training Contributed Talk & Poster

Sebastian Loeschcke · Mads Toftrup · Michael Kastoryano · Serge Belongie · Vésteinn Snæbjarnarson

Link

SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors Contributed Talk & Poster

Vijay Lingam · Atula Tejaswi · Aditya Vavre · Aneesh Shetty · Gautham Krishna Gudur · Joydeep Ghosh · Eunsol Choi · Alexandros Dimakis · Aleksandar Bojchevski · Sujay Sanghavi

Link