Skip to yearly menu bar Skip to main content


Workshop

The First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward

Huaxiu Yao ⋅ Hugo Larochelle ⋅ Percy Liang ⋅ Colin Raffel ⋅ Jian Tang ⋅ Ying WEI ⋅ Saining Xie ⋅ Eric Xing ⋅ Chelsea Finn
Project Page

Abstract

Video

Chat is not available.

Schedule

Timezone: Asia/Seoul
10:00 PM
Neural Scaling of Deep Chemical Models
Connor Coley ⋅ Nathan C. Frey
11:00 PM
Multimodal Masked Autoencoders Learn Transferable Representations
Xinyang Geng ⋅ Hao Liu ⋅ Lisa Lee ⋅ Dale Schuurmans ⋅ Sergey Levine ⋅ Pieter Abbeel
12:15 AM
Panel Discussion
3:00 AM
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior
Ravid Shwartz-Ziv ⋅ Micah Goldblum ⋅ Hossein Souri ⋅ Sanyam Kapoor ⋅ Chen Zhu ⋅ Yann LeCun ⋅ Andrew Wilson
3:45 AM
Plex: Towards Reliability using Pretrained Large Model Extensions
Dustin Tran ⋅ Andreas Kirsch ⋅ Balaji Lakshminarayanan ⋅ Huiyi Hu ⋅ Du Phan ⋅ D. Sculley ⋅ Jasper Snoek ⋅ Jeremiah Liu ⋅ JIE REN ⋅ Joost van Amersfoort ⋅ Kehang Han ⋅ Estefany Kelly Buchanan ⋅ Kevin Murphy ⋅ Mark Collier ⋅ Michael Dusenberry ⋅ Neil Band ⋅ Nithum Thain ⋅ Rodolphe Jenatton ⋅ Tim G. J Rudner ⋅ Yarin Gal ⋅ Zachary Nado ⋅ Zelda Mariet ⋅ Zi Wang ⋅ Zoubin Ghahramani
4:00 AM
Poster Session
Efficient Task Adaptation by Mixing Discovered Skills
Eunseok Yang ⋅ JUNGSUB RHIM ⋅ Taesup Kim
Non-Markovian Policies for Unsupervised Reinforcement Learning in Multiple Environments
Pietro Maldini ⋅ Mirco Mutti ⋅ Riccardo De Santi ⋅ Marcello Restelli
On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning
Diane Wagner ⋅ Fabio Ferreira ⋅ Danny Stoll ⋅ Robin Tibor Schirrmeister ⋅ Samuel Gabriel Müller ⋅ Frank Hutter
Learning Large-scale Universal User Representation with Sparse Mixture of Experts
Caigao Jiang ⋅ Siqiao Xue ⋅ James Zhang ⋅ Lingyue Liu ⋅ Zhibo Zhu ⋅ Hongyan Hao
Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?
Nenad Tomasev ⋅ Ioana Bica ⋅ Brian McWilliams ⋅ Lars Buesing ⋅ Razvan Pascanu ⋅ Charles Blundell ⋅ Jovana Mitrovic
How robust are pre-trained models to distribution shift?
Yuge Shi ⋅ Imant Daunhawer ⋅ Julia Vogt ⋅ Phil Torr ⋅ Amartya Sanyal
Multimodal Masked Autoencoders Learn Transferable Representations
Xinyang Geng ⋅ Hao Liu ⋅ Lisa Lee ⋅ Dale Schuurmans ⋅ Sergey Levine ⋅ Pieter Abbeel
Is Self-Supervised Contrastive Learning More Robust Than Supervised Learning?
Yuanyi Zhong ⋅ Haoran Tang ⋅ Junkun Chen ⋅ Jian Peng ⋅ Yu-Xiong Wang
Pixel-level Correspondence for Self-Supervised Learning from Video
Yash Sharma ⋅ Yi Zhu ⋅ Chris Russell ⋅ Thomas Brox
Pre-Training on a Data Diet: Identifying Sufficient Examples for Early Training
Mansheej Paul ⋅ Brett Larsen ⋅ Surya Ganguli ⋅ Jonathan Frankle ⋅ Gintare Karolina Dziugaite
Enhancing Multi-hop Connectivity for Graph Convolutional Networks
Songtao Liu ⋅ Shixiong Jing ⋅ Tong Zhao ⋅ Zengfeng Huang ⋅ Dinghao Wu
Investigating Why Contrastive Learning Benefits Robustness against Label Noise
Yihao Xue ⋅ Kyle Whitecross ⋅ Baharan Mirzasoleiman
Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming
Hanlin Zhang ⋅ Ziyang Li ⋅ Jiani Huang ⋅ Mayur Naik ⋅ Eric Xing
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior
Ravid Shwartz-Ziv ⋅ Micah Goldblum ⋅ Hossein Souri ⋅ Sanyam Kapoor ⋅ Chen Zhu ⋅ Yann LeCun ⋅ Andrew Wilson
How well do contrastively trained models transfer?
M. Moein Shariatnia ⋅ Rahim Entezari ⋅ Mitchell Wortsman ⋅ Olga Saukh ⋅ Ludwig Schmidt
Vote for Nearest Neighbors Meta-Pruning of Self-Supervised Networks
Haiyan Zhao ⋅ Tianyi Zhou ⋅ Guodong Long ⋅ Jing Jiang ⋅ Chengqi Zhang
On Combining Global and Localized Self-Supervised Models of Speech
Sri Harsha Dumpala ⋅ Chandramouli Shama Sastry ⋅ Rudolf Uher ⋅ Sageev Oore
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
Weixin Liang ⋅ Yuhui Zhang ⋅ Yongchan Kwon ⋅ Serena Yeung ⋅ James Zou
Self-Destructing Models: Increasing the Costs of Harmful Dual Uses in Foundation Models
Eric Mitchell ⋅ Peter Henderson ⋅ Christopher Manning ⋅ Dan Jurafsky ⋅ Chelsea Finn
Flaky Performances when Pre-Training on Relational Databases with a Plan for Future Characterization Efforts
Shengchao Liu ⋅ David Vazquez ⋅ Jian Tang ⋅ Pierre-André Noël
Plex: Towards Reliability using Pretrained Large Model Extensions
Dustin Tran ⋅ Andreas Kirsch ⋅ Balaji Lakshminarayanan ⋅ Huiyi Hu ⋅ Du Phan ⋅ D. Sculley ⋅ Jasper Snoek ⋅ Jeremiah Liu ⋅ JIE REN ⋅ Joost van Amersfoort ⋅ Kehang Han ⋅ Estefany Kelly Buchanan ⋅ Kevin Murphy ⋅ Mark Collier ⋅ Michael Dusenberry ⋅ Neil Band ⋅ Nithum Thain ⋅ Rodolphe Jenatton ⋅ Tim G. J Rudner ⋅ Yarin Gal ⋅ Zachary Nado ⋅ Zelda Mariet ⋅ Zi Wang ⋅ Zoubin Ghahramani
Contrastive Learning Can Find An Optimal Basis For Approximately Invariant Functions
Daniel D. Johnson ⋅ Daniel D. Johnson ⋅ Ayoub El Hanchi ⋅ Ayoub El Hanchi ⋅ Chris Maddison ⋅ Chris Maddison
Memorization in NLP Fine-tuning Methods
FatemehSadat Mireshghallah ⋅ FatemehSadat Mireshghallah ⋅ Archit Uniyal ⋅ Archit Uniyal ⋅ Tianhao Wang ⋅ Tianhao Wang ⋅ David Evans ⋅ David Evans ⋅ Taylor Berg-Kirkpatrick ⋅ Taylor Berg-Kirkpatrick
Feed-Forward Source-Free Latent Domain Adaptation via Cross-Attention
Ondrej Bohdal ⋅ Da Li ⋅ Xu Hu ⋅ Timothy Hospedales
On the Subspace Structure of Gradient-Based Meta-Learning
Gustaf Tegnér ⋅ Alfredo Reichlin ⋅ Hang Yin ⋅ Mårten Björkman ⋅ Danica Kragic
Reinforcement Learning Assisted Layer-wise Fine-Tuning for Transfer Learning
Tanvir Mahmud ⋅ Natalia Frumkin ⋅ Diana Marculescu
Improved Generalization Bounds for Transfer Learning via Neural Collapse
Tomer Galanti ⋅ Andras Gyorgy ⋅ Marcus Hutter
Predicting Human Similarity Judgments Using Large Language Models
Raja Marjieh ⋅ Ilia Sucholutsky ⋅ Theodore R Sumers ⋅ Nori Jacoby ⋅ Thomas Griffiths ⋅ Thomas Griffiths
Federated Learning from Pre-Trained Models: A Contrastive Learning Approach
Yue Tan ⋅ Yue Tan ⋅ Guodong Long ⋅ Guodong Long ⋅ Jie Ma ⋅ Jie Ma ⋅ LU LIU ⋅ LU LIU ⋅ Tianyi Zhou ⋅ Tianyi Zhou ⋅ Jing Jiang ⋅ Jing Jiang
Similarity of Pre-trained and Fine-tuned Representations
Thomas Goerttler ⋅ Thomas Goerttler ⋅ Klaus Obermayer
Hyper-Representation for Pre-Training and Transfer Learning
Konstantin Schürholt ⋅ Konstantin Schürholt ⋅ Boris Knyazev ⋅ Boris Knyazev ⋅ Xavier Giro-i-Nieto ⋅ Damian Borth ⋅ Damian Borth
What Do We Maximize In Self-Supervised Learning?
Ravid Shwartz-Ziv ⋅ Ravid Shwartz-Ziv ⋅ Randall Balestriero ⋅ Yann LeCun ⋅ Yann LeCun
ECLIP: Efficient Contrastive Language-Image Pretraining via Ensemble Confidence Learning and Masked Language Modeling
Jue Wang ⋅ Jue Wang ⋅ Haofan Wang ⋅ Haofan Wang ⋅ Weijia Wu ⋅ Weijia Wu ⋅ Jincan Deng ⋅ Jincan Deng ⋅ Yu Lu ⋅ Xiaofeng Guo ⋅ Xiaofeng Guo ⋅ Debing Zhang ⋅ Debing Zhang
Boosting Monolingual Sentence Representation with Large-scale Parallel Translation Datasets
Jue Wang ⋅ Jue Wang ⋅ Haofan Wang ⋅ Haofan Wang ⋅ Xing Wu ⋅ Xing Wu ⋅ Chaochen Gao ⋅ Chaochen Gao ⋅ Debing Zhang
Knowledge Distillation for Efficient Sequences of Training Runs
Xingyu Liu ⋅ Xingyu Liu ⋅ Alexander Leonardi ⋅ Alexander Leonardi ⋅ Lu Yu ⋅ Lu Yu ⋅ Christopher Gilmer-Hill ⋅ Christopher Gilmer-Hill ⋅ Matthew Leavitt ⋅ Matthew Leavitt ⋅ Jonathan Frankle ⋅ Jonathan Frankle
Energy-Inspired Self-Supervised Pretraining for Vision Models
Ze Wang ⋅ Ze Wang ⋅ Jiang Wang ⋅ Jiang Wang ⋅ Zicheng Liu ⋅ Zicheng Liu ⋅ Qiang Qiu ⋅ Qiang Qiu
On the Connection between Pre-training Data Diversity and Robustness
Vivek Ramanujan ⋅ Vivek Ramanujan ⋅ Thao Nguyen ⋅ Thao Nguyen ⋅ Ludwig Schmidt ⋅ Ali Farhadi ⋅ Ali Farhadi
Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning
Zhecheng Yuan ⋅ Zhecheng Yuan ⋅ Zhengrong Xue ⋅ Zhengrong Xue ⋅ Bo Yuan ⋅ Bo Yuan ⋅ Xueqian Wang ⋅ Xueqian Wang ⋅ Yi Wu ⋅ Yi Wu ⋅ Yang Gao ⋅ Yang Gao ⋅ Huazhe Xu ⋅ Huazhe Xu
Self-Supervised Time Series Representation Learning with Temporal-Instance Similarity Distillation
Ainaz Hajimoradlou ⋅ Ainaz Hajimoradlou ⋅ Leila Pishdad ⋅ Leila Pishdad ⋅ Frederick Tung ⋅ Frederick Tung ⋅ Maryna Karpusha ⋅ Maryna Karpusha
Protein Representation Learning by Geometric Structure Pretraining
Zuobai Zhang ⋅ Zuobai Zhang ⋅ Minghao Xu ⋅ Minghao Xu ⋅ Arian Jamasb ⋅ Arian Jamasb ⋅ Vijil Chenthamarakshan ⋅ Vijil Chenthamarakshan ⋅ Aurelie Lozano ⋅ Payel Das ⋅ Payel Das ⋅ Jian Tang ⋅ Jian Tang
Manifold Characteristics That Predict Downstream Task Performance
Ruan van der Merwe ⋅ Ruan van der Merwe ⋅ Gregory Newman ⋅ Gregory Newman ⋅ Etienne Barnard ⋅ Etienne Barnard
PARS-Push: Personalized, Asynchronous and Robust Decentralized Optimization
Mohammad Taha Toghani ⋅ Mohammad Taha Toghani ⋅ Soomin Lee ⋅ Soomin Lee ⋅ Cesar Uribe
Evaluating Self-Supervised Learned Molecular Graphs
Hanchen Wang ⋅ Hanchen Wang ⋅ Shengchao Liu ⋅ Shengchao Liu ⋅ Jean Kaddour ⋅ Jean Kaddour ⋅ Qi Liu ⋅ Qi Liu ⋅ Jian Tang ⋅ Jian Tang ⋅ Matt Kusner ⋅ Matt Kusner ⋅ Joan Lasenby ⋅ Joan Lasenby
PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models
Salehe Erfanian Ebadi ⋅ Salehe Erfanian Ebadi ⋅ Saurav Dhakad ⋅ Saurav Dhakad ⋅ Sanjay Vishwakarma ⋅ Sanjay Vishwakarma ⋅ Chunpu Wang ⋅ Chunpu Wang ⋅ You-Cyuan Jhang ⋅ Maciek Chociej ⋅ Maciek Chociej ⋅ Adam Crespi ⋅ Adam Crespi ⋅ Alex Thaman ⋅ Alex Thaman ⋅ Sujoy Ganguly ⋅ Sujoy Ganguly
Generative Self-training Improves Pre-training for Visual Dialog
Gi-Cheon Kang ⋅ Gi-Cheon Kang ⋅ Sungdong Kim ⋅ Sungdong Kim ⋅ Jin-Hwa Kim ⋅ Jin-Hwa Kim ⋅ Donghyun Kwak ⋅ Donghyun Kwak ⋅ Byoung-Tak Zhang ⋅ Byoung-Tak Zhang
The Trade-off between Label Efficiency and Universality of Representations from Contrastive Learning
Zhenmei Shi ⋅ Zhenmei Shi ⋅ Jiefeng Chen ⋅ Jiefeng Chen ⋅ Kunyang Li ⋅ Kunyang Li ⋅ Jayaram Raghuram ⋅ Jayaram Raghuram ⋅ Xi Wu ⋅ Xi Wu ⋅ Yingyiu Liang ⋅ Yingyiu Liang ⋅ Somesh Jha ⋅ Somesh Jha
Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision
Yufeng Cui ⋅ Yufeng Cui ⋅ Lichen Zhao ⋅ Lichen Zhao ⋅ Feng Liang ⋅ Feng Liang ⋅ Yangguang Li ⋅ Yangguang Li ⋅ Jing Shao ⋅ Jing Shao
LAVA: Language Audio Vision Alignment for Pre-Training Transformers on Video Data
Sumanth Gurram ⋅ Sumanth Gurram ⋅ David Chan ⋅ David Chan ⋅ Andy Fang ⋅ Andy Fang ⋅ John Canny ⋅ John Canny