Workshop
Sat Jul 23 05:50 AM -- 02:30 PM (PDT) @ Hall F
The First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward
The past five years have seen rapid progress in large-scale pre-trained models across a variety of domains, such as computer vision, natural language processing, robotics, bioinformatics, etc. Leveraging a huge number of parameters, large-scale pre-trained models are capable of encoding rich knowledge from labeled and/or unlabeled examples. Supervised and self-supervised pre-training have been the two most representative paradigms, through which pre-trained models have demonstrated large benefits on a wide spectrum of downstream tasks. There are also other pre-training paradigms, e.g., meta-learning for few-shot learning, where pre-trained models are trained so that they quickly adapt to solve new tasks. However, there are still many remaining challenges and new opportunities ahead for pre-training, In this workshop, we propose to have the following two foci: (1) Which pre-training methods transfer across different applications/domains, which ones don't, and why? (2) In what settings should we expect pre-training to be effective, compared to learning from scratch?
| Introduction and Opening Remarks | |
| Neural Scaling of Deep Chemical Models (Invited Talk) | |
| Chinchillas, Flamingos, and Gatos: Few-Shot Learning through Pre-training (Invited Talk) | |
| Multimodal Masked Autoencoders Learn Transferable Representations (Oral) | |
| How Neural Networks See, Learn and Forget (Invited Talk) | |
| Program Synthesis, Program Semantics, and Large Language Models (Invited Talk) | |
| Panel Discussion | |
| Exploring the Limits of Large Scale Pre-training (Invited Talk) | |
| Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior (Oral) | |
| Simplifying and Simplifying Self-Supervised Visual Representation Pre-Training (Invited Talk) | |
| Plex: Towards Reliability using Pretrained Large Model Extensions (Oral) | |
| Poster Session | |
| Unified and Efficient Multimodal Pretraining across Vision and Language (Invited Talk) | |
| Benefits and Challenges of Pre-training for Environmental Monitoring (Invited Talk) | |
| Knowledge Distillation for Efficient Sequences of Training Runs (Poster) | |
| Boosting Monolingual Sentence Representation with Large-scale Parallel Translation Datasets (Poster) | |
| ECLIP: Efficient Contrastive Language-Image Pretraining via Ensemble Confidence Learning and Masked Language Modeling (Poster) | |
| Hyper-Representation for Pre-Training and Transfer Learning (Poster) | |
| Similarity of Pre-trained and Fine-tuned Representations (Poster) | |
| Predicting Human Similarity Judgments Using Large Language Models (Poster) | |
| Improved Generalization Bounds for Transfer Learning via Neural Collapse (Poster) | |
| Reinforcement Learning Assisted Layer-wise Fine-Tuning for Transfer Learning (Poster) | |
| Feed-Forward Source-Free Latent Domain Adaptation via Cross-Attention (Poster) | |
| Memorization in NLP Fine-tuning Methods (Poster) | |
| Contrastive Learning Can Find An Optimal Basis For Approximately Invariant Functions (Poster) | |
| Plex: Towards Reliability using Pretrained Large Model Extensions (Poster) | |
| Pretraining a Neural Network before Knowing Its Architecture (Poster) | |
| Flaky Performances when Pre-Training on Relational Databases with a Plan for Future Characterization Efforts (Poster) | |
| Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming (Poster) | |
| Robustness to Adversarial Gradients: A Glimpse Into the Loss Landscape of Contrastive Pre-training (Poster) | |
| Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (Poster) | |
| On Combining Global and Localized Self-Supervised Models of Speech (Poster) | |
| Vote for Nearest Neighbors Meta-Pruning of Self-Supervised Networks (Poster) | |
| How well do contrastively trained models transfer? (Poster) | |
| Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior (Poster) | |
| Investigating Why Contrastive Learning Benefits Robustness against Label Noise (Poster) | |
| Enhancing Multi-hop Connectivity for Graph Convolutional Networks (Poster) | |
| Pre-Training on a Data Diet: Identifying Sufficient Examples for Early Training (Poster) | |
| Leader-based Pre-training Framework for Cooperative Multi-Agent Reinforcement Learning (Poster) | |
| Is Self-Supervised Contrastive Learning More Robust Than Supervised Learning? (Poster) | |
| Multimodal Masked Autoencoders Learn Transferable Representations (Poster) | |
| How robust are pre-trained models to distribution shift? (Poster) | |
| Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet? (Poster) | |
| Learning Large-scale Universal User Representation with Sparse Mixture of Experts (Poster) | |
| On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning (Poster) | |
| Non-Markovian Policies for Unsupervised Reinforcement Learning in Multiple Environments (Poster) | |
| Efficient Task Adaptation by Mixing Discovered Skills (Poster) | |
| Pixel-level Correspondence for Self-Supervised Learning from Video (Poster) | |
| PARS-Push: Personalized, Asynchronous and Robust Decentralized Optimization (Poster) | |
| On the Connection between Pre-training Data Diversity and Robustness (Poster) | |
| What Do We Maximize In Self-Supervised Learning? (Poster) | |
| Training strategies with unlabeled and few labeled examples under 1-pixel attack by combining supervised and self-supervised learning (Poster) | |
| Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (Poster) | |
| Evaluating Self-Supervised Learned Molecular Graphs (Poster) | |
| On the Subspace Structure of Gradient-Based Meta-Learning (Poster) | |
| Self-Destructing Models: Increasing the Costs of Harmful Dual Uses in Foundation Models (Poster) | |
| LAVA: Language Audio Vision Alignment for Pre-Training Transformers on Video Data (Poster) | |
| Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision (Poster) | |
| The Trade-off between Label Efficiency and Universality of Representations from Contrastive Learning (Poster) | |
| Generative Self-training Improves Pre-training for Visual Dialog (Poster) | |
| PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models (Poster) | |
| Manifold Characteristics That Predict Downstream Task Performance (Poster) | |
| Protein Representation Learning by Geometric Structure Pretraining (Poster) | |
| Self-Supervised Time Series Representation Learning with Temporal-Instance Similarity Distillation (Poster) | |
| Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning (Poster) | |
| Energy-Inspired Self-Supervised Pretraining for Vision Models (Poster) | |
Successful Page Load