Workshop
1st ICML Workshop on In-Context Learning (ICL @ ICML 2024)
Beyza Ermis · Erin Grant · Frank Hutter · Julien Siems · Noah Hollmann · Jelena Bratulić
Lehar 4
Sat 27 Jul, midnight PDT
In-context learning (ICL) is an emerging capability of large-scale models, including large language models (LLMs) like GPT-3, to acquire new capabilities directly from the context of an input example without separate training or fine-tuning, enabling these models to adapt rapidly to new tasks, datasets, and domains. This workshop brings together diverse perspectives on this new paradigm to assess progress, synthesize best practices, and chart open problems. Core topics will include architectural and other inductive biases enabling in-context skill acquisition, and reliable evaluation of ICL in application domains including reinforcement learning, representation learning, and safe and reliable machine learning.
Schedule
Sat 12:00 a.m. - 12:05 a.m.
|
Opening Remarks
SlidesLive Video |
🔗 |
Sat 12:05 a.m. - 12:45 a.m.
|
Towards Understanding the Modern Alchemy
(
Invited Talk
)
>
SlidesLive Video |
Ekin Akyürek 🔗 |
Sat 12:45 a.m. - 1:25 a.m.
|
What do you need for in-context learning? Data, subcircuits, and dynamics
(
Invited Talk
)
>
SlidesLive Video |
Stephanie Chan 🔗 |
Sat 1:25 a.m. - 2:15 a.m.
|
Poster Session
(
Poster
)
>
|
🔗 |
Sat 2:20 a.m. - 2:30 a.m.
|
A Theoretical Understanding of Self-Correction through In-context Alignment
(
Oral
)
>
link
SlidesLive Video |
Yifei Wang · Yuyang Wu · Zeming Wei · Stefanie Jegelka · Yisen Wang 🔗 |
Sat 2:30 a.m. - 2:40 a.m.
|
Transformers Learn Temporal Difference Methods for In-Context Reinforcement Learning
(
Oral
)
>
link
SlidesLive Video |
Jiuqi Wang · Ethan Blaser · Hadi Daneshmand · Shangtong Zhang 🔗 |
Sat 2:40 a.m. - 2:50 a.m.
|
LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language
(
Oral
)
>
link
SlidesLive Video |
James Requeima · John Bronskill · Dami Choi · Richard E Turner · David Duvenaud 🔗 |
Sat 2:50 a.m. - 3:30 a.m.
|
ICL for Bayesians & TabPFN
(
Invited Talk
)
>
SlidesLive Video |
Samuel Gabriel Müller 🔗 |
Sat 5:00 a.m. - 5:40 a.m.
|
In-Context Deductive Reasoning
(
Invited Talk
)
>
SlidesLive Video |
Seyed Mehran Kazemi 🔗 |
Sat 5:40 a.m. - 6:20 a.m.
|
Exploring Model Expressivity and Optimization Landscape in in-context Learning
(
Invited Talk
)
>
SlidesLive Video |
🔗 |
Sat 6:20 a.m. - 7:15 a.m.
|
Poster Session
(
Poster
)
>
|
🔗 |
Sat 7:15 a.m. - 7:55 a.m.
|
Panel
(
Panel
)
>
SlidesLive Video |
🔗 |
Sat 7:55 a.m. - 8:00 a.m.
|
Closing Remarks
(
Closing Remarks
)
>
SlidesLive Video |
🔗 |
-
|
Improve Temporal Awareness of LLMs for Domain-general Sequential Recommendation ( Poster ) > link | Zhendong Chu · Zichao Wang · Ruiyi Zhang · Yangfeng Ji · Hongning Wang · Tong Sun 🔗 |
-
|
In-Context Principle Learning from Mistakes ( Poster ) > link | Tianjun Zhang · Aman Madaan · Luyu Gao · Steven Zhang · Swaroop Mishra · Yiming Yang · Niket Tandon · Uri Alon 🔗 |
-
|
In-Context Symmetries: Self-Supervised Learning through Contextual World Models ( Poster ) > link | Sharut Gupta · Chenyu Wang · Yifei Wang · Tommi Jaakkola · Stefanie Jegelka 🔗 |
-
|
Localized Zeroth-Order Prompt Optimization ( Poster ) > link | Wenyang Hu · Yao Shu · Zongmin Yu · Zhaoxuan Wu · Xiaoqiang Lin · Zhongxiang Dai · See-Kiong Ng · Bryan Kian Hsiang Low 🔗 |
-
|
Fast Training Dataset Attribution via In-Context Learning ( Poster ) > link | Milad fotouhi · Mohammad Bahadori · Oluwaseyi Feyisetan · Payman Arabshahi · David Heckerman 🔗 |
-
|
In-context learning in presence of spurious correlations ( Poster ) > link | Hrayr Harutyunyan · Rafayel Darbinyan · Samvel Karapetyan · Hrant Khachatrian 🔗 |
-
|
DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning ( Poster ) > link | Zijian Zhou · Xiaoqiang Lin · Xinyi Xu · Alok Prakash · Daniela Rus · Bryan Kian Hsiang Low 🔗 |
-
|
TabMDA: Tabular Manifold Data Augmentation for Any Classifier using Transformers with In-context Subsetting ( Poster ) > link | Andrei Margeloiu · Adrian Bazaga · Nikola Simidjievski · Pietro Lió · Mateja Jamnik 🔗 |
-
|
Probing the Decision Boundaries of In-context Learning in Large Language Models ( Poster ) > link | Siyan Zhao · Tung Nguyen · Aditya Grover 🔗 |
-
|
Many-Shot In-Context Learning in Multimodal Foundation Models ( Poster ) > link | Yixing Jiang · Jeremy Irvin · Ji Wang · Muhammad Ahmed Chaudhry · Jonathan Chen · Andrew Ng 🔗 |
-
|
Many-shot In-Context Learning ( Poster ) > link | Rishabh Agarwal · Avi Singh · Lei Zhang · Bernd Bohnet · Luis Rosias · Stephanie Chan · Biao Zhang · Aleksandra Faust · Hugo Larochelle 🔗 |
-
|
How In-Context Learning Emerges from Training on Unstructured Data: The Role of Co-Occurrence, Positional Information, and Noise Structures ( Poster ) > link | Kevin Christian Wibisono · Yixin Wang 🔗 |
-
|
Automatic Domain Adaptation by Transformers in In-Context Learning ( Poster ) > link | Ryuichiro Hataya · Kota Matsui · Masaaki Imaizumi 🔗 |
-
|
Learning Fast and Slow: Representations for In-Context Weight Modulation ( Poster ) > link | Andrey Zhmoginov · Jihwan Lee · Max Vladymyrov · Mark Sandler 🔗 |
-
|
Transformers are Minimax Optimal Nonparametric In-Context Learners ( Poster ) > link | Juno Kim · Tai Nakamaki · Taiji Suzuki 🔗 |
-
|
Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars ( Poster ) > link | Zhaoxuan Wu · Xiaoqiang Lin · Zhongxiang Dai · Wenyang Hu · Yao Shu · See-Kiong Ng · Patrick Jaillet · Bryan Kian Hsiang Low 🔗 |
-
|
Can Transformers Solve Least Squares to High Precision? ( Poster ) > link | Jerry Liu · Jessica Grogan · Owen Dugan · Simran Arora · Atri Rudra · Christopher Re 🔗 |
-
|
Linear Transformers are Versatile In-Context Learners ( Poster ) > link | Max Vladymyrov · Johannes Von Oswald · Mark Sandler · Rong Ge 🔗 |
-
|
Transformers Can Perform Distributionally-robust Optimisation through In-context Learning ( Poster ) > link | Taeyoung Kim · Hongseok Yang 🔗 |
-
|
In-Context Generalization to New Tasks From Unlabeled Observation Data ( Poster ) > link | Anthony Liang · Pavel Czempin · Yutai Zhou · Stephen Tu · Erdem Biyik 🔗 |
-
|
Universal Self-Consistency for Large Language Models ( Poster ) > link | Xinyun Chen · Renat Aksitov · Uri Alon · JIE REN · Kefan Xiao · Pengcheng Yin · Sushant Prakash · Charles Sutton · Xuezhi Wang · Denny Zhou 🔗 |
-
|
Retrieval & Fine-Tuning for In-Context Tabular Models ( Poster ) > link | Valentin Thomas · Junwei Ma · Rasa Hosseinzadeh · Keyvan Golestan · Guangwei Yu · Maksims Volkovs · Anthony Caterini 🔗 |
-
|
Can Mamba In-Context Learn Task Mixtures? ( Poster ) > link | Yingcong Li · Xupeng Wei · Haonan Zhao · Taigao Ma 🔗 |
-
|
Learning Task Representations from In-Context Learning ( Poster ) > link | Baturay Saglam · Zhuoran Yang · Dionysios Kalogerias · Amin Karbasi 🔗 |
-
|
An In-Context Learning Theoretic Analysis of Chain-of-Thought ( Poster ) > link | Chenxiao Yang · Zhiyuan Li · David Wipf 🔗 |
-
|
Can LLMs predict the convergence of Stochastic Gradient Descent? ( Poster ) > link | Oussama ZEKRI · Abdelhakim Benechehab · Ievgen Redko 🔗 |
-
|
Cross-lingual QA: A Key to Unlocking In-context Cross-lingual Performance ( Poster ) > link | Sunkyoung Kim · Dayeon Ki · Yireun Kim · Jinsik Lee 🔗 |
-
|
LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law ( Poster ) > link | Toni J.B. Liu · Nicolas Boulle · Raphaël Sarfati · Christopher Earls 🔗 |
-
|
In-Context Learning of Energy Functions ( Poster ) > link | Rylan Schaeffer · Mikail Khona · Sanmi Koyejo 🔗 |
-
|
Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment ( Poster ) > link | Max Wilcoxson · Morten Svendgård · Ria Doshi · Dylan Davis · Reya Vir · Anant Sahai 🔗 |
-
|
Can large language models explore in-context? ( Poster ) > link | Akshay Krishnamurthy · Keegan Harris · Dylan Foster · Cyril Zhang · Alex Slivkins 🔗 |
-
|
In-Context Reinforcement Learning Without Optimal Action Labels ( Poster ) > link | Juncheng Dong · Moyang Guo · Ethan Fang · Zhuoran Yang · Vahid Tarokh 🔗 |
-
|
Task Descriptors Help Transformers Learn Linear Models In-Context ( Poster ) > link | Ruomin Huang · Rong Ge 🔗 |
-
|
Transformers as Stochastic Optimizers ( Poster ) > link | Ryuichiro Hataya · Masaaki Imaizumi 🔗 |
-
|
Verbalized Machine Learning: Revisiting Machine Learning with Language Models ( Poster ) > link | Tim Xiao · Robert Bamler · Bernhard Schölkopf · Weiyang Liu 🔗 |
-
|
Fine-grained Analysis of In-context Linear Estimation ( Poster ) > link | Yingcong Li · Ankit Singh Rawat · Samet Oymak 🔗 |