Skip to yearly menu bar Skip to main content


Workshop Thu, Jul 9, 2026 • 4:00 PM – 1:00 AM PDT ROOM E1 - E4 (3rd Floor)

High Dimensional Learning Dynamics: the Science of Scaling

Elliot Paquette ⋅ Inbar Seroussi
[ OpenReview] 

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
5:00 PM
Opening Remarks
7:30 PM
Lunch
10:50 PM
Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods
Depen Morwani ⋅ Alex Meterez ⋅ Pranav Nair ⋅ Sham Kakade
10:50 PM
Lightning Talks
10:51 PM
Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
LIU Hanqing ⋅ Jianjun Cao ⋅ Yuanze Li ⋅ Zijian Zhou
10:52 PM
Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
Connall Garrod ⋅ Jonathan Keating ⋅ Christos Thrampoulidis
10:54 PM
How to Scale Mixture-of-Experts: From μP to the Maximally Scale-Stable Parameterization
Leena Chennuru Vankadara ⋅ Moritz Haas ⋅ Luke Hayward ⋅ Sebastian Bordt ⋅ Alessandro Breccia
10:55 PM
Fast Learning Rate Transfer for Gradient Descent in Sketched Linear Regression
Alberto Bietti ⋅ Nikhil Ghosh ⋅ Theodor Misiakiewicz ⋅ Garrett Wen ⋅ Denny Wu
10:56 PM
Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
Lu Miao ⋅ Zeyu Bian ⋅ Kaiyue Wen ⋅ Beining Wu ⋅ Siyu Chen ⋅ Tianhao Wang ⋅ Zhiyuan Li
10:57 PM
Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory
Juno Kim ⋅ Eshaan Nichani ⋅ Denny Wu ⋅ Alberto Bietti ⋅ Jason Lee
10:58 PM
Why Are DMD Students Lazy? Understanding the Copying Behavior in Few-Step Distillation
Shucheng Li ⋅ Iolo Jones ⋅ Alexander Tong ⋅ Michael Bronstein
11:00 PM
Closing Remarks
11:30 PM
Poster Session
On the Mean-field Analysis of Normalized Steepest Descent via Linear Minimization Oracles
Yongtao Wu ⋅ Fanghui Liu ⋅ Taiji Suzuki ⋅ Volkan Cevher
Worker Disagreement Reveals Sharp Directions in Local SGD
Tolga Dimlioglu ⋅ Kristi Topollai ⋅ Anna Choromanska
Edge of Stability Selectively Shapes Learning Across the Data Distribution
Shauna Kwag ⋅ Anakha Ganesh ⋅ Tomaso A Poggio ⋅ Pierfrancesco Beneventano
Dimension-Free Scaling Laws for Invariant Score Matching
Behrooz Tahmasebi ⋅ Melanie Weber
Activation Functions Control Finite-Width Concentration in Wide Neural Networks
Soumya Ganguly ⋅ Nilava Metya ⋅ Alexandre V Morozov ⋅ Anirvan Sengupta
Global Linear Convergence of Inexact TD Under Generalized Smoothness
Alokendu Mazumder ⋅ Ila Ananta ⋅ Punit Rathore
AMUSE: Anytime Muon with Stable Gradient Evaluation
Jueun Kim ⋅ Baekrok Shin ⋅ Jihun Yun ⋅ Beomhan Baek ⋅ Minhak Song ⋅ Chulhee Yun
Optimal Scaling Needs Optimal Norm
Oleg Filatov ⋅ J. Wang ⋅ Jan Ebert ⋅ Stefan Kesselheim
Is your LLM a Sequence Model on the Training History? The Origins and Consequences of Anticipation
Szilvia Ujvary ⋅ Bruno Mlodozeniec ⋅ Jose Miguel Hernandez-Lobato ⋅ Ferenc Huszár ⋅ Dmitrii Krasheninnikov
On Lipschitz Explosion in Deep Neural Networks with Normalization: Consequences for Optimization and Adversarial Robustness
Ashkan Soleymani ⋅ Reyhaneh Hosseinpourkhoshkbari ⋅ Hadi Daneshmand ⋅ Patrick Jaillet
Critical Batch Size for LLM Policy Optimization
Rachit Bansal ⋅ Clara Mohri ⋅ Natalie Abreu ⋅ David Alvarez-Melis ⋅ Sham Kakade
Learning Rates Do Not Transfer Across Double Descent
Dayal Singh Kalra ⋅ Maissam Barkeshli
Mode Collapse Emerges from Low-Rank Biases in the Learning Dynamics of Generative Models
Julian Brandon ⋅ Bruno Loureiro ⋅ N Alex Cayco Gajic ⋅ Arthur Pellegrino
NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama
Logan Mann ⋅ Abdur Rahman ⋅ Mohammad Saifullah ⋅ Taaha Kazi ⋅ Vasu Sharma
Scaling with Recursion in Masked Discrete Diffusion Models
Alba Carballo-Castro ⋅ Julianna Piskorz ⋅ Paulius Rauba ⋅ Mihaela van der Schaar ⋅ Pascal Frossard
How Cross-Entropy Learns Data Modes: Emergence and Implicit Bias in the Unconstrained Features Model
Arman Lotfalikhani ⋅ Mohammad Moshtaghifar ⋅ Connall Garrod ⋅ Jonathan Keating ⋅ Christos Thrampoulidis
How Excess Latent Dimensionality Delays Memorization in Diffusion Models
Trevor Chen ⋅ Ryan Shahbaba ⋅ Avni Garg ⋅ Kevin Peng
A Coulomb Particle Model for Learning Kernel Attention in Transformers
Masoud Badiei Khuzani ⋅ Sharath Honnaiah ⋅ Atiq Islam ⋅ Alex Cozzi ⋅ Abraham Bagherjeiran
Deep Learning as Neural Low-Degree Filtering: A Theory of Hierarchical Feature Learning
Yatin Dandi ⋅ Matteo Vilucchio ⋅ Luca Arnaboldi ⋅ Hugo Tabanelli ⋅ FLORENT KRZAKALA
In-Context Benign Overfitting: A Feature-Selection Model in In-Context Linear Regression
Puneesh Deora ⋅ Bhavya Vasudeva ⋅ Christos Thrampoulidis
Continuous Sparsification via Minimizing Movement
Hoang Pham ⋅ Tom Jacobs ⋅ Binh T. Nguyen ⋅ Rebekka Burkholz ⋅ Long Tran-Thanh
Self-Distillation for Data-Scarce Language Model Pretraining
Javid Lakha ⋅ Nihal Nayak ⋅ Bingbin Liu ⋅ Sham Kakade ⋅ David Alvarez-Melis
Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise
Bingbin Liu ⋅ Rachit Bansal ⋅ Depen Morwani ⋅ David Alvarez-Melis ⋅ Sham Kakade
RRD: Routing-and-Residual Distillation for Efficient MoE Recovery in Large Language Models
Hoyoon Byun ⋅ Kangjun Noh ⋅ Somin Kim ⋅ Heedong Kim ⋅ Jaeyoon Shim ⋅ Sungjun Lim ⋅ Youngjun Choi ⋅ Kyungwoo Song
How does feature learning change the function space evolution?
João Lobo ⋅ Bruno Loureiro ⋅ Long Tran-Thanh ⋅ Fanghui Liu
Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization
Kristi Topollai ⋅ Allan Ma ⋅ Tolga Dimlioglu ⋅ Sui Jiet Tay ⋅ Anna Choromanska
Uniform Spectral Growth under Factor-wise Muon Orthogonalization in Matrix Factorization and LoRA
Changmin Kang ⋅ Jihun Yun ⋅ Baekrok Shin ⋅ Yeseul Cho ⋅ Chulhee Yun
Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model
Arie Zurich ⋅ Hugo Tabanelli ⋅ Yatin Dandi ⋅ FLORENT KRZAKALA ⋅ Bruno Loureiro
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Jing Huang ⋅ Daniel Wurgaft ⋅ Rachit Bansal ⋅ Laura Ruis ⋅ Naomi Saphra ⋅ David Alvarez-Melis ⋅ Andrew K Lampinen ⋅ Christopher Potts ⋅ Ekdeep Singh Lubana
Provable Data Scaling Law for Meta Learning via Complexity Minimization
Kazuto Fukuchi ⋅ Ryuichiro Hataya ⋅ Kota Matsui
Deriving Hyperparameter Scaling Laws via Modern Optimization Theory
Egor Shulgin ⋅ Jörg Franke ⋅ Dimitri von Rütte ⋅ Tianyue Zhang ⋅ Niccolò Ajroldi ⋅ Korbinian Pöppel ⋅ Bernhard Schölkopf ⋅ Aaron Klein ⋅ Peter Richtarik ⋅ Antonio Orvieto
Optimistic Online Learning for Data Mixture Optimization
Sofija Orlovic ⋅ Francesco Tonin ⋅ Volkan Cevher
HORST: Composing Optimizer Geometries for Sparse Transformer Training
Tom Jacobs ⋅ Rohan Jain ⋅ Rebekka Burkholz
Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
Yu-Ang Lee ⋅ Ching-Yun Ko ⋅ Pin-Yu Chen ⋅ MI-YEN YEH
Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality
Arpan Mukherjee ⋅ Marcello Bullo ⋅ Debabrota Basu ⋅ Deniz Gunduz
DynMuon: A Dynamic Spectral Shaping View of Muon
Fangzhou Wu ⋅ Rikhav Shah ⋅ Sandeep Silwal ⋅ Richard Zhang
KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers
Leon A Trochelmann ⋅ Sajad Movahedi ⋅ Shiwei Liu ⋅ Antonio Orvieto
Causal Volterra Dynamics of Mamba
Ming-Ching Chang ⋅ Ting Yu Tsai ⋅ Davis Wertheimer ⋅ Felix X.-F. Ye
Optimal learning rate scaling depends on data in deep scalar linear networks
Yedi Zhang ⋅ Peter Latham ⋅ Leena Chennuru Vankadara ⋅ Andrew Saxe
Layer Collapse in Diffusion Language Models
Alexander Conzelmann ⋅ Albert Catalan-Tatjer ⋅ Shiwei Liu
On How Muon Reshapes Skill Learning Dynamics
Alex Reyes Aranda ⋅ Vincent-Daniel (Juyoung) Yun ⋅ Vatsal Sharan ⋅ Bhavya Vasudeva
Predicting Cross-Domain RAG Retrieval Quality using Von Neumann Graph Entropy
John Carlsson ⋅ Daniel Barcklow ⋅ Nikita Seleznev ⋅ Senthil Kumar ⋅ Ke Xu
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Zhiwei Xu ⋅ Shihao Wu ⋅ Hanseul Cho ⋅ Wei Hu ⋅ Yixin Wang
Feature Learning in High-Dimensions under Structured Covariance: Scaling Laws in Quadratic Networks
Qingyuan Yu ⋅ Nuri Mert Vural ⋅ Xin T Tong ⋅ Murat Erdogdu
A Quadratic Lens on Muon: Orthogonalization, Invariance, and Implicit Preconditioning
Egor Shulgin ⋅ Sam Laing ⋅ Antonio Orvieto ⋅ Peter Richtarik
Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models
Thomas Tulinski ⋅ Simona Cocco ⋅ Remi Monasson ⋅ Jorge FERNANDEZ-DE-COSSIO-DIAZ
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
Bing Liu ⋅ WenJie Zhou ⋅ Chengcheng Zhao
Refresh-Scaling the Memory of Balanced Adam
Alberto Fernandez-Hernandez ⋅ Cristian Pérez-Corral ⋅ Jose I. Mestre ⋅ Manuel F Dolz ⋅ Enrique S. Quintana-Orti
Too Sharp, Too Sure: When Calibration Follows Curvature
Alessandro Morosini ⋅ Matea Gjika ⋅ Tomaso A Poggio ⋅ Pierfrancesco Beneventano
A loss curvature account of fine-tuning fragility
Ivaylo Dimitrov ⋅ Leo Karoubi ⋅ Sunny Howard ⋅ Dmitrii Krasheninnikov
A Compute-Matched Study of Hidden Layer Distillation for LLM Pre-Training
Maxime Guigon ⋅ Lucas Dixon ⋅ Michael Sander
Be Greedy, Stay Linear: Universally Robust Feature Engineering
Yunze Leng ⋅ Rohan Ghosh ⋅ Mehul Motani
A $p$-adic Perspective on Low-Bit Training of Neural Networks
Daniil Bershatskiy ⋅ Marina Munkhoeva ⋅ Ivan Oseledets
Scaling Laws for Grid-Based Approximate Nearest Neighbor Search in High Dimensions
Matthew Liu ⋅ Wei H Zheng ⋅ Vidhan Purohit ⋅ Siqi Xie ⋅ Chieh-En Li ⋅ Jerry Li ⋅ Noah Flynn
Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima
Huanran Chen ⋅ Huaqing ZHANG ⋅ Xiao Li ⋅ Yinpeng Dong ⋅ Ke Shen ⋅ Jun Zhu
MIDUS: Memory-Infused Depth Up-Scaling
Taero Kim ⋅ Hoyoon Byun ⋅ Youngjun Choi ⋅ Sungrae Park ⋅ Kyungwoo Song
On the Convergence of Low-Precision LoRA Training
Dechen Zhang ⋅ Xuan Tang ⋅ Difan Zou
Neural Neural Scaling Laws
Michael Hu ⋅ Jane Pan ⋅ Ayush R Jhaveri ⋅ Nicholas Lourie ⋅ Kyunghyun Cho
Training Transformers for KV Cache Compressibility
Yoav Gelberg ⋅ Yam Eitan ⋅ Michael Bronstein ⋅ Yarin Gal ⋅ Haggai Maron
Internal Data Repetition Destroys Language Models
Jessica Chudnovsky ⋅ Joshua Kazdan ⋅ Noam Levi ⋅ Rylan Schaeffer ⋅ Yegor Denisov-Blanch ⋅ Sanmi Koyejo ⋅ David Donoho
Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models
Samy Jelassi ⋅ Mujin Kwun ⋅ Rosie Zhao ⋅ Yuanzhi Li ⋅ Nicolò Fusi ⋅ Yilun Du ⋅ Sham Kakade ⋅ Carles Domingo i Enrich
Dynamics of Nonlinear Feature Learning in Two-Layer GCNs on XOR-CSBM
Zen Inagaki ⋅ Guillaume Braun ⋅ Masaaki Imaizumi
When and Why Grouping Attention Heads Accelerates Muon Optimization
Hongtao Zhang ⋅ WenJie Zhou ⋅ Wei Chen ⋅ Xueqi Cheng
Understanding Clipping in Zeroth-order Optimization
Saket Gollapudi ⋅ Elisa Bertino ⋅ Sewoong Oh
What it means by learning in a neural network: easing the knot
Atiya Ibnat Tasnim ⋅ Rushmila S Khan ⋅ Md. Shahriar Karim
Optimal scaling laws in learning hierarchical multi-index models
Leonardo Defilippis ⋅ FLORENT KRZAKALA ⋅ Bruno Loureiro ⋅ Antoine Maillard
Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
Alex Meterez ⋅ Pranav Nair ⋅ Depen Morwani ⋅ Cengiz Pehlevan ⋅ Sham Kakade
Reparametrizing Shampoo and SOAP for Subspace Basis Updates and BFloat16 Storage
Alan Milligan ⋅ Zikun Xu ⋅ Simon Lacoste-Julien ⋅ Felix Dangel ⋅ Wu Lin
Why Routers Freeze: Infinite Width Learning Dynamics for Mixture of Experts
Anish Dhir ⋅ Volkan Cevher ⋅ Leena Chennuru Vankadara
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Mary Letey ⋅ Yue Lu ⋅ Cengiz Pehlevan ⋅ Jacob A Zavatone-Veth
BReD: Stabilizing Quantized EMA Dynamics for Memory-Efficient Large-Scale Training
Heshen Zhan ⋅ Youhan Huang ⋅ Yunke Peng ⋅ Yao Wang ⋅ Linghui Kong ⋅ Ziwei Zhu ⋅ Qingyu Han ⋅ Yaoyuan Wang ⋅ Congliang Chen ⋅ Ruoyu Sun
Weight Anisotropy in Mean-Field Theory: Learning on Isotropic Data
Niclas Göring ⋅ Chris Mingard ⋅ Yoonsoo Nam ⋅ Jake Reid ⋅ Ard Louis
Signal Frequency Imbalance and Ill-Conditioning
Tianyue Zhang ⋅ Francis Bach ⋅ Frederik Kunstner
Loss and Optimizer as Two Essential Mechanisms Behind Knowledge Distillation
Satoki Ishikawa ⋅ Sameer S Deshmukh ⋅ Sakina Fatima ⋅ Takumi Honda ⋅ Rio Yokota
On the Surprising Effectiveness of Masking Updates in LLM Training
Taejong Joo ⋅ Wenhan Xia ⋅ Cheolmin Kim ⋅ Ming Zhang ⋅ Eugene Ie
Physics-Guided Policy Optimization with Self-Distillation
Ke Wang ⋅ Yuning Wu ⋅ Haoran Liu ⋅ Chaoqun Jia ⋅ Devin Chen ⋅ Kai Wei
Learnability and Competition in High-Dimensional Multi-Component ICA
Eser İlke Genç ⋅ Samet Demir ⋅ Zafer Dogan
Reset-and-Discard (ReD) Improves Coverage at every Budget under Inference Power-Law Scaling
Sagi Meir ⋅ Tommer D Keidar ⋅ Noam Levi ⋅ Shlomi Reuveni ⋅ Barak Hirshberg
On the Optimizer Dependence of Neural Scaling Laws
Shourya V Jain ⋅ Vansh Ramani
Fixed-Point Reasoning: Stable and Adaptive Deep Looped Models
Sajad Movahedi ⋅ Shlomo L Feigin ⋅ Vera Milovanović ⋅ Alexander Theus ⋅ Thomas Hofmann ⋅ Valentina Boeva ⋅ T. Konstantin Rusch ⋅ Antonio Orvieto
Efficient Clustering with Provable Guardrails for LLM Inference at Scale
Longshaokan Wang ⋅ Waitsang Keung ⋅ Punit Ghodasara ⋅ Roman Z Wang ⋅ Ali Dashti ⋅ Francesc Moreno-Noguer
Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics
Binghua Li ⋅ Mengzhe Li ⋅ Denny Wu ⋅ Tianhao Wang
Generalization Analysis of Linear Knowledge Distillation
Taesun Yeom ⋅ Taehyeok Ha ⋅ Jaeho Lee
Hourglass MLP: Rethinking the Shape of Residual Architectures
Meng-Hsi Chen ⋅ Yu-Ang Lee ⋅ Feng-Ting Liao ⋅ Da-shan Shiu
Depth scaling and Muon enable balanced expert usage in MoE training
Xi Wang ⋅ Soufiane Hayou ⋅ Eric Nalisnick
A Horizon-Dependent Intrinsic-Dimension Theory of Scaling for Biological Forecasting
Bryan Cheng ⋅ Austin Jin ⋅ Jasper Zhang ⋅ Arnav Pemmaraju ⋅ Brendan Lo ⋅ Joshua Chang
Explaining Data Mixing Scaling Laws
Rui Dai ⋅ SHURAN ZHENG
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Julius Girardin ⋅ Emanuele Troiani ⋅ Yizhou Xu ⋅ Vittorio Erba ⋅ FLORENT KRZAKALA ⋅ Lenka Zdeborova