Phase transition in high-dimensional learning, Lenka Zdeborová

Emergence in LLMs is surrounded by a plethora of open questions. Emergence in physics is linked to phase transitions. We will describe recent progress in characterizing phase transitions in the performance of neural networks and their consequences for algorithmic hardness. In particular, we will discuss how the staircase picture changes when batches are reused. We also unveil a phase transition between semantic and positional learning in a toy model of dot-product attention.