Despite the impressive performance of foundation models in recent years, their underlying mechanisms remain poorly understood. Most studies treat these models as black boxes, hindering our ability to grasp how high-level representations emerge from the optimization process. In this talk, I will present a summary of our research efforts to tackle this challenging problem, by studying the intricacies of training dynamics. We uncover explanations for many counterintuitive behaviors of these models, paving the way for a deeper understanding of their inner workings and shedding light on the mystery of feature emergence in modern foundation models.