Common Origins, Divergent Destinations: The Development of Cross-Layer Alignment Under GELU and SwiGLU

Transformer language models coordinate computation across independently parameterized layers through a shared residual stream, yet the developmental process of this coordination remains poorly understood. We track the emergence of inter-layer alignment in six models across four families, finding a common early formation followed by divergent architectural fates. Early in training, adjacent layers align their dominant MLP transformation directions. This alignment forms via a backward cascade originating at the output and propagating toward the input. The persistence of the resulting structure throughout the rest of training is architecturally determined: GELU models (Pythia, BLOOM) maintain weight-level alignment, while SwiGLU models (OLMo-2, TinyLlama) progressively dissolve it. We argue that early establishment of cross-layer alignment is a developmental priority for transformer training, and that the activation-function-dependent divergence is among the most consequential dynamical differences across modern transformer architectures.