SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm

The long-standing tension between Pre- and Post-Norm remains an open problem in Transformer architecture, reflecting a fundamental trade-off between training stability and representational capacity. Prior attempts to combine their strengths have made progress, but often show limited robustness across training settings, restricting their broader applicability. We revisit this dilemma, showing that single-stream architectures struggle to reconcile Pre-Norm's stable identity-gradient propagation with Post-Norm's normalization of the main residual path. To address this structural tension, we propose SiameseNorm, a simple yet effective two-stream architecture that remains compatible with Pre-Norm training recipes. SiameseNorm couples Pre-Norm-like and Post-Norm-like streams through shared residual blocks, allowing each residual block to receive optimization signals from both pathways with negligible overhead. Extensive experiments on 400M and 1.3B dense language models, 15B MoE models, Vision Transformers, and Diffusion Transformers show that SiameseNorm consistently improves performance while maintaining strong training stability across architectures and modalities. Code is available at https://github.com/Qwen-Applications/SiameseNorm.

Modern AI systems such as large language models are built from many repeated layers. As these models become larger and deeper, it becomes increasingly difficult to make them both powerful and easy to train. A common design choice in these models helps training remain stable, but may limit how effectively deeper layers contribute. Another design choice can better control the information flowing through the model, but often makes training less reliable. In this paper, we study this trade-off and propose SiameseNorm, a simple change to the Transformer architecture. Instead of forcing the model to use only one information flow, SiameseNorm uses two connected flows that share the same main computations. One flow helps preserve stable training, while the other helps regulate the model’s internal representations. This design adds very little extra cost and can be used with standard training settings. We test SiameseNorm on large language models, mixture-of-experts models, vision models, and image generation models. Across these settings, SiameseNorm consistently improves performance while keeping training stable. These results suggest that using multiple connected information flows is a promising direction for building stronger and more reliable AI models.