Stabilizing Native Low-Rank LLM Pretraining

Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promising route to reduce training and inference costs, but the community lacks a stable recipe for training models from scratch using exclusively low-rank weights while matching performance of the dense model. We demonstrate that Large Language Models (LLMs) can be trained from scratch using exclusively low-rank factorized weights for all non-embedding matrices without auxiliary "full-rank" guidance required by prior methods. While native low-rank training often suffers from instability and loss spikes, we identify uncontrolled growth in the spectral norm (largest singular value) of the weight matrix update as the dominant factor. To address this, we introduce Spectron: Spectral renormalization with orthogonalization, which dynamically bounds the resultant weight updates based on the current spectral norms of the factors. Our method enables stable, end-to-end factorized training with negligible overhead. Finally, we establish compute-optimal scaling laws for natively low-rank transformers, demonstrating predictable power-law behavior and improved inference efficiency relative to dense models. Our code is available at https://github.com/Pauljanson002/spectron

Large language models keep growing in size, which makes them expensive and memory-hungry to train and run. One way to shrink them is to replace each big block of numbers inside the model with two much smaller blocks that multiply together — but when researchers try to train a model this way from the very start, training tends to spiral out of control and fail. We traced this failure to one culprit: when the two small blocks are updated separately, one can quietly balloon in size without limit, throwing the whole model off balance. Our method, Spectron, acts like a stabilizer: at each step it cheaply measures how large the blocks have grown and rescales the updates so they stay within a safe range. This keeps training stable from scratch without the "training wheels" of full-size backup weights that earlier methods relied on, and adds less than 1% extra computation. With Spectron, a compressed model matched a full-size one trained on the same budget while using 42% fewer parameters. The result is cheaper, faster, and more energy-efficient to run, lowering the cost and broadening access to building capable AI.