OLion: Approaching the Hadamard Ideal by Intersecting Spectral and L inf Implicit Biases

Many optimizers can be interpreted as steepest-descent methods under norm-induced geometries, and thus inherit corresponding implicit biases. We introduce Orthogonal Lion which combines spectral control from orthogonalized update directions with $\ell_\infty$-style coordinate control from sign updates. OLion forms a Lion-style momentum direction, approximately orthogonalizes it via a few Newton--Schulz iterations, and then applies an entrywise sign, providing an efficient approximation to taking a maximal step over the intersection of the spectral and $\ell_\infty$ constraint sets (a scaled Hadamard-like set for matrix parameters). Despite the strong nonlinearity of orthogonalization and sign, we prove convergence under a mild, empirically verified diagonal-isotropy assumption. Across large-scale language and vision training, including GPT-2 and Llama pretraining, SiT image pretraining, and supervised fine-tuning, OLion matches or outperforms AdamW and Muon under comparable tuning while using only momentum-level optimizer state, and it mitigates optimizer mismatch when fine-tuning AdamW-pretrained checkpoints.

Large AI models are expensive to train, and the choice of training method can strongly affect both their speed and the quality of the final model. Existing methods often make a tradeoff: some control the overall shape of model updates well, while others control individual numbers in the model well, but they do not naturally do both. We propose OLion, a new training method that combines these two kinds of control in one simple update. It first reshapes each update so that it is well balanced overall, then keeps only the direction of each individual value, which helps prevent a few unusually large values from dominating training. This design uses less extra memory than widely used adaptive methods such as AdamW, which is important when training very large models. We tested OLion on language model pretraining, image model pretraining, and fine-tuning large language models. Across these settings, OLion matches or improves over strong existing methods while keeping memory costs low. This matters because better training methods can reduce the cost of building large AI systems and make large-model training and fine-tuning more practical.