Can Muon Fine-tune Adam-Pretrained Models?

Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to an optimizer mismatch. We investigate this mismatch through controlled experiments and relate it to the distinct implicit biases of Adam and Muon. We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength. This leads us to hypothesize that constraining updates should mitigate the mismatch. We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning. Studies on LoRA rank, catastrophic forgetting, and LoRA variants further confirm that mismatch severity correlates with update strength. These results shed light on how optimizer mismatch affects fine-tuning and how it can be mitigated. Our code is available here.

As AI models grow larger, training them from scratch becomes prohibitively expensive. Instead, researchers typically take a model that has already learned general knowledge (a pretrained model) and adapt it to specific tasks. This adaptation step (fine-tuning) requires an optimizer, an algorithm that decides how to update the model's parameters. Most existing AI models were trained using the Adam optimizer. A newer alternative, Muon, is faster and uses less memory, making it an attractive option. However, prior work discovered that models trained with Adam perform poorly when adapted using Muon, and vice versa. We trace this optimizer mismatch to a fundamental difference in how Adam and Muon shape a model's internal structure: switching optimizers disrupts the knowledge the model has already learned. We then show that this problem can be mitigated by limiting the extent to which the model is allowed to change during adaptation. By keeping most of the original model frozen and only updating a small number of added parameters, Muon matches or outperforms Adam across language and vision tasks. These findings shed light on how optimizer mismatch affects model adaptation, paving the way for leveraging Muon when building on existing AI models.