Sparse structured matrices: Efficient adapter rank in fine-tuning Foundation models

Parameter-efficient fine-tuning (PEFT) has become a standard tool for adapting large models, yet most existing adapters are designed around fixed rank regimes. We present Scalable Morpho Adaptation (SaMA), a structured PEFT method that enables unified omni-rank adaptation within a single parameterization and mitigates Gauge symmetry in parameter space. Theoretically, we show that SaMA is more expressive than LoRA with respect to adapter parameter budget. Empirically, our method yields better performance than LoRA and other PEFT baselines on reasoning tasks.