A Geometric View of Model Merging: Quotient Fréchet Averages from Toy Models to LoRA

Model merging combines task-specialized models without additional training, but weight-space averaging is not intrinsically well-defined when parameters have symmetries. Two checkpoints can represent the same or closely related function while occupying different points along a symmetry orbit; averaging arbitrary representatives can therefore produce a degraded merge. We formulate model merging as Fréchet averaging on a Riemannian parameter geometry, and argue that in the presence of symmetries the appropriate object is a Fréchet mean on the quotient space of parameterizations modulo symmetry. We work through a two-parameter linear model with a scaling symmetry, where the quotient geometry is closed form: quotient GeoMerge reduces to averaging the invariant predictors, while Euclidean and Fisher-weighted parameter averaging can collapse for gauge-equivalent checkpoints. We then instantiate the same principle for LoRA adapters and report proof-of-concept ViT-B/32 merging results, where quotient-aware GeoMerge improves over strong alignment-based LoRA merging baselines.