AMIGO: Adapters Meet Information Geometry

Low-parametric adapters have become the standard for efficient fine-tuning, enabling the capture of complex concepts and styles with minimal parameter overhead. However, merging low-rank adapters is a non-trivial task, and common approaches frequently rely on simple weight-space interpolations that do not account for the underlying generative distribution. We consider the problem of informing the merging procedure through the geometry of diffusion processes modeled as stochastic differential equations. We show that this task can be effectively reformulated as a stochastic optimal control problem. This framework allows us to approximate the Fisher information metric via drift differences between model trajectories, a perspective grounded in Girsanov’s theorem. By minimizing cumulative Kullback-Leibler divergence, we optimize a piecewise linear path in parameter space that serves as an approximate geodesic. We provide the necessary theoretical foundations and connect our approach to established geometric techniques. Experiments demonstrate that our method recovers exact geodesic structures in controlled settings, such as the Ornstein-Uhlenbeck process, and consistently outperforms existing baselines while preserving generative fidelity across various tasks.