Fine-Tune Once, Reuse Across Models: Bayesian Task-Update Factors and Approximations

As pre-trained models evolve rapidly, transferring fine-tuning knowledge to updated models without retraining has become a critical challenge. Most existing methods reuse parameter updates, yet the same dataset can induce substantially different updates across base models due to mismatched local loss landscapes, making such transfer unstable. We instead adopt a Bayesian-updating perspective: a base model defines a prior, while fine-tuning contributes a task-update factor that is prior-agnostic, thereby making it feasible to reuse the update across base models. Specifically, we formalize a reusable task-update factor by requiring invariance across base models and a fixed-dimensional parameterization. Our main theoretical result shows that such reusable factors exist when the variational family is a half-space, and it is already maximal among convex families. In particular, an ideal regime arises when the priors and their Bayesian posteriors remain within a shared exponential family, as it always admits a reusable update factor. Building on this existence, we propose ***Bayesian Task Update Transfer(BTransfer), which extracts a reusable task-update factor from a single fine-tuning run and applies it to a new prior. For deep networks, we implement BTransfer with a ``lift–transfer–return'' pipeline: 1) lift model parameters to distributions; 2) transfer the extracted task-update factor in the exponential family distributions; and 3) return the updated posterior distribution to parameter space. Extensive experiments demonstrate that our approach effectively reuses fine-tuning knowledge across models without post-training.

Modern AI models are updated frequently, and adapting every new version to the same task can be costly and sometimes impossible when the original training data are unavailable. Existing approaches often reuse the parameter difference between an old model and its fine-tuned version, but this difference may not work well for a new model because different models can respond to the same task in different ways. We propose BTransfer, a method for reusing fine-tuning knowledge across updated models without fine-tuning them again. Instead of directly reusing parameter differences, BTransfer extracts the task-specific effect learned during fine-tuning and transfers it to a newer compatible model. We also provide theoretical conditions under which such reusable updates can exist. Our experiments show that BTransfer transfers fine-tuning knowledge in a more stable and reliable way than parameter-difference methods. This could help reduce the cost of repeatedly adapting updated AI models and make past fine-tuning efforts more reusable.