Learning from One Another: Toward Complementary Knowledge Transfer via Localized Preference Deltas

Modern language models increasingly exhibit complementary local strengths, creating heterogeneous model ecosystems where useful capabilities are distributed across models rather than concentrated in a single dominant teacher. This challenges existing transfer pipelines, which typically assume that supervision should originate from a globally stronger model. We propose Delta Preference Transfer (DPT), a localized synthetic preference construction framework for preference-based transfer across language models. DPT constructs chosen/rejected preference pairs within the response space of a single generator by introducing controlled local failures into the rejected response, encouraging the target model to learn localized behavioral refinements rather than directly imitating a globally stronger teacher. This enables targeted preference transfer without requiring external judges, reward models, or globally ordered teacher--student assumptions during either data construction or training. Across diverse settings, DPT consistently improves mathematical reasoning performance and achieves competitive or superior results compared with existing preference-based transfer and knowledge distillation baselines. Our analyses further suggest that DPT provides a practical step toward complementary knowledge transfer in heterogeneous language model ecosystems.