FedSDR: Federated Self-Distillation with Rectification

Federated fine-tuning of Large Language Models faces severe statistical heterogeneity. However, existing model-level defenses often overlook the root cause: intrinsic data distribution mismatches. In this work, we first establish Federated Self-Distillation (FedSD) as a fundamental and potent strategy. By projecting client representations into a smoothed ``model-understanding space,'' FedSD alone serves as a universal booster, demonstrating superior performance over conventional algorithms. Despite its success, we identify a subtle trade-off termed the Rewrite Paradox---unconstrained self-distillation can inadvertently increase hallucinations and redundancy. To refine this paradigm, we further propose FedSDR (Federated Self-Distillation with Rectification), the ultimate reinforced framework. It augments FedSD with a dual-stream mechanism: a local LoRA-S (Smoothing) branch to implicitly absorb heterogeneity via distilled data, and a parallel global LoRA-R (Rectification) branch anchored to raw data to enforce factual correctness. By selectively aggregating only LoRA-R, FedSDR yields a globally aligned and faithful model. Extensive experiments verify its superior performance.

When multiple organizations collaboratively fine-tune a large language model without sharing their private data — a setup known as federated learning — a key challenge arises: each participant's data can look very different from the others', making it hard for the shared model to learn well. We first show that a technique called self-distillation, where each participant's model learns from its own softened predictions rather than raw data, is a surprisingly powerful remedy for this problem. It works by translating each participant's unique data into a common "language" the model already understands, smoothing over the differences. However, we discovered a hidden trade-off we call the Rewrite Paradox: relying too heavily on the model's own predictions can cause it to drift from facts, producing more hallucinations and repetitive text. To fix this, we designed a two-branch system — one branch absorbs data differences through self-distillation, while the other stays anchored to the original data to preserve factual accuracy. Only the fact-checking branch is shared across participants, yielding a model that is both well-coordinated and trustworthy.