Provably Protecting Fine-Tuned LLMs from Training Data Extraction while Preserving Utility

Fine-tuning large language models (LLMs) on sensitive datasets raises privacy concerns, as training data extraction (TDE) attacks can expose highly confidential information. Existing defenses against such attacks either lack formal privacy guarantees or incur substantial utility degradation. We observe that fine-tuning induces widespread probability shifts, yet preserving only a small subset of influential token-level deviations is sufficient; the remaining shifts can be aggressively smoothed with minimal impact on utility. Motivated by this insight, we propose SCP-$\Delta_r$, a Near Access Freeness (NAF)-based algorithm that operates on relative probabilities and explicitly smooths low-impact tokens using a base model. SCP-$\Delta_r$ achieves orders-of-magnitude better theoretical bounds than existing NAF based methods and provides strong empirical protection against TDE attacks with minimal performance loss.

Large Language Models (LLMs) are often customized using private data through a process called fine-tuning. However, attackers can sometimes trick these models into revealing parts of their training data, including confidential information. Existing defenses either provide weak privacy guarantees or significantly reduce performance. Our work builds on a privacy framework called Near Access Freeness (NAF). NAF-based methods split sensitive data into separate parts and train different models on each part. The models’ outputs are then combined in a way that reduces the chance of revealing private information. However, previous NAF-based methods do not work well for LLMs because fine-tuning causes widespread changes in output probabilities, making the models appear more different than they really are. We discovered that most of these changes are not important for the model’s behavior. Our method smooths away unnecessary probability shifts while preserving the most important ones. To do this effectively, we operate on the models’ relative preferences between possible next words rather than their raw probabilities. Our approach provides strong mathematical guarantees against training data extraction attacks while preserving the model’s reasoning and language capabilities. This could help organizations safely fine-tune LLMs on sensitive data such as private documents, medical information, or proprietary code.