PEARL: Differentially Private and Entropy-Aware Regulated Language Generation

Large language models (LLMs) often employ Retrieval-Augmented Generation (RAG) to improve factuality. However, this also increases the risk of sensitive private information leakage. Differential Privacy (DP) has therefore been integrated into LLM inference and is widely regarded as a standard safeguard; yet most studies focus narrowly on the privacy–utility trade-off, leaving the trustworthiness of DP outputs underexplored. To assess trustworthiness, we employ the confidence gap (CG), which quantifies an LLM’s internal knowledge conflict. We show that CG correlates with both hallucination and exposure of personally identifiable information (PII). Building on this insight, we propose PEARL, a CG‑guided, entropy‑aware private decoding framework. PEARL adaptively allocates the privacy budget across tokens and sentences based on CG, concentrating protection on PII-bearing spans while stabilizing low-confidence, hallucination-prone regions. In experiments, PEARL improves both trustworthiness and robustness against PII extraction attacks. Notably, while applying DP alone significantly increases hallucination, our framework demonstrates that it is possible to preserve privacy while reducing hallucination.

Large language models such as ChatGPT increasingly assist us in sensitive areas like healthcare and banking by looking up relevant documents before answering — a technique called retrieval-augmented generation. But these documents often contain personal information, and the model can accidentally reveal names, dates of birth, or medical record numbers in its answers. A mathematical safeguard called differential privacy can protect this information by injecting carefully calibrated noise into the model's output. Unfortunately, this noise has a hidden side effect: it makes the model more likely to invent false facts — a phenomenon known as hallucination. We discovered that a simple internal signal, called the confidence gap, can tell us when the model is about to leak private information and when it is about to hallucinate. Building on this, we developed PEARL, a system that uses this signal to selectively redact sensitive content and rewrite unreliable sentences, all while keeping mathematical privacy guarantees intact. PEARL shows that privacy and trustworthiness do not have to be at odds. It opens the door to deploying AI assistants safely in fields where both accuracy and confidentiality are essential.