Differentiable Conformal Training for LLM Reasoning Factuality

Large Language Models (LLMs) frequently hallucinate, limiting their reliability in critical applications. Conformal Prediction (CP) addresses this by calibrating error rates on held-out data to provide statistically valid confidence guarantees. Recent work extends CP to LLM factuality: outputs are decomposed into subclaims, each assigned a risk score, and a calibrated threshold filters out risky claims to guarantee hallucination rates below a user-specified level (e.g., 10%). While prior methods treat claims independently, Coherent Factuality extends to multi-step reasoning by representing outputs as dependency graphs and jointly validating claims with their logical ancestors. A key limitation is that Coherent Factuality is not differentiable, requiring hand-crafted scorers that at high reliability levels remove nearly 60% of true claims. We introduce Differentiable Coherent Factuality (DCF), a fully differentiable relaxation that enables learning improved scorers while provably recovering the original algorithm's guarantees. Experiments on two reasoning datasets demonstrate DCF achieves up to 141% improvement in claim retention while maintaining reliability guarantees, representing a significant step towards reliable conformal LLM systems.

AI language models like ChatGPT increasingly handle multi-step reasoning — solving math problems, fact-checking claims, answering complex questions — but they often produce confident-sounding statements that turn out to be wrong, known as "hallucinations." Existing methods can mathematically guarantee that the claims they keep are mostly correct, but they're so cautious that they discard nearly all the useful information along with the wrong claims. They also evaluate each claim independently, ignoring that later steps in a chain of reasoning are only valid if the earlier steps are. We introduce Differentiable Coherent Factuality (DCF), a method that learns to filter unreliable AI outputs while respecting how claims build on one another. Prior methods relied on hand-crafted rules that could not be improved through training. Our technical contribution is rewriting the procedure so it can be trained directly from data using standard machine learning techniques, then plugged back into the original procedure at deployment without losing its mathematical correctness guarantees. The result: DCF retains more than double the correct claims of the previous best method while maintaining the same guarantee on error rates. This moves rigorous AI hallucination detection from a theoretical tool toward something practical to deploy, especially for the emerging class of AI systems that perform extended reasoning and need to flag — or repair — steps they are not sure about.