Learning to Reason for Factuality

Reasoning Large Language Models (R-LLMs) have significantly advanced complex reasoning tasks but often struggle with factuality, generating substantially more hallucinations than their non-reasoning counterparts on long-form factuality benchmarks. However, extending online Reinforcement Learning (RL), a key component in recent R-LLM advancements, to the long-form factuality setting poses several unique challenges due to the lack of reliable verification methods. Previous work has utilized automatic factuality evaluation frameworks such as FActScore to curate preference data in the offline RL setting, yet we find that directly leveraging such methods as the reward in online RL leads to reward hacking in multiple ways, such as producing less detailed or relevant responses. We propose a novel reward function that simultaneously considers the factual precision, response detail level, and answer relevance, and applies online RL to learn high quality factual reasoning. Evaluated on six long-form factuality benchmarks, our factual reasoning model achieves an average reduction of 23.1 percentage points in hallucination rate, a 23% increase in answer detail level, and no degradation in the overall response helpfulness.

Artificial Intelligence models, known as Large Language Models (LLMs), often suffer from "hallucinations"—a phenomenon where the AI model confidently fabricates incorrect facts. This problem is especially severe when models generate long, complex answers. Surprisingly, we find that AI models designed specifically to "reason" step-by-step can actually hallucinate even more than standard models. To solve this, we developed a trial-and-error training method (online Reinforcement Learning) that uses external automated judges to carefully calibrate the AI's behavior. During training, the AI is continually graded on its factual accuracy, level of detail, and on-topic relevance. By learning from these signals, the model is trained to naturally refrain from making unsupported claims when answering questions. This approach significantly reduces hallucinations and boosts the factual accuracy of the AI's long-form responses without making them overly brief or unhelpful, offering a practical way toward more reliable AI assistants.