TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning

While large language models (LLMs) have demonstrated strong performance on factoid question answering, they are still prone to hallucination and untruthful responses, particularly when tasks demand information outside their parametric knowledge. Indeed, truthfulness requires more than accuracy---models must also recognize uncertainty and abstain when unsure to avoid hallucinations. This presents a fundamental challenge for existing methods: approaches that optimize for accuracy often amplify hallucinations, while those that encourage abstention can become overly conservative, sacrificing correct answers. Both extremes ultimately compromise truthfulness. In this work, we present TruthRL, a general reinforcement learning (RL) framework that directly optimizes the truthfulness of LLMs. Specifically, we implement TruthRL using GRPO with a simple yet effective ternary reward that distinguishes correct answers, hallucinations, and abstentions. It incentivizes models to reduce hallucinations not only by providing correct responses, but also by enabling abstention when uncertain, thereby improving truthfulness. Extensive experiments across four knowledge-intensive benchmarks show that TruthRL significantly reduces hallucinations (e.g., 43.5\% $\rightarrow$ 19.4\%) and improves truthfulness (e.g., 5.3\% $\rightarrow$ 37.2\%), with consistent gains across various backbone models. Analysis shows that the improvement of TruthRL arises from enhanced capability of LLMs to recognize their knowledge boundary, hence avoiding being overly conservative as the baselines are.

Large language models can answer many factual questions, but they sometimes give confident answers that are wrong. This is especially risky when people use them for knowledge-heavy tasks, where a made-up answer can be worse than no answer at all. A more reliable system should know when to answer and when to say “I don’t know.” We introduce TruthRL, a training method that teaches a language model this distinction. The method rewards correct answers, penalizes false answers, and treats “I don’t know” as an acceptable response when the model is uncertain. This encourages the model to avoid guessing when it lacks enough knowledge. Across several question-answering benchmarks, TruthRL reduces wrong answers while keeping strong accuracy. Our analysis suggests that the model becomes better at recognizing the limits of its own knowledge, making it more trustworthy for tasks where factual reliability matters.