Conflict-Aware Adaptive Alignment for LLM Hallucination Mitigation

Despite strong performance, large language models (LLMs) still suffer from hallucinations. Most existing mitigation methods operate at inference time, without addressing the alignment limitation: LLMs are not trained to recognize their own lack of knowledge, and therefore tend to generate plausible responses even when the required knowledge is missing. Preference alignment approaches encourage uncertainty expression or refusal to improve truthfulness, but often consequently degrade helpfulness. To address this trade-off, existing preference alignment methods typically treat truthfulness and helpfulness as either universally collaborative or universally conflicting objectives across all samples. In contrast, we show that these objectives are consistent for most samples and conflict only in a small subset—where adaptive trade-off is truly needed. Based on this insight, we propose Conflict-Aware Adaptive Margin Preference Alignment (CAMP), which explicitly models when conflicts arise and adaptively regulates optimization strength. Experiments on UltraFeedback and representative hallucination benchmarks demonstrate that CAMP consistently improves truthfulness while maintaining a favorable helpfulness trade-off compared to strong hallucination mitigation and multi-objective alignment baselines.

Large language models can produce useful answers, but they sometimes generate responses that sound reasonable yet are incorrect. Many existing methods reduce hallucinations by addressing missing knowledge. We study a complementary preference alignment perspective: models are not sufficiently trained to recognize and acknowledge when they do not know the answer. A key alignment challenge is that making a model more truthful may reduce its helpfulness. For example, a model may refuse too often in order to be fully correct. We find that this trade-off does not occur for all samples: truthfulness and helpfulness are often aligned, and they conflict only in a small set of cases. Based on this insight, we propose a training method that adapts the alignment objective only when such conflicts appear. Our findings suggest that models do not always need a uniform trade-off between being truthful and being helpful. Instead, they can benefit from adapting this trade-off to each case. This can help future models generate responses that are both accurate and helpful.