TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

Aligning large language models (LLMs) with human preferences is commonly done via reinforcement learning from human feedback (RLHF) with Proximal Policy Optimization (PPO) or, more simply, via Direct Preference Optimization (DPO). While DPO is stable and RL-free, it treats preferences as flat winner vs. loser signals and is sensitive to noisy or brittle preferences arising from fragile chains of thought. We propose TUR-DPO, a topology- and uncertainty-aware variant of DPO that rewards how answers are derived, not only what they say, by eliciting lightweight reasoning topologies and combining semantic faithfulness, utility, and topology quality into a calibrated uncertainty signal. A small learnable reward is factorized over these signals and incorporated into an uncertainty-weighted DPO objective that remains RL-free and relies only on a fixed or moving reference policy. Empirically, across open 7–8B models and benchmarks spanning mathematical reasoning, factual question answering, summarization, and helpful/harmless dialogue, TUR-DPO improves judge win-rates, faithfulness, and calibration relative to DPO while preserving training simplicity and avoiding online rollouts. We further observe consistent gains in multimodal and long-context settings, and show that TUR-DPO matches or exceeds PPO on reasoning-centric tasks while maintaining operational simplicity.

This paper introduces a new method to make AI systems that answer questions using external documents more reliable and less likely to produce misleading information. Current retrieval-based AI systems often select documents based only on topic similarity, which can accidentally reinforce incorrect or misleading answers. Existing solutions usually require expensive manually labeled examples showing both correct and incorrect answers. We propose a label-free approach called SDCP (Self-Distilled Contrastive Priors), which allows the language model to generate its own guidance signals without human annotations. For each question, the model produces both a likely correct answer and a possible misconception, then uses the contrast between them to better identify trustworthy supporting documents while filtering misleading ones. Experiments on several benchmark datasets show that SDCP consistently improves retrieval quality and factual accuracy compared to standard retrieval methods and other label-free approaches. The method performs particularly well on open-ended factual reasoning tasks, while more structured multiple-choice science questions remain challenging. Overall, the results suggest that AI systems can use their own internal knowledge to improve retrieval reliability without relying on costly labeled training data.