RLSF-V: Mitigating Hallucinations in MLLMs via Fuzzy Semantic Self-Feedback

Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception for open-world understanding, but exacerbate LLMs' hallucinations, in which generated text contradicts visual evidence or common sense. To mitigate hallucinations, a dominant strategy is Direct Preference Optimization (DPO) using hallucination-labeled responses. Existing pipelines, however, face two key limitations: they either (i) rely on human inspection or proprietary models to correct hallucinated outputs, producing off-policy preference data that violate the assumptions of DPO, or (ii) depend on stronger models to evaluate responses, leading to an unfavorable trade-off between performance and scalability. Departing from these paradigms, we propose a reference-policy \emph{self-feedback} framework that constructs preference data for hallucination mitigation without any external supervision (\textit{e.g.}, large models or humans). Specifically, we present a novel \emph{local fuzzy semantic} evaluation paradigm that derives a hallucination-sensitive confidence signal directly from the internal logits, which is then used to automatically rank diverse generated responses to build preference pairs for fine-tuning. Trained on a 10k-scale dataset, our method achieves competitive performance on both generative and discriminative benchmarks compared to existing RLHF and RLAIF baselines.

Artificial Intelligence (AI) models that process both images and text are becoming incredibly advanced, but they suffer from a major flaw known as "hallucination". This occurs when the AI confidently describes objects, attributes, or details in an image that are simply not there, which damages user trust in critical areas like medical advice or self-driving cars. Currently, researchers fix this by having humans or more powerful, proprietary AI models (like GPT-4) double-check and correct the AI's mistakes to create training data. However, this traditional approach has significant drawbacks: It is expensive and hard to scale: Relying on human reviewers or premium external AI systems takes time and resources. It can confuse the AI: Correcting a model using answers generated by a completely different system can lead to mismatched training data that violates the assumptions of how these models learn. To solve this, our paper introduces RLSF-V, a framework that teaches the AI to grade and correct its own work without any outside help. Instead of relying on external judges, our method looks deep inside the AI's internal decision-making process to measure its true confidence, focusing specifically on the important words in a sentence—like nouns and verbs—where hallucinations usually happen. By having the AI generate multiple answers and comparing its internal confidence scores, it learns to favor factual, confident responses and reject made-up, uncertain information. Our testing shows that with a relatively small amount of this self-generated training data, our method reduces hallucinations just as effectively as the older, more expensive methods that rely on outside help.