The Truth Stays in the Family: Enhancing Contextual Truthfulness via Inherited Heads in Model Lineages

Recent advances in large language models (LLMs) have produced many specialized multimodal LLMs (MLLMs) that share common foundational LLMs, forming distinct model lineages. It remains unclear whether a fundamental behavioral link exists between the foundational LLMs and downstream variants. We investigate this question by quantifying head-level context-truthfulness scores. Across diverse LLM and MLLM lineages, including Vicuna-, Qwen2.5-, LLaMA2-, and Mistral-based models, we find that Truth Scores are strongly preserved within model families, even after instruction tuning or multimodal adaptation. We further show that this inheritance is consistent with attention-head weight preservation, and that context-truthful heads attend to query-relevant evidence. Building on this finding, we propose TruthProbe, a soft-gating strategy that amplifies context-truthful heads while preserving other head contributions. TruthProbe improves contextual truthfulness on HaluEval and reduces multimodal hallucination on POPE and CHAIR, with base-LLM Truth Scores transferring effectively to their fine-tuned LLM and MLLM descendants. Code is available at https://github.com/miso-choi/TruthProbe.

Many AI models are built by adapting the same base language model for different tasks, such as image understanding. We show that these related models often preserve similar internal components linked to truthful reasoning. Using this insight, we propose TruthProbe, a lightweight method that strengthens these components without retraining the model. TruthProbe improves truthfulness in language tasks and reduces hallucinations in vision-language models, suggesting that reliability can be improved across an entire model family rather than one model at a time.