Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Data

While post-training has successfully improved large language models (LLMs) across a variety of domains, these gains heavily rely on human-labeled data or external verifiers. Existing data has already been exploited, and new data is expensive to collect. Moreover, true intelligence goes far beyond verifiable tasks. Therefore, we need self-improvement frameworks that are less dependent on external signals and more broadly applicable to both verifiable and non-verifiable domains. We propose Mutual Information Preference Optimization (MIPO), a contrastive data augmentation method that constructs preference pairs by generating a positive response conditioning on the correct prompt, and a negative response by conditioning on a random, unrelated prompt. We show that using Direct Preference Optimization to learn from this paired data maximizes pointwise mutual information under the base LLM between prompts and model responses. Experiments with with 1-7B parameter Llama and Qwen instruct models show that MIPO achieves 3-16% gains (and 51% increase for Qwen2.5-1.5B-Instruct) on personalization compared to prompting baselines. Surprisingly, MIPO can also be useful in verifiable domains, such as math and multiple-choice question answering, yielding 1-20% gains without any additional data or external supervision. These results suggest a promising direction for self-improvement using intrinsic signals derived from contrastive data pairs.

While large language models (LLMs) are becoming increasingly powerful, training such models typically relies on large amounts of human-labeled data or is limited to settings with ground-truth solutions (e.g., math and multiple-choice question answering). However, this paradigm poses two challenges: human data is expensive to collect, and true intelligence goes beyond verifiable tasks. Given this in mind, how do we train models with their own data without relying on external feedback from humans or other stronger models and additional data? We turn to Direct Preference Optimization (DPO) with a simple data augmentation technique based on contrastive data pair construction. While DPO is typically trained using human-labeled preference data of chosen and rejected responses to a prompt, we propose generating chosen responses by conditioning on the correct prompt, and rejected responses by sampling from a random prompt in the prompt set. This trains the model to distinguish responses answering the correct prompt from responses answering the incorrect prompt, and as a result, leads the model to produce responses that better reflect the information in the prompt. We call this method MIPO, short for Mutual Information Preference Optimization, due to its theoretical connection to mutual information. We show that this simple data augmentation goes a long way, yielding gains in both verifiable and non-verifiable domains with 1-7B parameter open-weight Llama and Qwen instruct models. In particular, MIPO achieves 3-16% gains (and a 51% increase for Qwen2.5-1.5B-Instruct) on personalization tasks (i.e., given a user query and specific context, the model needs to personalize its response to better address the user specific information) compared to prompting with user-specific contexts and instructions, and yields 1-20% gains in verifiable domains of math and reasoning-based MCQ. These results suggest a promising direction for self-improvement using intrinsic signals derived from contrastive data pairs.