Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future

Self-Rewarding Language Models propose an architecture in which the Large Language Models(LLMs) both generates responses and evaluates its own outputs via LLM-as-a-Judge prompting, dynamically improving its generative capabilities through iterative Direct Preference Optimization (DPO). However, our analysis reveals a critical limitation in existing self-rewarding paradigms: the synchronized improvement of chosen and rejected responses progressively narrows the representational difference between contrasting samples, undermining effective preference learning. We propose Temporal Self-Rewarding Language Models that strategically coordinate past, present, and future model generations to sustain learning signals. Our dual-phase framework introduces: (1) Anchored Rejection - fixing rejected responses using the past initial model's outputs and (2) Future-Guided Chosen - dynamically curating chosen samples using next-generation model predictions. Extensive experiments across three model families (Llama, Qwen, Mistral) and different model sizes (Llama3B/8B/70B) demonstrate significant improvements when trained with our method compared to self-rewarding using same computation resources. For example, Llama3.1-8B reaches a 29.44 win rate on AlpacaEval 2.0 with our method, outperforming the self-rewarding baseline (19.69) by 9.75. Notably, our method also demonstrates superior out-of-distribution generalization across mathematical reasoning (GSM8K), knowledge-based QA (ARC, TruthfulQA), and code generation (HumanEval) tasks, even though we do not specifically collect such training data. The generality of our temporal strategy is further validated by its benefits when extended to online reinforcement learning settings on mathematical reasoning tasks.

Large language models can be improved by having them judge their own outputs and learning to prefer better responses over worse ones, which is a process called "self-rewarding. However, as the model improves over multiple rounds, the gap between its good and bad outputs shrinks, making it increasingly difficult to learn what counts as a meaningful improvement. We propose a time-aware training strategy that addresses this issue by mixing outputs from different stages of the model's development. Specifically, we keep earlier, lower-quality outputs as negative examples to maintain a clear contrast, and use predictions from a future, stronger version of the model to select higher-quality positive examples. Experiments across multiple model families and sizes show that our approach substantially outperforms standard self-rewarding methods, and the benefits extend beyond the in-distribution problems to tasks like math reasoning, factual question answering, and code generation, suggesting that maintaining meaningful training contrasts is a broadly useful principle for self-improving AI systems.