Real-Time Aligned Reward Model beyond Semantics

Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for aligning large language models (LLMs) with human preferences, yet it is susceptible to reward overoptimization, in which policy models overfit to the reward model, exploit spurious reward patterns instead of faithfully capturing human intent. Prior mitigations primarily relies on surface semantic information and fails to efficiently address the misalignment between the reward model (RM) and the policy model caused by continuous policy distribution shifts. This inevitably leads to an increasing reward discrepancy, exacerbating reward overoptimization. To address these limitations, we introduce R2M (Real-Time Aligned Reward Model), a novel lightweight RLHF framework. R2M goes beyond vanilla reward models that solely depend on the semantic representations of a pretrained LLM. Instead, it leverages the evolving hidden states of the policy (namely policy feedback) to align with the real-time distribution shift of the policy during the RL process. This work points to a promising new direction for improving the performance of reward models through real-time utilization of feedback from policy models.

Many AI assistants are trained using feedback from people so that their answers better match human preferences. However, this process can sometimes go wrong. Instead of learning to be genuinely helpful, the AI system may learn to produce answers that look good to an automatic judge, even when those answers do not fully reflect what people actually want. Existing methods mainly examine the surface meaning of the AI's answers. This can miss an important problem: as the AI keeps learning, its behavior changes, while the automatic judge may not keep up. Over time, this gap can become larger, making it easier for the AI to take advantage of the judge's weaknesses. We propose R2M, a lightweight method that helps the judge stay better matched with the AI system as it changes during training. By using real-time information from the AI being trained, R2M allows the judge to adjust more closely to the AI's current behavior. This provides a promising way to make AI assistants more reliable, helpful, and better aligned with human expectations.