Unbiased Reward Modeling from Implicit Feedback for LLM Alignment

Despite the success of reinforcement learning from human feedback (RLHF), existing reward modeling methods largely rely on explicit feedback, which is costly to collect and difficult to scale. This work studies implicit reward modeling, learning reward models from implicit user feedback, such as clicks, copies and skips. While scalable and cost-effective, implicit feedback poses two key challenges: (1) It lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) It suffers from selection bias, where responses have heterogeneous propensities to elicit feedback, which further obscures definitive negative samples. To address these challenges, we propose ImplicitRM, which learns unbiased reward models from implicit feedback. It stratifies training samples into four latent groups using a stratification model and derives a likelihood-maximization objective that is theoretically unbiased, thereby addressing both challenges. Experiments across diverse LLM backbones and benchmark datasets validate that ImplicitRM learns accurate reward models from implicit feedback and improves performance on downstream RLHF tasks.

Existing reward modeling methods typically rely on explicit feedback, such as ratings or pairwise comparisons, which is costly to collect and difficult to scale. Implicit feedback, such as clicks, copies, and upvotes, offers a scalable alternative but poses challenges due to positive-unlabeled nature and selection bias. We propose ImplicitRM, a method for learning unbiased reward models from implicit feedback. Experiments show that ImplicitRM learns accurate reward models from implicit-feedback data, which can further enhance downstream RLHF performance. It enables cost-effective and scalable alignment from natural user interactions.