When Labelers Stay Silent: The Power of Ties in Cost-Effective Preference Learning

Standard preference alignment relies on a binary forced-choice paradigm, assuming definitive preferences for all pairs. However, we find that indistinguishable pairs are prevalent even in standard benchmarks, where quality differences of two responses often fall below the labeler's discriminative resolution limit. Forcing a choice in such cases could inject significant noise that undermines policy optimization. In this work, we propose a silent-aware framework that introduces a principled way to allow annotators to stay silent (i.e., express ties) and then explicitly model these ties during optimization. Our findings reveal a compelling phenomenon: when ties are properly modeled, supervision from small models yields alignment surpassing that of forced-choice LLMs or human experts. This discovery highlights a cost-effective path for alignment: respecting a labeler’s resolution limit is more critical than increasing its capability, while simultaneously unlocking the latent value in existing benchmarks by properly modeling inherent tie signals without requiring any re-labeling effort. To leverage these signals, we propose several optimization objectives to drive the policy toward high-reward regions while mitigating unreliable updates that lead to arbitrary distribution shifts. Our approaches significantly surpass conventional alignment performance, consistently outperforming the strongest available baselines across diverse benchmarks.

Standard AI training relies on forcing an evaluator to choose the better of two responses. However, evaluators are frequently forced to pick a winner even when the two responses are virtually identical in quality. Forcing a choice in these situations introduces harmful noise that damages the AI's learning process. To fix this, we created a "silent-aware" method that allows evaluators to declare a tie — or stay silent — when they cannot reliably distinguish between responses. We also take these tied responses and sort them into "Both-Good" or "Both-Bad" categories to provide the AI with richer guidance. Surprisingly, we found that when these ties are used correctly, smaller AI evaluators can train systems better than human experts or massive AI models that are forced to guess. This approach offers a highly cost-effective way to build better AI. It reveals that respecting an evaluator's limits is more important than simply increasing its size, unlocking hidden value in existing data without the need for expensive relabeling.