ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ActiveUltraFeedback, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside Double Reverse Thompson Sampling (DRTS) and DeltaUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ActiveUltraFeedback yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.

Many modern chatbots are improved by training them on pairs of answers to the same question, together with feedback about which answer is better. However, collecting these preference labels is expensive, especially when the questions require expert knowledge. Existing data-collection pipelines often spend this limited labeling budget on comparisons that provide little useful information. We introduce ActiveUltraFeedback, a system for collecting preference data more selectively. For each prompt, it generates several possible answers from a diverse set of language models, estimates both the quality of each answer and how uncertain this estimate is, and then selects the pair of answers that is expected to be most informative to label. We also introduce two new selection methods that focus on pairs with clear quality differences, since these contrasts can provide a stronger learning signal than comparisons between two very similar answers. Across several evaluations, the datasets created with ActiveUltraFeedback lead to better models for both judging answers and improving language models. In some settings, our approach reaches comparable or better performance with far fewer labeled comparisons, reducing annotation costs when feedback is the main bottleneck. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback, enabling other researchers to test new selection rules, judges, and data-collection strategies.