Positive–Unlabeled Reinforcement Learning Distillation for On-Premise Small Models

Due to constraints on privacy, cost, and latency, on-premise deployment of small models is increasingly common. However, most practical pipelines stop at supervised fine-tuning (SFT) and fail to reach the reinforcement learning (RL) alignment stage. The main reason is that RL alignment typically requires either expensive human preference annotation or heavy reliance on high-quality reward models with large-scale API usage and ongoing engineering maintenance, both of which are ill-suited to on-premise settings. To bridge this gap, in this paper, we propose a positive-unlabeled (PU) RL distillation method for on-premise small-model deployment. Without human-labeled preferences or a reward model, our method distills the teacher’s preference-optimization capability from black box generations into a locally trainable student. For each prompt, we query the teacher once to obtain an anchor response, locally sample multiple student candidates, and perform anchor-conditioned self-ranking to induce pairwise or listwise preferences, enabling a fully local training loop via direct preference optimization or group relative policy optimization. Theoretical analysis justifies that the induced preference signal by our method is order-consistent and concentrates on near-optimal candidates, supporting its stability for preference optimization. Experiments demonstrate that our method achieves consistently strong performance under a low-cost setting.

Many organizations want to run small AI models on their own machines because this can better protect private data, reduce cost, and lower response delay. However, after these models are trained to follow examples, it is often too expensive to further improve them using human feedback or repeated calls to powerful online models. This paper proposes a low-cost way to help such local models improve. For each user question, we ask a stronger black-box model only once to provide a high-quality reference answer. The local model then generates several of its own answers and compares them with this reference to decide which answers are better. These comparisons are used to further train the local model without human preference labels or a separate reward model. Experiments show that this approach improves local small models across both text and vision-language tasks while keeping the cost low.