RLRank: Distilling Offline Oracles into Online Policies for Document Reranking

We investigate whether small language models (SLMs) can learn effective document reranking through reinforcement learning, using a state-of-the-art cross-encoder as an offline oracle reward signal, without any supervised labels. We compare two action space formulations under Group Relative Policy Optimization (GRPO): listwise reranking, where the model outputs a full permutation of document indices, and pairwise reranking, where the model makes binary document preference judgments. We identify a zero-gradient deadlock that occurs when all rollouts within a GRPO group receive identical rewards, and propose a partial reward shaping strategy that resolves it. After this fix, a 1B-parameter listwise model reaches $\mathrm{NDCG@10} = 0.854$ after only 515 training steps ($\approx 3\%$ of a full run), while a 3B pairwise model achieves $76.1\%$ pairwise accuracy after 396 steps. Listwise training is substantially more sample-efficient, requiring one LLM call per query vs.\ $O(K^2)$ for pairwise.