ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation

Proactive Recommender Systems (PRSs) aim to guide user preference shift toward target items by generating paths of intermediate recommendations. Reinforcement learning (RL) provides a principled framework for optimizing such sequential decision tasks, as path rewards can naturally capture both short-term acceptance and long-term guidance effectiveness. However, naively applying policy gradients to PRS results in deficient gradient estimation. We identify two deficiencies: (1) path-level rewards decompose into step-level rewards with positive mean, creating a length-dependent bias that causes gradients to favor path extension over meaningful exploration; (2) weighting each step by the entire path-level reward ignores the decomposition structure, leading to high gradient variance. To rectify these two deficiencies, we propose an effective RL framework ProRL with two novel mechanisms for proactive recommendation. First, Stepwise Reward Centering subtracts expected rewards to neutralize length-dependent bias, ensuring that path extension yields zero expected gradient signal. Second, Position-Specific Advantage Estimation leverages the reward decomposition structure to compute step-dependent baselines, reducing gradient variance. Together, these mechanisms yield policy gradients that precisely target path quality. Our experiments on three real-world datasets demonstrate that ProRL significantly outperforms state-of-the-art PRSs. Our code is available at https://github.com/hongruhou89/ProRL.

Recommender systems on platforms like streaming services or online stores usually show you items similar to what you already like, which can trap users in filter bubbles and make it hard to discover genuinely new interests. A growing line of research called proactive recommendation tries to gently steer users toward a desired item over a sequence of suggestions — like a thoughtful friend who introduces you to jazz by first recommending songs close to your current taste. Teaching a computer to plan such a sequence is tricky: existing learning methods accidentally reward the system for simply making the recommendation list longer, rather than making each step genuinely better. We designed ProRL, a new training method that fixes both problems by carefully recalibrating how the system is rewarded at each step along the recommendation path. On three real-world datasets, ProRL guides users to target items more reliably than previous approaches, paving the way for recommender systems that broaden, rather than narrow, what people discover.