Reward Shaping Control Variates for Off-Policy Evaluation Under Sparse Rewards

Off-policy evaluation (OPE) is essential for deploying reinforcement learning in safety-critical settings, yet existing estimators such as importance sampling and doubly robust (DR) often exhibit prohibitively high variance when rewards are sparse. In this work, we introduce Reward-Shaping Control Variates, a new family of unbiased estimators that leverage potential-based reward shaping to construct additional zero-mean control variates. We prove that shaped estimators always yields valid variance reduction, and that combining shaping-based and Q-based control variates strictly expands the variance-reduction subspace beyond DR and its minimax variant MRDR. Empirically, we provide a systematic regime map across synthetic chains, a cancer simulator, 5 single-stock and 1 multi-stock DOW-30 trading environments and an ICU-sepsis benchmark showing that shaping-based OPE consistently outperforms DR in sparse-reward settings, while a hybrid estimator achieves state-of-the-art performance across sparse, noisy, and misspecified environments. Our results highlight reward shaping as a powerful and interpretable tool for robust OPE, offering both theoretical guarantees and practical improvements in domains where standard estimators fail.

When AI systems make high-stakes decisions, like treating ICU patients or managing investments, we need to know whether a new strategy is actually better before rolling it out. Off-policy evaluation (OPE) tackles this by using historical data to estimate how a new policy would perform. The catch: most real-world settings only give feedback at the very end, a patient survives or doesn't, a portfolio gains or loses. With almost no reward signal along the way, standard evaluation methods fall apart. Existing approaches either produce unstable estimates or try to fill in the missing signal by learning a model of long-term outcomes, which introduces errors that compound over time. Neither works well when success is rare. Our method, Reward-Shaping Control Variates, takes a different angle. Even when rewards are sparse, trajectories are not: every sequence of decisions visits a rich sequence of states. We learn to score each state by how likely it is to lead to success, then use these scores to reduce the noise in our estimates. Mathematically, the scores are guaranteed to never bias the estimate, only sharpen it. The results are striking. On a real ICU sepsis benchmark, our approach reduces estimation error by three to four orders of magnitude compared to standard methods, while staying unbiased. Across stock trading, cancer treatment simulation, and maze navigation, the story is the same: more reliable estimates from the same data. As a bonus, the learned scores are interpretable. They could potentially flag which states are good or bad, giving clinicians and analysts something they can actually reason about, rather than a black-box correction term.