Timezone: »

 
Spotlight
Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning
Nathan Kallus · Xiaojie Mao · Kaiwen Wang · Zhengyuan Zhou

Thu Jul 21 10:50 AM -- 10:55 AM (PDT) @ Room 301 - 303
in Reinforcement Learning »
Off-policy evaluation and learning (OPE/L) use offline observational data to make better decisions, which is crucial in applications where online experimentation is limited. However, depending entirely on logged data, OPE/L is sensitive to environment distribution shifts --- discrepancies between the data-generating environment and that where policies are deployed. Si et al., (2020) proposed distributionally robust OPE/L (DROPE/L) to address this, but the proposal relies on inverse-propensity weighting, whose estimation error and regret will deteriorate if propensities are nonparametrically estimated and whose variance is suboptimal even if not. For standard, non-robust, OPE/L, this is solved by doubly robust (DR) methods, but they do not naturally extend to the more complex DROPE/L, which involves a worst-case expectation. In this paper, we propose the first DR algorithms for DROPE/L with KL-divergence uncertainty sets. For evaluation, we propose Localized Doubly Robust DROPE (LDR$^2$OPE) and show that it achieves semiparametric efficiency under weak product rates conditions. Thanks to a localization technique, LDR$^2$OPE only requires fitting a small number of regressions, just like DR methods for standard OPE. For learning, we propose Continuum Doubly Robust DROPL (CDR$^2$OPL) and show that, under a product rate condition involving a continuum of regressions, it enjoys a fast regret rate of $O(N^{-1/2})$ even when unknown propensities are nonparametrically estimated. We empirically validate our algorithms in simulations and further extend our results to general $f$-divergence uncertainty sets.
Keywords: OPT: Learning for Optimization · OPT: Optimization and Learning under Uncertainty · OPT: Stochastic · RL: Batch/Offline · RL: Risk Sensitive · T: Reinforcement Learning and Planning

Author Information

Nathan Kallus (Cornell University)
Xiaojie Mao (Tsinghua University)
Kaiwen Wang (Cornell University and Cornell Tech)
Zhengyuan Zhou (Arena Technologies)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors