Timezone: »
Poster
Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making
Chengchun Shi · Runzhe Wan · Rui Song · Wenbin Lu · Ling Leng
Thu Jul 16 05:00 PM -- 05:45 PM & Fri Jul 17 04:00 AM -- 04:45 AM (PDT) @
The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an important role for identifying the optimal policy in high-order Markov decision processes (MDPs) and partially observable MDPs. Theoretically, we establish the validity of our test. Empirically, we apply our test to both synthetic datasets and a real data example from mobile health studies to illustrate its usefulness.
Author Information
Chengchun Shi (London School of Economics and Political Science)
Runzhe Wan (North Carolina State University)
Rui Song (North Carolina State University)
Wenbin Lu (North Carolina State University)
Ling Leng (Amazon)
More from the Same Authors
-
2023 Poster: Multiplier Bootstrap-based Exploration »
Runzhe Wan · Haoyu Wei · Branislav Kveton · Rui Song -
2023 Poster: Multiply Robust Off-policy Evaluation and Learning under Truncation by Death »
Jianing Chu · Shu Yang · Wenbin Lu -
2023 Poster: An Instrumental Variable Approach to Confounded Off-Policy Evaluation »
Yang Xu · Jin Zhu · Chengchun Shi · Shikai Luo · Rui Song -
2023 Poster: A Robust Test for the Stationarity Assumption in Sequential Decision Making »
Jitao Wang · Chengchun Shi · Zhenke Wu -
2023 Poster: A Reinforcement Learning Framework for Dynamic Mediation Analysis »
Lin Ge · Jitao Wang · Chengchun Shi · Zhenke Wu · Rui Song -
2022 Poster: Safe Exploration for Efficient Policy Evaluation and Comparison »
Runzhe Wan · Branislav Kveton · Rui Song -
2022 Spotlight: Safe Exploration for Efficient Policy Evaluation and Comparison »
Runzhe Wan · Branislav Kveton · Rui Song -
2022 Poster: A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes »
Chengchun Shi · Masatoshi Uehara · Jiawei Huang · Nan Jiang -
2022 Oral: A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes »
Chengchun Shi · Masatoshi Uehara · Jiawei Huang · Nan Jiang -
2021 Poster: Deeply-Debiased Off-Policy Interval Estimation »
Chengchun Shi · Runzhe Wan · Victor Chernozhukov · Rui Song -
2021 Oral: Deeply-Debiased Off-Policy Interval Estimation »
Chengchun Shi · Runzhe Wan · Victor Chernozhukov · Rui Song -
2020 Poster: Causal Effect Estimation and Optimal Dose Suggestions in Mobile Health »
Liangyu Zhu · Wenbin Lu · Rui Song -
2020 Poster: On Validation and Planning of An Optimal Decision Rule with Application in Healthcare Studies »
Hengrui Cai · Wenbin Lu · Rui Song