Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Random

In offline Reinforcement Learning, immediate rewards in logged batch data are often unobserved due to sparse or irregular record-keeping, or censored beyond certain reward values. This issue arises in practical settings, including health care and marketing. We investigate off-policy evaluation (OPE) in finite-horizon Markov decision processes when rewards are missing not at random (MNAR), which breaks ignorability and induces selection bias even after conditioning on states and actions. To address this, we formalize a reward-dependent propensity model and use future states as shadow variables to identify the full-data conditional mean reward. We further introduce a bridge function that recovers the conditional mean reward without explicitly modeling the MNAR mechanism, and estimate it via a min-max procedure to avoid double sampling. Building upon these identification results, we propose an Fitted-Q-Evaluation-style estimator that propagates the recovered rewards while allowing target policies to depend on past missingness indicators. Finally, we establish consistency and finite-sample error bounds for our OPE estimator, and show through experiments the strong performance of our method compared to existing methods on simulated and MIMIC-III Sepsis data.

Machine-learning systems are often used to choose actions over time, such as treatment choices for patients or marketing actions for customers. Before using a new decision rule, researchers would like to know how well it would have performed using only past records, because testing it directly can be risky or expensive. This becomes difficult when important outcomes are missing for a reason: for example, a patient’s health survey may be absent because of how the patient is doing, even though later lab results are recorded. Standard approaches that ignore missing outcomes, fill them in too simply, or give large weight to some records can produce biased estimates. We develop a method that uses later information in the record as clues to recover information about the missing outcomes. This lets us estimate the value of new decision rules, including rules that react to whether earlier outcomes were recorded. We prove when the method is reliable and show, in simulations and a sepsis-care dataset with simulated missing outcomes, that it gives more accurate estimates than existing alternatives under biased missing outcomes.