Standard reinforcement learning (RL) aims to find an optimal policy that identifies the best action for each state. However, in healthcare settings, many actions may be near-equivalent with respect to the reward (e.g., survival). We consider an alternative objective -- learning set-valued policies to capture near-equivalent actions that lead to similar cumulative rewards. We propose a model-free algorithm based on temporal difference learning and a near-greedy heuristic for action selection. We analyze the theoretical properties of the proposed algorithm, providing optimality guarantees and demonstrate our approach on simulated environments and a real clinical task. Empirically, the proposed algorithm exhibits good convergence properties and discovers meaningful near-equivalent actions. Our work provides theoretical, as well as practical, foundations for clinician/human-in-the-loop decision making, in which humans (e.g., clinicians, patients) can incorporate additional knowledge (e.g., side effects, patient preference) when selecting among near-equivalent actions.
Shengpu Tang (University of Michigan)
Shengpu Tang is a PhD candidate in the computer science department at the University of Michigan. He is a member of the Machine Learning for Data-Driven Decisions (MLD3) research group led by Jenna Wiens. His current research focuses on developing computational methods that help solve important problems in healthcare, such as risk stratification and dynamic treatment recommendations. More generally, he is interested in broader applications of AI, machine learning and graph mining, computer game design, self-driving cars, security and hacking, as well as teaching. For more details, see his website at https://shengpu-tang.me/