Timezone: »

 
Complementing a Policy with a Different Observation Space
Gokul Swamy · Sanjiban Choudhury · J. Bagnell · Steven Wu

@
in Interactive Learning with Implicit Human Feedback »
Event URL: https://openreview.net/forum?id=ukEv7WNotJ »

We consider the problem of improving upon a black-box policy which operates on a different observation space than the learner. Such problems occur when augmenting an existing hand-engineered system with a new machine learning model or in a shared autonomy / human-AI complementarity context. We prove that following the naive policy gradient can lead to a decrease in performance because of incorrect grounding in a different observation space. Then, if we have access to both sets of observation at train time, we derive a method for correctly estimating a policy gradient via an application of the backdoor criterion. If we don't, we prove that under certain assumptions, we can use the proxy correction to correctly estimate a direction of improvement.

Author Information

Gokul Swamy (Carnegie Mellon University)
Sanjiban Choudhury (Cornell University)
J. Bagnell (Aurora Innovation)
Steven Wu (Carnegie Mellon University)

More from the Same Authors