Timezone: »

 
Poster
Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation
Andi Peng · Aviv Netanyahu · Mark Ho · Tianmin Shu · Andreea Bobu · Julie Shah · Pulkit Agrawal

Thu Jul 27 01:30 PM -- 03:00 PM (PDT) @ Exhibit Hall 1 #614
in Poster Session 5 »

Policies often fail at test-time due to distribution shifts---changes in the state and reward that occur when an end user deploys the policy in environments different from those seen in training. Data augmentation can help models be more robust to such shifts by varying specific concepts in the state, e.g. object color, that are task-irrelevant and should not impact desired actions. However, designers training the agent don't often know which concepts are irrelevant a priori. We propose a human-in-the-loop framework to leverage feedback from the end user to quickly identify and augment task-irrelevant visual state concepts. Our framework generates counterfactual demonstrations that allow users to quickly isolate shifted state concepts and identify if they should not impact the desired task, and can therefore be augmented using existing actions. We present experiments validating our full pipeline on discrete and continuous control tasks with real human users. Our method better enables users to (1) understand agent failure, (2) improve sample efficiency of demonstrations required for finetuning, and (3) adapt the agent to their desired reward.

[ PDF]  [ Poster] 

Author Information

Andi Peng (MIT)
Aviv Netanyahu (MIT)
Mark Ho (New York University)
Tianmin Shu (Massachusetts Institute of Technology)
Andreea Bobu (University of California Berkeley)
Julie Shah (MIT)
Pulkit Agrawal (MIT)

More from the Same Authors