Timezone: »

 
How to Query Human Feedback Efficiently in RL?
Wenhao Zhan · Masatoshi Uehara · Wen Sun · Jason Lee

Fri Jul 28 07:45 PM -- 08:00 PM (PDT) @
in The Many Facets of Preference-Based Learning »
Event URL: https://openreview.net/forum?id=2ZaszaehLs »

Reinforcement Learning with Human Feedback (RLHF) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories, rather than explicit reward signals. While RLHF has demonstrated practical success in fine-tuning language models, existing empirical work does not address the challenge of how to efficiently sample trajectory pairs for querying human feedback. In this study, we propose an efficient sampling approach to acquiring exploratory trajectories that enable accurate learning of hidden reward functions before collecting any human feedback. Theoretical analysis demonstrates that our algorithm requires less human feedback for learning the optimal policy under preference-based models with linear parameterization and unknown transitions, compared to the existing literature. Specifically, our framework can incorporate linear and low-rank MDPs with efficient sample complexity. Additionally, we investigate RLHF with action-based comparison feedback and introduce an efficient querying algorithm tailored to this scenario.

Author Information

Wenhao Zhan (Princeton University)
Masatoshi Uehara (Cornell University)
Wen Sun (Cornell University)
Jason Lee (Princeton University)

More from the Same Authors