Learning Optimal Advantage from Preferences and Mistaking it for Reward

Most recent work that involves learning reward functions from human preferences over pairs of trajectory segments---as used in reinforcement learning from human feedback (RLHF), including for ChatGPT and many contemporary language models---are built with the assumption that such human preferences are generated based only upon the reward accrued within those segments, which we call their partial return.But if this assumption is false because people base their preferences on information other than partial return, then what type of function is their algorithm learning from preferences? We argue that this function is better thought of as an approximation of the optimal advantage function, not a reward function as previously believed.