Proxy objectives are a fundamental concept in machine learning. That is, there's a true objective that we care about, but it's hard to compute or estimate, so instead we construct a locally-valid approximation and optimize that. I will examine reinforcement from human feedback with this lens, as a chain of approximations, each of which can widen the gap between the desired and achieved result.