DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and Its Loss' Convexity is Dispensable)

Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimization (DPO) cleverly bypasses reward modeling by making an explicit link with a specific normative model of human choice. Our paper elevates this connection to the full generality of DPO's normative framework. Getting there requires reworking social choice theory's textbook path for a better RLHF/ML fit. It elevates the connection to a remarkably broad viewpoint on preference optimization, considering the current panorama of DPO follow-ups. It also unveils unexpected riches for ML, chief among which the support for non-convex losses, the fact that any compliant ML analytical choice can be embedded with any human choice model, and a normative framework's umbrella wide enough to safeguard DPO's extensions (margins, length correction, ...). A toy experiment ``far away'' from the DPO crowd is given.

How can we improve the likelihood that a LLM learned by an algorithm from human preference data ends up producing outputs aligned with human preferences ? By carving, in the objective function (the core function that any ML algorithm has to optimize), the explicit “knowledge” that the data is human in nature and not just any kind of data. To do so, Direct Preference Optimization (DPO) introduced the use of a popular model of human choice. Elevating this very specific model to match ML’s required generality has proven tricky so far, to the extent that some have opted to discard it, a position whose associated risks are hard to exaggerate in the context of LLMs. Our paper closes the gap, giving this human choice model a generalization that matches ML’s generality. Doing so got us to a surprising result: instead of having each applicable objective function correspond to a specific model of human choice (for example authorizing abstention or not, etc.), each can be mapped to all of them. That the ML objective is in fact “secretly disentangled” in human choice is thought-provoking, but it is not the only surprise we got: the conditions on “applicable objective functions” are indeed weaker than what is usually required from such functions in general ML. Our findings unveil an extensive, essentially still unexplored map to design the ML task and considerably relax its design choices in the human context.