Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO allows for a more unified view over preference optimization, encompasses existing popular algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, revealing its connections and differences from the KL divergence regularization required by the canonical RLHF formulation. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners.