Convex Optimization for Alignment and Preference Learning on a Single GPU

Fine-tuning large language models (LLMs) to align with human preferences has driven the success of systems such as Gemini and ChatGPT. However, approaches like Reinforcement Learning from Human Feedback (RLHF) remain computationally expensive and complex. Direct Preference Optimization (DPO) offers a simpler alternative but has limitations such as inconsistent ranking accuracy, high dependence on GPU resources, and expensive hyperparameter tuning. We propose the Convex Optimization for Alignment and Preference Learning Algorithm (COALA): a novel lightweight strategy with strong theoretical guarantees. By leveraging the convex optimization reformulation of neural networks, COALA eliminates the need for a reference model and obtains significant reduction in both training time and VRAM consumption, thus enabling efficient training on a single GPU. Experiments across four datasets—including a 26621-sample synthetic Educational Feedback dataset—and six models (including Llama-3.1-8B) demonstrate COALA's competitive performance and efficiency while utilizing as little as ${\sim}17.6\%$ of DPO's total TFLOPs. COALA exhibits stable, monotonically increasing rewards and reaches peak margins in significantly shorter time in comparison to traditional methods such as DPO and ORPO. To the best of our knowledge, this is the first time convex optimization has been effectively applied to preference fine-tuning of LLMs.

Teaching LLMs to act as personal assistants by following human preferences is increasingly prevalent and important. However, the current methods used to train these massive models are computationally expensive, require an army of expensive specialized hardware, and are frustratingly slow. We introduce a new, lightweight method called COALA. Instead of relying on traditional, complex training loops that require expensive resources and power, COALA uses a branch of mathematics called convex optimization to frame the problem efficiently. This mathematical shortcut dramatically simplifies the training process, allowing anyone to align LLMs using just a single GPU. Since your data never leaves your machine, this has the additional advantage of data security and privacy. When tested across various models—including the popular Llama model family—COALA achieved highly competitive performance while using as little as 17.6% of the computational cost incurred by popular standard methods. We are sharing our algorithm and a new Educational Tutor dataset to help make advanced AI training faster, more stable, more practical and accessible to anyone with a single computer.