PICACO: Pluralistic In-Context Value Alignment via Total Correlation Optimization

In-Context Learning has shown great potential for aligning Large Language Models (LLMs) with human values, helping reduce harmful outputs and accommodate diverse preferences without costly post-training, known as In-Context Alignment (ICA). However, LLMs' comprehension of input prompts remains agnostic, limiting ICA's ability to address value tensions—human values are inherently pluralistic, often imposing conflicting demands, e.g., stimulation vs. tradition. Current ICA methods therefore face the Instruction Bottleneck challenge, where LLMs struggle to reconcile multiple intended values within a single prompt, leading to incomplete or biased alignment. To address this, we propose PICACO, a novel pluralistic ICA method. Without fine-tuning, PICACO optimizes a meta-instruction that incorporates multiple values to better elicit LLMs' understanding of them and improve alignment. This is achieved by maximizing the total correlation between specified values and LLM responses, which theoretically reinforces value conformity and reduces distractive noise, resulting in more effective instructions. Extensive experiments on five value sets show that PICACO works well with both black-box and open-source LLMs, outperforms several recent strong baselines, and achieves a better balance across up to 8 distinct values.

AI chatbots like ChatGPT are increasingly used for advice, writing, and decision-making. But people hold diverse values: what one person considers helpful, another might find harmful. Today's methods for aligning AI with human values often focus on one value at a time, causing the AI to overlook or favor certain values over others when multiple values are at play. We call this the Instruction Bottleneck problem. We developed PICACO, a method that helps AI systems respect multiple human values simultaneously and without expensive retraining. PICACO works by automatically crafting a carefully worded instruction that teaches the AI to balance competing values, such as being both honest and kind, or respecting tradition while encouraging personal freedom. It does this by iteratively refining the instruction so that the AI's responses better reflect all intended values rather than defaulting to just one. In experiments across five different value sets and multiple AI models, PICACO consistently outperformed existing methods at producing responses that balanced up to eight distinct values. This work is a step toward AI systems that can better accommodate the diversity of human values across cultures and individuals.