Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling

Aligning large language models (LLMs) to serve users with heterogeneous and potentially conflicting preferences is a central challenge for personalized and trustworthy AI. We formalize an ideal notion of *universal alignment* through *test-time scaling*: for each prompt, the model produces $k\ge 1$ candidate responses and a user selects their preferred one. We introduce *$(k,f(k))$-robust alignment*, which requires the $k$-output model to have win rate $f(k)$ against any other single-output model, and *asymptotic universal alignment (U-alignment)*, which requires $f(k)\to 1$ as $k\to\infty$. Our main result characterizes the optimal convergence rate: there exists a family of *single-output* policies whose $k$-sample product policies achieve U-alignment at rate $f(k)=\frac{k}{k+1}$, and no method can achieve a faster rate in general. We show that popular post-training methods, including Nash learning from human feedback (NLHF), can fundamentally underutilize the benefits of test-time scaling. Even though NLHF is optimal for $k=1$, sampling from the resulting (often deterministic) policy cannot guarantee win rates above $\tfrac{1}{2}$ except for an arbitrarily small slack. This stems from a lack of output diversity: existing alignment methods can collapse to a single majority-preferred response, making additional samples redundant. In contrast, our approach preserves output diversity and achieves the optimal test-time scaling rate. In particular, we propose a family of symmetric *multi-player alignment games* and prove that any symmetric Nash equilibrium policy of the $(k+1)$-player alignment game achieves the optimal $(k,\frac{k}{k+1})$-robust alignment. Finally, we provide theoretical convergence guarantees for self-play learning dynamics in these games and extend the framework to opponents that also generate multiple responses.

AI assistants increasingly need to serve people who want different things: one user may prefer a short answer, another may want detail, and another may value a different tone or viewpoint. We ask whether one language model can better serve many such users without training a separate model for each person. The key idea is to have the model produce several possible answers at test time and let the user, or a lightweight helper system, choose the one they like best. We show mathematically that, with the right kind of training, this approach can get closer and closer to satisfying almost everyone as the number of answer options increases. For example, the best possible guarantee improves from one out of two with one answer to three out of four with three answers, and continues to approach full coverage as more answers are shown. Existing alignment methods can miss this benefit because they often learn to give the single answer preferred by the majority, reducing variety and leaving minority preferences unsupported. We propose a new game-based training framework that preserves diversity, so sampling more answers gives users meaningfully different choices. This matters because future AI systems may need to be both reliable and pluralistic: not just good on average, but able to adapt to different people’s needs and values.