Improved Bounds for Private and Robust Alignment

In this paper, we study the private and robust alignment of language models from a theoretical perspective by establishing upper bounds on the suboptimality gap in both offline and online settings. We consider preference labels subject to privacy constraints and/or adversarial corruption, and analyze two distinct interplays between them: privacy-first and corruption-first. For the privacy-only setting, we show that log loss with an MLE-style algorithm achieves near-optimal rates, in contrast to conventional wisdom. For the joint privacy-and-corruption setting, we first demonstrate that existing offline algorithms in fact provide stronger guarantees---simultaneously in terms of corruption level and privacy parameters---than previously known, which further yields improved bounds in the corruption-only regime. In addition, we also present the first set of results for private and robust online alignment. Our results are enabled by new uniform convergence guarantees for log loss and square loss under privacy and corruption, which we believe have broad applicability across learning theory and statistics.

Language models are often improved by asking people which of two responses they prefer. However, these preferences can be sensitive since they may reveal a person’s values, and it can raise safety concerns and private judgments. Feedback can also be unreliable or intentionally manipulated, so it is harder to train models that behave well. We study how to learn from human preferences while protecting individual choices and remaining reliable when some feedback is corrupted. Our main finding is that a simple and widely used training approach can still have strong guarantees when preference labels are privatized. We also proposed improved guarantees for settings where privacy protection and unreliable feedback occur together. We support our analysis with experiments on real-world safety preference datasets. This work helps to show how language models can be more safe when human feedback is sensitive.