T-POP: Test-Time Personalization with Online Preference Feedback

Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses. However, current personalization methods are ill-suited for new users, as they typically require either slow, resource-intensive fine-tuning or a substantial amount of pre-existing user data, creating a significant cold-start problem. To address this challenge, we introduce a new paradigm for real-time personalization by learning from online pairwise preference feedback collected during text generation. We propose T-POP (Test-Time Personalization with Online Preference Feedback), a novel algorithm that synergistically combines test-time alignment with dueling bandits. Without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences. By leveraging dueling bandits, T-POP intelligently queries the user to efficiently balance between exploring their preferences and exploiting the learned knowledge to generate personalized text. Extensive experiments demonstrate that T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines and showing consistent improvement with more user interactions.

As AI assistants shift from answering isolated questions to supporting ongoing work, they need to adapt to each person. But personalization often faces a cold-start problem: for a new user, the assistant knows almost nothing about them. Most current approaches depend on stored histories, setup, or retraining, making personalization slow, costly, and hard to scale. We built T-POP to let an assistant learn during normal use. The system shows two possible answers and asks which one the user prefers. Behind the scenes, T-POP uses dueling bandits, a strategy for deciding when to try responses that teach the system more and when to use what it has already learned. From these simple choices, it guides future responses while keeping the original language model unchanged. This matters because useful personalization should not require long personal histories or large computing budgets. By learning from small amounts of feedback as people interact with the system, T-POP offers a practical path for AI assistants that become helpful sooner.