REAR: Test-time Preference Realignment through Reward Decomposition

Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task. While post-training methods can adapt models to specific needs, they often require costly data curation and additional training. Test-time scaling (TTS) presents an efficient, training-free alternative, but its application has been largely limited to verifiable domains like mathematics and coding, where response correctness is easily judged. To extend TTS to preference alignment, we introduce a novel framework that models the task as a realignment problem, since the base model often fails to sufficiently align with the stated preference. Our key insight is to decompose the underlying reward function into two components: one related to the question and the other to preference information. This allows us to derive a REAlignment Reward (REAR) that selectively rescales the proportions of these two reward terms. We then show that REAR can be formulated as a linear combination of token-level policy log-probabilities, making it computationally efficient and easy to integrate with various TTS algorithms such as best-of-$N$ sampling and tree search. Experiments show that compared to other test-time baselines, REAR not only enables scable test-time realignment for preference alignment tasks under diverse user requirements, but also generalizes to mathematical and visual tasks under appropriate preference settings.

When you ask an AI chatbot for help, you often have personal preferences about how you want the answer -- maybe you dislike a certain style, or want it tailored to your situation. Today's AI models are trained once on broad data, so they don't always follow these individual preferences well, and customizing them for every user would require expensive retraining. We introduce a method called REAR that adjusts an AI model's answers to better match a user's stated preference without any retraining. The idea is simple: when the model considers several possible answers to a question, REAR scores each one by separating "how well it answers the question" from "how well it respects the preference," and then gives extra weight to the preference part. Picking the highest-scoring answer steers the model toward responses the user actually wants. Because REAR reuses information the model already produces while generating text, it is fast and easy to add on top of existing systems. In our experiments it followed user preferences more reliably than comparable approaches, and the same idea also improved performance on math and image-understanding tasks when the goal was phrased as a preference. This offers a low-cost way to make AI assistants more personalized and responsive to what each person needs.