Preference Goal Tuning: Post-Training as Latent Control for Frozen Policies

Goal-conditioned policies enable decision-making models to execute diverse behaviors based on specified goals, yet their downstream performance is often highly sensitive to the choice of instructions or prompts. To bypass the limitations of discrete text prompts, we formulate post-training adaptation as a latent control problem, where the goal embedding serves as a continuous control variable to modulate the behavior of a frozen policy. We propose Preference Goal Tuning (PGT), a framework that optimizes this latent control variable to align the induced trajectory distribution with task preferences. Unlike standard fine-tuning that updates policy parameters, PGT keeps the policy frozen and updates only the latent goal using a trajectory-level preference objective. This approach essentially searches for the optimal conditioning input that maximizes the likelihood of preferred behaviors while suppressing undesirable ones. We evaluate PGT on the Minecraft SkillForge benchmark across 17 tasks. With minimal data, PGT achieves average relative improvements of 72.0\% and 81.6\% on two foundation policies, consistently outperforming expert-crafted prompts. Crucially, by decoupling task alignment (latent goal) from physical dynamics (frozen policy), PGT surpasses full fine-tuning by 13.4\% in out-of-distribution settings, demonstrating superior robustness and generalization.

AI agents that follow goals, such as game or robot agents, can behave very differently depending on the exact instruction they receive. A prompt that sounds or looks clear to a person may still lead the agent to take the wrong actions. But updating the agent’s model weights for this task can make it forget other useful skills. In fact, many goal-following agents first convert an instruction into a small set of numbers, which is then used to guide the agent’s behavior. We introduce Preference Goal Tuning (PGT), which improves an agent by adjusting this goal representation instead of rewriting the underlying models. PGT lets the agent try a task, compares better and worse attempts using human judgments or task rewards, and updates the goal representation so that preferred behaviors become more likely. In Minecraft tasks, PGT improves performance beyond expert-selected prompts and remains effective when the environment changes. These results suggest that existing agents can often be adapted more safely and efficiently by learning how to steer their current abilities, rather than updating the whole model.