PortraitRL: Reinforcement Learning for Personalized Portrait Pose Transfer with Multi-Objective Reward Modeling

Portrait pose transfer (PPT) requires generative models to preserve fine-grained identity details while following complex pose and layout modification instructions. Existing methods often struggle with extensive data annotation requirements or employ optimization objectives that are suboptimal for addressing PPT's two key challenges. In this work, we propose PortraitRL, a novel post-training framework that addresses these challenges with a multi-objective reward mechanism. Specifically, we employ LVLM-based reward functions to effectively evaluate PPT's two challenges and apply within-group standardization to eliminate scale differences, allowing these rewards to effectively guide optimization. More importantly, we devise a novel reinforcement learning algorithm, Negative-aware Score Preference Optimization (NaSPO), which automatically identifies positive and negative preference samples through within-group advantages, eliminating annotation requirements while fully leveraging both positive and negative learning signals. Extensive experiments show state-of-the-art performance, with significant improvements in both detail preservation and editing accuracy.

Imagine wanting an AI to change the pose of a person in a photo—like making them turn their head—while keeping every fine detail of their identity exactly the same. This task, known as portrait pose transfer, is surprisingly difficult. Existing methods either require humans to painstakingly label thousands of training images, or they struggle to balance two conflicting goals: altering the pose without distorting the person’s face. In this work, we introduce PortraitRL, a training framework that solves these challenges. We use advanced vision-AI models as "judges" to automatically score how well the AI balances both the new pose and the original details. More importantly, we designed a novel learning algorithm called NaSPO that allows the AI to teach itself. Instead of relying on human annotations, NaSPO automatically identifies its own best and worst attempts, allowing the model to learn efficiently from both its successes and its mistakes. Our approach eliminates the need for tedious manual data labeling while achieving state-of-the-art results, making portrait editing significantly more accurate and lifelike.