AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in Unified Multimodal Models via Decompositional Verifiable Reward

In this paper, we propose AlphaGRPO, a novel framework that applies Group Relative Policy Optimization (GRPO) to AR-Diffusion Unified Multimodal Models (UMMs) to enhance multimodal generation capabilities without relying on external knowledge injection. Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks: Reasoning Text-to-Image Generation, where the model actively infers implicit user intents, and Self-Reflective Refinement, where it autonomously diagnoses and corrects misalignments in generated outputs. To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward). Unlike holistic scalar rewards, DVReward utilizes an LLM to decompose complex user requests into atomic, verifiable semantic and quality questions, which are then evaluated by a general MLLM to provide reliable and interpretable feedback. Extensive experiments demonstrate that AlphaGRPO yields robust improvements across multimodal generation benchmarks, including GenEval, TIIF-Bench, DPG-Bench and WISE, while also achieving significant gains in editing tasks on GEdit without training on editing tasks. These results validate that our self-reflective reinforcement approach effectively leverages inherent understanding to guide high-fidelity generation.

Imagine asking an AI to paint "two orange cats sitting on a red couch" and getting three cats on a blue couch instead. Modern image-generating AI models are skilled artists, but poor proofreaders — they produce an image in one shot and rarely check whether it matches your request. The standard fix is to train them by mimicking the output of a larger, more expensive AI "teacher," but this raises a question: are the gains coming from real learning, or just from copying a smarter teacher? We developed AlphaGRPO, a method that teaches a unified AI model to inspect and refine its own creations, much like a student catching mistakes in their own essay. Rather than asking a single "is this image good?" question, our system breaks each request into simple yes/no checks — "Are there two cats?", "Is the couch red?" — and uses a separate AI to grade each one, providing precise, interpretable feedback. The result: the model becomes noticeably better at generating accurate images on standard benchmarks, and surprisingly gets better at editing photos too — even though it was never directly trained for editing. This shows that AI models can learn to think and self-correct without copying a more expensive teacher.