GenAlign: Towards Unified Alignment Framework of MLLMs via Generative Reward Model

Aligning Multimodal Large Language Models (MLLMs) with human preferences remains a fundamental challenge. While Generative Reward Models (GRMs) offer a promising reasoning-based alternative to scalar models, they are often hindered by severe position bias and prohibitively high computational overhead. To address these limitations, we propose GenAlign, a unified framework that synergizes robust generative reward modeling with efficient MLLM alignment. First, we introduce a rubric-based GRM that explicitly models the preference judgment process. By employing reinforcement learning with verifiable rewards and an online position debiasing mechanism, our model produces interpretable reasoning critiques and robust preference predictions. Second, we propose a policy optimization strategy utilizing advantage-smoothed dynamic reference anchoring. This approach reduces computational complexity while mitigating gradient instability caused by variance collapse. Extensive experiments demonstrate that GenAlign achieves state-of-the-art preference prediction accuracy on multimodal reward modeling benchmarks. Moreover, it consistently improves the performance of three MLLMs across seven diverse evaluation benchmarks, particularly making significant progress in safety and hallucination.

Artificial Intelligence systems that can process both text and images are becoming remarkably powerful, but teaching them to consistently follow human values remains a major challenge. Developers currently rely on "AI judges" to evaluate and reward these models during training. Unfortunately, these judges often struggle with bias and favor answers simply based on the order they are presented, and require massive amounts of computing power. To solve this, we introduce GenAlign, a unified system designed to better align AI behavior with what humans actually want. GenAlign improves the training process through two key innovations. First, we developed a smarter AI judge that uses a clear grading rubric and explicitly explains its reasoning, ensuring it cannot be easily tricked by the mere order of the answers it reviews. Second, we created a faster, more stable, and highly efficient method for the main AI to learn from this judge's feedback. When tested, GenAlign proved to be highly accurate at predicting human preferences. Furthermore, applying this system to train three different AI models significantly improved their overall performance, making them much safer to use and far less likely to confidently invent false information.