Turning Drift into Constraint: Robust Reasoning Alignment in Non-Stationary Multi-Stream Environments

This paper identifies a critical yet underexplored challenge in reasoning alignment from multiple multi-modal large language models (MLLMs): In non-stationary environments, the diverse reasoning distributions of source models often evolve unpredictably, transmitting systematic biases and drift to the target model. To address this, we formulate multi-source reasoning alignment as a constraint satisfaction problem under concept drift theory. We propose Autonomous Preference Optimization (APO), a novel framework that treats inter-model divergences not as noise, but as dynamic negative constraints. APO operates via a two-stage protocol: first, supervised bootstrapping projects the target model into the capability union of source models; second, constraint-aware optimization synthesizes a consistent consensus manifold by explicitly suppressing drifting trajectories via a multi-negative Plackett-Luce objective. Extensive experiments on chest X-ray interpretation demonstrate that our 7B model achieves superior robustness, outperforming even proprietary source models in average accuracy. Furthermore, we release CXR-MAX, a large-scale benchmark comprising 170,982 reasoning trajectories from seven large-scale MLLMs to facilitate research on reasoning alignment under drift. Code and data are available at: https://github.com/XiaoyuYoung/APO.

Researchers are increasingly using artificial intelligence (AI) systems with multiple large models. However, combining knowledge from multiple AI is difficult because these systems often change over time and may develop different reasoning habits or biases. When these inconsistencies are passed to a new model, they can reduce reliability and make it less trustworthy, especially in some safety-critical areas, like medical diagnosis. In this work, we introduce a new training approach called Autonomous Preference Optimization (APO). Instead of treating disagreements between AI systems as simple errors, APO uses them as signals to identify and avoid unreliable reasoning patterns. The method first teaches a new model to learn from the strengths of several existing AI systems, and then helps it build more stable and consistent decision-making behavior by filtering out drifting or conflicting reasoning processes. We evaluate APO on chest X-ray interpretation tasks and show that our compact 7-billion-parameter model achieves strong and reliable performance, even outperforming some larger proprietary AI systems on average. To support future research, we also release CXR-MAX, a large-scale benchmark dataset containing more than 170,000 medical reasoning examples collected from seven state-of-the-art multimodal AI models. Our work aims to make AI systems more robust, reliable, and trustworthy in real-world clinical environments where conditions and data can change over time.