The Double Dilemma in Multi-Task Radiology Report Generation: A Gradient Dynamics Analysis and Solution

While multi-task learning based automatic radiology report generation (RRG) is widely adopted to ensure clinical consistency, most focus on architectural designs yet remain limited to coarse linear scalarization strategies. These strategies cannot effectively balance the hard constraints of discriminative clinical supervision with the smoothness requirements of report generation. To address these problems, we analyze the failure mechanism of linear scalarization from the perspective of gradient dynamics, utilizing the stochastic differential equation (SDE) framework to characterize it as a "Double Dilemma" of drift term deviation and diffusion term decay. Based on this, we propose a backbone-agnostic optimizer named Conflict-Averse Magnitude-Enhanced Gradient Descent (CAME-Grad). Through conflict-averse direction rectification and magnitude-enhanced energy injection, the algorithm not only ensures geometric validity, but also avoids local optimal solutions. Then, the adaptive gradient fusion mechanism is used to establish a dynamic balance between the theoretical optimal direction and the task-specific inductive bias. Experiments show that as a universal plug-and-play optimizer, CAME-Grad brings substantial and consistent improvements across eight diverse RRG methods, elevating overall clinical efficacy performance by an average of 2.3\% on MIMIC-CXR and 1.9\% on IU X-Ray. Our code is available at https://github.com/vpsg-research/CAME-Grad.

Training artificial intelligence (AI) to automatically write radiology reports is a big challenge. To ensure clinical consistency, researchers often train the AI on multiple tasks at the same time, such as identifying diseases and writing fluent sentences. However, these tasks often conflict during training, pulling the AI in different learning directions and reducing its learning momentum, a problem we call the "Double Dilemma." To solve this problem, we introduce CAME-Grad, a universal plug-and-play plugin. Acting as an "autopilot system" for the AI's learning process, it corrects the learning path and injects energy to prevent the AI from getting stuck in a learning bottleneck. Our method helps the AI effectively balance multiple tasks without compromising either one. As a result, the AI-generated medical reports read naturally while remaining accurate in diagnosing diseases, ultimately helping to reduce the heavy workload of radiologists and prevent missed diagnoses.