SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment

Unifying multimodal understanding and generation is a compelling frontier that is beginning to emerge in the medical field. However, the limited existing unified medical models typically treat understanding and generation as disjoint objectives, lacking a meaningful functional synergy. In this work, we identify and address a critical question in unified medical modeling: what form of “understanding” truly benefits generation. We present SynerMedGen, a unified framework built on the proposed principle of generation-aligned understanding, which synergizes understanding objectives with generation tasks via task alignment. SynerMedGen introduces three generation-aligned understanding tasks and a two-stage training strategy that transfers generation-beneficial representations learned during understanding training to medical image synthesis. Remarkably, even with understanding training alone, our SynerMedGen achieves strong zero-shot performance across 22 medical image synthesis tasks and demonstrates robust generalization. When combined with generation training, SynerMedGen consistently outperforms state-of-the-art specialized medical image synthesis models as well as recent unified medical models. We also release SynerMed, a large-scale dataset of 1M paired synthesis samples and 2M understanding instances for studying understanding–generation synergy. Our project can be accessed at https://github.com/piooip/SynerMedGen.

Medical AI systems are increasingly expected not only to read scans, but also to create useful images, such as showing how the same patient might look in a different scan type, like CT or MRI. This is hard because the AI must change the scan's appearance while keeping the patient's anatomy and disease details unchanged. Many current systems learn these two abilities separately, so they may answer questions about scans but still create images with missing details or made-up structures. We introduce SynerMedGen, a way to teach an AI system the visual details it needs before creating medical images. Using paired scans from the same patient, the system learns to find the matching image slice, recognize the scan type, and describe what should change and what must stay fixed when moving between scan types. These lessons then guide image creation. We also build SynerMed, a dataset with 1 million paired examples and 2 million related questions. In tests covering 22 tasks, SynerMedGen creates more accurate images than previous systems and works better on new tasks and datasets. This suggests that teaching AI the right kind of understanding can make medical image creation more faithful and reliable.