FACT: Fuzzy Alignment with Comorbidity Topology for Reliable Multi-Label Medical Image Diagnosis

In clinical practice, patients often present with multiple co-occurring diseases, yet most existing Multi-Label-Diagnosis (MLD) methods treat diagnosis as a rigid discriminative partitioning task, implicitly assuming that overlapping pathologies are separable. This assumption is problematic in medical images, where identical or highly similar visual observations may simultaneously support multiple disease labels, and disease concepts are inherently correlated rather than independent. Enforcing hard decision boundaries under such overlap suppresses shared evidence, biases feature representations, and ultimately undermines model reliability. To address this limitation, we propose Fuzzy Alignment with Comorbidity Topology FACT, a novel paradigm that reformulates MLD as a fuzzy alignment problem between atomic visual evidence and disease semantic anchors. FACT is characterized by three key features: (1) modeling visual polysemy through shared and reusable atomic visual evidence; (2) encoding disease correlation via semantic anchors structured by comorbidity topology; and (3) employing a metric-based fuzzy membership function for non-discriminative visual-semantic alignment. Extensive experiments on three public clinical benchmarks demonstrate that FACT consistently improves diagnostic performance while delivering clinically plausible predictions. The code is available at \url{https://github.com/yyuChen9/FACT}.

In medical imaging, a single patient may have several diseases at the same time, and these diseases can share similar visual signs. However, many AI systems still try to separate each disease with hard decision boundaries, as if every disease had completely distinct image patterns. This can be unreliable when the same visual evidence may support multiple diagnoses. We propose FACT, a new AI framework for multi-label medical image diagnosis. Instead of forcing each image feature to belong to only one disease, FACT allows visual evidence to partially support multiple diseases. It also uses known relationships between diseases, such as common co-occurrence patterns, to make the disease representations more clinically meaningful. In this way, FACT treats diagnosis more like a flexible matching process between image evidence and disease concepts. We evaluate FACT on several public medical imaging datasets, including challenging long-tailed and noisy settings. The results show that FACT improves diagnostic performance and produces more clinically plausible predictions. This work may help build more reliable AI tools for medical image analysis, especially when diseases overlap or appear together.