S2M-Net: Spectral-Spatial Mixing with Morphology-Aware Adaptive Loss for Medical Image Segmentation

Medical image segmentation requires balancing global context with computational efficiency, where self-attention mechanisms suffer from quadratic $\mathcal{O}((HW)^2 C)$ complexity. We propose S2M-Net, a parameter-efficient architecture (4.7M parameters) that achieves computational savings through Spectral--Spatial Token Mixing (SSTM). SSTM achieves $\mathcal{O}(HWC^2)$ complexity through efficient combination of $\mathcal{O}(HWC \log(HW))$ frequency-domain processing and $\mathcal{O}(HWCd)$ bottlenecked spatial gating ($d{=}16$), exploiting spectral concentration where $>93\%$ of energy is captured by $K{=}32$ low-frequency components ($\sim$0.8\% of the spectrum at $352{\times}352$ resolution). This design avoids self-attention's prohibitive $\mathcal{O}((HW)^2C)$ attention map computations while preserving global receptive fields. To handle geometric diversity, we introduce Morphology-Aware Adaptive Segmentation Loss (MASL), which automatically modulates five loss objectives based on per-sample morphological descriptors (tubularity, compactness, irregularity, and scale). Evaluation across 15 datasets spanning 8 modalities demonstrates competitive performance, obtaining the best performance on 14 of 15 datasets, with statistically significant improvements ($p < 0.0033$, Bonferroni-corrected) on 7 challenging tasks (complex morphology, class imbalance, and multi-class segmentation), and clinically meaningful gains ($0.5$--$1.6\%$ Dice) on 8 mature benchmarks. Notably, S2M-Net achieves $83.43\%$ Dice on EndoVis17 multiclass instrument segmentation ($+8.69\%$ over TransUNet and $+9.14\%$ over the best baseline UMamba at $74.29\%$), while using $12.8{\times}$ fewer parameters (4.7M vs.\ 60M).

Medical image segmentation (the task of precisely outlining organs, tumors, or surgical instruments in scans) is a cornerstone of modern clinical care, supporting everything from cancer diagnosis to robotic surgery planning. Today's best AI models for this task borrow heavily from large language model technology, using mechanisms that examine every part of an image in relation to every other part simultaneously. This is powerful, but extraordinarily expensive: as image resolution grows, computational cost explodes, putting these tools out of reach for hospitals with limited hardware. We introduce S2M-Net, a compact AI model that achieves state-of-the-art segmentation accuracy while using up to 12.8 times fewer parameters than leading alternatives. Our key insight is that medical images (unlike natural photographs) concentrate most of their meaningful information in low frequency patterns, the broad shapes and smooth gradients that define organ boundaries. By working primarily in the frequency domain and processing only the most informative signal components, S2M-Net captures the global picture efficiently without the computational explosion of conventional approaches. Beyond architecture, we introduce a smart training objective that automatically adapts to the geometry of whatever structure is being segmented, behaving differently for thin blood vessels, irregular tumors, and compact polyps, without requiring manual tuning for each new dataset. Tested across 15 diverse medical imaging datasets spanning MRI, CT, ultrasound, endoscopy, and retinal imaging, S2M-Net outperforms all baselines on 14 of 15 benchmarks. Its small footprint makes it deployable on standard clinical workstations and edge devices, broadening access to AI assisted diagnostics in resource limited healthcare settings worldwide.