Med-SegLens: Latent-Level Model Diffing for Interpretable Medical Image Segmentation

Modern segmentation models achieve strong predictive performance but remain largely opaque, limiting our ability to diagnose failures, understand dataset shift, or intervene in a principled manner. We introduce $\textbf{Med-SegLens}$, a model-diffing framework that decomposes segmentation model activations into interpretable latent features using sparse autoencoders trained on SegFormer and U-Net. Through cross-architecture and cross-dataset latent alignment across healthy, adult, pediatric, and sub-Saharan African glioma cohorts, we identify a stable backbone of shared representations, while dataset shift is driven by differential reliance on population-specific latents. We show that these latents act as causal bottlenecks for segmentation failures, and that targeted latent-level interventions can correct errors and improve cross-dataset adaption without retraining, recovering performance in 70% of failure cases and improving Dice score from 39.4% to 74.2%. Our results demonstrate that latent-level model diffing provides a practical and mechanistic tool for diagnosing failures and mitigating dataset shift in segmentation models.

Doctors use AI to outline tumors in medical scans, but these systems can fail when patients come from different hospitals, age groups, or populations. The problem is that we often do not know why the AI fails or how to fix it without retraining the whole model. We introduce Med-SegLens, a tool that looks inside medical segmentation models and breaks their internal decisions into smaller, understandable pieces. By comparing models across healthy, adult, pediatric, and Sub-Saharan African brain MRI datasets, we found that some internal features are shared across groups, while others are specific to certain populations. Most importantly, we show that some of these features directly cause segmentation mistakes. By adjusting them, Med-SegLens can correct many failures without retraining the model, recovering performance in about 70% of failure cases. This makes AI medical imaging systems easier to inspect, diagnose, and improve when they face real-world patient differences.