Rare, Distinctive, Memorized: Auditing Memorization in Fine-Tuned Medical Foundation Models

Medical foundation models fine-tuned on private patient data risk leaking individual training samples; in this regime a single recoverable image is a privacy violation. We audit per-class memorization in fine-tuned medical foundation models with a loss-difference memorization score: each canary's loss is compared between a model trained with the canary and an otherwise identical model trained without it. Memorization occurs in every architecture-dataset cell we test. Interestingly, the rarest class is not the most memorized: the rarity-monotonicity assumed by prior memorization work outside medical imaging is broken in this regime. Instead, visual distinctiveness contributes to memorization beyond rarity in our setting: the same controlled grayscale intervention applied to two classes at comparable rarity shifts memorization in opposite directions, and on a balanced dataset $M(x)$ rises from a near-zero baseline (0.004) to high memorization (0.480) under the same intervention. DP-LoRA at $\epsilon = 1$ reduces leakage from the most-memorized class by 90% in our setup while preserving usable accuracy on focused diagnostic tasks, with the protection driven by the DP noise rather than the parameter restriction. These findings point towards more privacy-preserving adaptation of medical foundation models.