Sparse Autoencoders for LoRA-Adapted CLIP Under Domain Shift

Sparse autoencoders (SAEs) are increasingly used to interpret CLIP representations, but they are often trained once on base-model activations and then reused after downstream adaptation. We study whether such generic SAEs remain faithful interpreters of LoRA-adapted CLIP under domain shift. Comparing three configurations--base CLIP with a generic SAE, LoRA-adapted CLIP with the same generic SAE, and LoRA-adapted CLIP with a domain-matched SAE--we find that generic SAE reuse becomes unreliable as the target domain moves away from ImageNet-like data. The degradation appears in downstream accuracy after SAE reconstruction, reconstruction error, sparsity, latent usage, and class selectivity. Retraining the SAE on adapted CLIP activations recovers much of this lost fidelity. We further introduce a Domain-Aware Monosemanticity Score (DAMS), which penalizes broad, non-discriminative feature firing that standard top-activation monosemanticity scores can overestimate under shift.