FTC: Fourier Transcoder for Monosemantic Sparse Feature Decomposition in Vision Language Model

Sparse autoencoders (SAEs) and transcoders (TCs) have been successfully applied in interpreting neural activations in large language models (LLMs), particularly those in MLP sublayers, through sparse feature representations. Building on this success, SAEs and TCs have also been extended to vision-language models (VLMs), including CLIP, with promising results. However, significant challenges remain, including feature polysemanticity and incomplete disentanglement. In this work, we introduce the Fourier Transcoder (FTC), a transcoder architecture based on a Fourier basis, to analyze MLP sublayers in VLM. We show that FTC discovers more monosemantic features and reduces the number of dead features in CLIP compared to TopK TC.