Sparse Autoencoders are Topic Models

Sparse autoencoders (SAEs) are used to analyze embeddings, but their role and practical value are debated. We propose a new perspective on SAEs by demonstrating that they can be naturally understood as topic models. We propose a continuous topic model (CTM) inspired by Latent Dirichlet Allocation (LDA) for embedding spaces and derive the SAE objective as a maximum a posteriori estimator under this model. This view implies SAE features are thematic components rather than steerable directions. To confirm our theoretical findings, we introduce SAE-TM, a topic modeling framework that: (1) trains an SAE to learn reusable topic atoms, (2) interprets them as word distributions on downstream data, and (3) merges them into any number of topics without retraining. SAE-TM yields more coherent topics than strong baselines on text and image datasets while maintaining diversity. Finally, we analyze thematic structure in image datasets and trace topic changes over time in Japanese woodblock prints. Our work positions SAEs as effective tools for large-scale thematic analysis across modalities. Code is available at https://github.com/ExplainableML/SAE-TM.

Modern AI systems represent text and images as long lists of numbers, internal "fingerprints" that encode what each system has learned. Sparse autoencoders are a popular tool for taking these fingerprints apart into smaller pieces, but researchers have actively debated what those pieces actually mean. We show that sparse autoencoders are, in a precise mathematical sense, topic models, which are methods that automatically discover the recurring themes in a large collection of documents or images. This explains why they sometimes disappoint as fine-grained "control knobs" for AI behavior: each extracted piece is best understood as one thematic ingredient that combines with others, rather than as a single isolated concept. Building on this insight, we develop a practical method that trains a single sparse autoencoder once and then reuses it to find themes in any new dataset, with as many or as few themes as a user wants, with no retraining required. On both text collections (news articles, movie reviews, tweets) and image collections, our approach finds themes that are clearer and more interpretable than those produced by existing topic modeling methods. We then apply the same tool to compare the contents of popular image datasets used to train AI, and to trace how the themes of Japanese woodblock prints shifted from the 1780s to the 1940s, showing the technique is useful not only for AI research but also for studying art and cultural history.