Improving Topic Modeling by Distilling Soft Labels from Language Models

Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we introduce a novel topic model training framework by Distilling Soft Labels (DSL) from Language Models (LMs). To construct the contextually enriched reconstruction signals, we project the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary, and train the topic models to reconstruct the soft labels using the LM hidden states. This produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Extensive experiments demonstrate that DSL achieves substantial improvements in topic coherence and assignment accuracy over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications.

Topic modeling helps researchers and practitioners make sense of large collections of text by automatically discovering the main themes that appear across documents. Traditional topic models usually do this by looking at which words occur in each document. This works well in some cases, but it can miss the broader meaning of a document, especially when the text is short or when related documents use different words. We a new way to train topic models using small language models. Instead of asking the topic model to reconstruct only the words that appear in a document, we ask a language model what the document is about and use its confidence over many possible theme words as a richer training signal. This lets the topic model learn from words that may not appear in the document but are still semantically relevant. Our experiments show that this approach produces clearer topics, better document-topic assignments, and stronger retrieval of semantically similar documents across three datasets. Importantly, DSL-Topic keeps the interpretability and probabilistic structure of traditional topic models while using language models to provide deeper semantic guidance.