Large Language Models Explore by Latent Distilling

Generating diverse responses is crucial for test-time scaling of large language models (LLMs), yet standard stochastic sampling mostly yields surface-level lexical variation, limiting semantic exploration. In this paper, we propose Exploratory Sampling (ESamp), a decoding approach that explicitly encourages semantic diversity during generation. ESamp is motivated by the well-known observation that neural networks tend to make lower-error predictions on inputs similar to those encountered before, and incur higher prediction error on novel ones. Building on this property, we train a lightweight Distiller at test time to predict deep-layer hidden representations of the LLM from its shallow-layer representations to model the LLM’s depth-wise representation transitions. During decoding, the Distiller continuously adapts to the mappings induced by the current generation context. ESamp uses the prediction error as a novelty signal to reweight candidate token extensions conditioned on the current prefix, thereby biasing decoding toward less-explored semantic patterns. ESamp is implemented with an asynchronous training--inference pipeline, with less than 5\% worst case overhead (1.2\% in the optimized release). Empirical results show that ESamp significantly boosts the Pass@k efficiency of reasoning models, showing superior or comparable performance to strong stochastic and heuristic baselines. Notably, ESamp achieves robust generalization across mathematics, science, and code generation benchmarks and breaks the trade-off between diversity and coherence in creative writing. Our code has released at: https://github.com/LinesHogan/tLLM.

Large Language Models (LLMs), such as ChatGPT, are widely used to generate text responses. However, they often lack diversity: when asked to produce multiple responses to the same question, they tend to say essentially the same thing in slightly different words. Yet in many scenarios, we want genuinely diverse responses — not just paraphrases of one another. Achieving this is deceptively hard. One must simultaneously solve several daunting challenges: How do you tell that two sentences mean the same thing when they use completely different words? How do you do so in real time, without the costly step of running a separate model to compare meanings? How do you keep a running count of "how often has this idea appeared before"? And how do you turn that count into a signal that actually steers generation? Our key insight is that a single, surprisingly simple design — a small neural network that learns to transform the LLM's own internal states on the fly — resolves all of these challenges at once. Because we operate directly inside the LLM's "thought process," semantic similarity is captured naturally (solving the first problem), no external model or extra inference pass is needed (solving the second), the network inherently accumulates a memory of what has been said (solving the third), and its output directly modulates the LLM's next-word predictions (solving the fourth) — all without degrading generation quality. A key strength of our work is the tight co-design between algorithm and system, enabling our method to run at industrial-grade speed. This makes it potentially the first semantically diverse generation method that is practical for real-world deployment — powering use cases such as AI agents that explore multiple strategies in parallel, or chatbots that offer creative, multi-perspective answers.