Escaping Mode Collapse in LLM Generation via Geometric Regulation

Mode collapse is a persistent challenge in generative modeling and manifests in autoregressive text generation as behaviors ranging from explicit looping to gradual loss of diversity and premature trajectory convergence. We take a dynamical-systems view and reinterpret mode collapse as reduced state-space accessibility caused by geometric collapse: during generation, the model's internal trajectory becomes confined to a low-dimensional region of its representation space. This implies mode collapse is not purely a token-level phenomenon and cannot be reliably mitigated by symbolic constraints or probability-only decoding heuristics. Guided by this perspective, we propose Reinforced Mode Regulation (RMR), a lightweight, online state-space intervention that regulates dominant self-reinforcing directions in the Transformer value cache (implemented as low-rank damping). Across multiple large language models, RMR substantially reduces mode collapse and enables stable, high-quality generation at extremely low entropy rates (down to 0.8 nats/step), whereas standard decoding typically collapses near 2.0 nats/step.

Large language models can write fluent text, but during long generation they can sometimes get stuck: they repeat themselves, become bland, or fall into narrow patterns that are hard to escape. Most existing fixes try to adjust which word the model picks next, but this only treats the visible symptom. We study what happens inside the model while it writes. Our finding is that collapse is not just about repeated words; it is also about the model’s internal activity becoming trapped in a much smaller range of possibilities. In other words, the model’s “path of thought” loses room to move. Based on this view, we propose Reinforced Mode Regulation, a lightweight method used only at generation time. It gently weakens internal directions that keep reinforcing themselves too strongly, without retraining the model. Across several large language models, this makes long-form generation much less likely to collapse, especially when the model is asked to write in a focused, low-randomness way. This work suggests a new way to make language models more reliable: control the internal dynamics that produce text, rather than only editing the next-word probabilities.