On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility

Large Language Models (LLMs) excel at long-context understanding but exhibit significant limitations in long-form generation. Existing studies primarily focus on single-generation quality, generally overlooking the volatility of the output (i.e., the inconsistency in length and content across multiple generations). This volatility not only leads to significant computational costs but also severely impacts the models' reliable application. To address this gap, our work unfolds in three stages: benchmarking, probing, and mitigation. We first propose the VOlatility in Long-form Text Benchmark (VOLTBench), a novel heterogeneous-task benchmark designed to systematically quantify the length volatility of long-form generation. Subsequently, by analyzing attention traces, we conduct an in-depth probe to identify several common internal patterns that cause this volatility. Finally, to mitigate long-form output volatility, we propose SELB (Structural Enforcement via Logits Boosting), a lightweight decoding-stage optimization strategy, designed to significantly enhance both the length accuracy and stability of long-form generation without additional training. Extensive experiments on VOLTBench provide the first systematic confirmation of severe long-form output instability in mainstream models and validate that our proposed method successfully improves the mean output length of the base model by 148% and reduces the length volatility by 69%, while maintaining high generation quality.

Large language models can now read very long documents, but they are still unreliable when asked to write long outputs. For example, when asked to write a long story, a many-section report, or a large code file, the same model may produce very different lengths across repeated runs. It may stop too early, skip sections, or fill the output with repetitive text. This makes long-form generation hard to trust and can waste computation. In this paper, we study this problem systematically. We introduce VOLTBench, a benchmark that tests whether models can generate long outputs with stable length and structure across different tasks, languages, and instruction types. We also analyze the model’s internal behavior and find that failures often happen when the model loses track of the original instructions during generation. Based on this observation, we propose GLoBo, a lightweight method that guides the model during decoding without additional training. Our results show that GLoBo makes long outputs much more stable: it increases the average output length by 148\% and reduces length variation by 69\%, while preserving generation quality. This can make language models more dependable for long writing, document generation, and code generation tasks.