Text Generation as Continuous Latent Dynamics via Reinforcement Learning

We propose to model text generation as a continuous-time latent dynamical process, where token generation is formulated as a Markov decision process whose internal state evolves via a neural ODE. This formulation bridges discrete token sequences and continuous semantic evolution, providing a theoretically grounded framework for text generation with continuous-time latent states. The framework is optimized via reinforcement learning, maximizing a composite objective that integrates cumulative rewards with a Kullback–Leibler divergence regularization term from a pre-trained language model. Both theoretical and empirical results demonstrate that our Continuous-Time Latent Language Model (CT-LLM) achieves superior effectiveness and efficiency in text generation, establishing a new paradigm for continuous-time language modeling.

Large language models struggle to maintain coherent meaning over long text generations because standard methods treat each word as an isolated jump. We introduce CT-LLM, which models text generation as a smooth, continuous flow through a hidden ``meaning'' space using neural ordinary differential equations. A tailored reinforcement learning algorithm jointly optimizes which tokens to output and how the internal meaning evolves between tokens. Our method generates more natural, coherent text than existing approaches, with provable convergence guarantees. This can improve conversational agents, writing assistants, and any application requiring machines to produce human-like text with consistent long-range meaning.