Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning

Decoding-based regression, which reformulates regression as a sequence generation task, has emerged as a promising paradigm of applying large language models for numerical prediction. However, its progress is hindered by the misalignment between discrete token-level objectives (e.g., cross-entropy) and continuous numerical values. Existing approaches relying on token-level constraints often fail to capture the global magnitude of the target value, limiting their precision and generalization. In this paper, we propose to unlock the potential of decoding-based regression via reinforcement learning. We formulate the generation process as a Markov decision process, utilizing sequence-level rewards to enforce global numerical coherence.Under this framework, we present GenRe$^2$, which combines policy gradient methods and on-policy distillation to provide dense expert supervision while preserving error magnitudes, thereby resolving the temporal credit assignment challenge. Extensive experiments across tabular regression, code metric prediction and generative reward modeling demonstrate that GenRe$^2$ consistently outperforms traditional baselines, establishing a robust paradigm for general-purpose numerical prediction.

Many real-world problems require predicting a number, such as a system’s latency, a program’s performance, or the quality score of a model response. Recent work tries to let language models solve such problems by generating the answer as a sequence of tokens, just like writing text. However, standard training methods usually judge each token separately, which is not ideal for numbers: a mistake in an early digit can make the final value completely wrong, while a mistake in a later digit may be much less serious. This paper studies how to train such number-generating models in a way that directly cares about the final numerical error. We propose GenRe$^2$, which uses reinforcement learning to reward the model based on the accuracy of the complete generated number. To make this learning more stable and informative, GenRe$^2$ further uses on-policy distillation: the model learns corrective feedback on its own generated outputs, instead of only imitating fixed ground-truth answers. Experiments on tabular regression, code metric regression, and reward modeling show that GenRe$^2$ makes numerical predictions more accurate and robust than previous token-level training methods. This work helps language models become better general-purpose tools for predicting continuous values across diverse data types.