Language Pretraining Gives Structured Forecasters a Sequential Prior

Structured-data foundation models now target transfer across time series, tabular data, text, and other sequential domains, but it remains unclear what language pretraining contributes to forecasting beyond a generic transformer architecture. We study this question by adapting Qwen3-0.6B to probabilistic time-series forecasting on GiftEval and comparing language-pretrained models with identical randomly initialized models under full and parameter-efficient finetuning. Language pretraining gives a large early optimization advantage, especially in limited-adaptation regimes: LoRA attention updates recover most of the effective transfer benefit of full finetuning. Frozen-state probes, retrieval forecasts, gradient coherence, and effective-rank dynamics indicate that language models already contain reusable sequential structure before time-series supervision. These results frame language-to-time-series transfer as efficient low-rank specialization of a pretrained sequential inductive bias, with implications for future structured-data foundation models.