LLM-PriorCB: Textual Contextual Bandits with LLM-Induced Priors

Many real-world decision-making problems, including recommendation, tool selection, and model routing, require choosing among actions described in natural language while observing feedback only for the selected action. Contextual bandits provide a principled framework for exploration under such partial feedback, but standard implementations typically start from uninformative reward-model priors and can adapt slowly in cold-start or evolving-action regimes. Large language models (LLMs) offer useful semantic knowledge from reward-free text, yet using an LLM directly as the online decision maker does not by itself provide calibrated uncertainty or principled exploration. We study textual contextual bandits, where both contexts and actions are represented by text, action descriptions are available without reward labels, and the action set may evolve over time. We propose LLM-PriorCB, a two-stage framework that separates semantic prior construction from online learning. In the offline stage, an LLM estimates rewards for reward-free context-action text pairs, and these estimates are distilled into action-specific prior parameters. In the online stage, LLM-PriorCB applies a disjoint linear UCB rule initialized by these action-specific priors, and updates only the selected action’s parameter using observed rewards, without further LLM queries. We derive prior-dependent regret guarantees for the resulting disjoint linear bandit, showing how prior misspecification affects the confidence radius while preserving no-regret learning. Experiments on MovieLens and GAIA show that LLM-PriorCB reduces cumulative regret relative to LinUCB, CBLI, and prompted LLM policy baselines.