Out-of-Distribution Generalization of Risk Aversion in Language Models

Training AIs to be risk-averse in resources could offer a second line of defense in the event that AIs turn out misaligned. Misaligned but risk-averse AIs would tend to prefer low-risk, low-reward strategies like cooperation over high-risk, high-reward strategies like rebellion, limiting the downsides of any misalignment. But to train AIs to be risk-averse in an affordable way, we will have to train them on choices between low-stakes gambles. And to keep us safe, the AI's learned risk aversion must generalize to astronomically-high-stakes gambles. Will it? To investigate this question, we use a variety of methods to make Qwen3-8B choose risk-aversely when the stakes are low. We find that these methods can induce substantial risk aversion when the stakes are astronomically high: our models' learned risk aversion generalizes at least partially across 98 orders of magnitude. From a baseline 2.45\% rate of choosing a safe `Cooperate' option, we see rates around 65\% (SFT), 67\% (tie training), and 46\% (DPO). Our fine-tuned reward model reliably scores risk-averse reasoning above risk-neutral or excessively risk-averse alternatives (100\% pairwise accuracy). We replicate these effects at different scales (Qwen3-1.7B and Qwen3-14B) and across model families (Gemma-3-12B-IT and Llama-3.1-8B-Instruct). Activation steering yields mixed results, performing poorly on Qwen and Gemma but better on Llama. We find that models' learned risk aversion does not significantly hurt capabilities on MMLU-Redux, and that it partially generalizes across distinct goods (GPU-hours, lives saved, and money for a user). Overall, our results suggest that risk aversion learned at low stakes can generalize out of distribution to astronomically high stakes.