Omitted Variable Bias in Language Models Under Distribution Shift

Despite their impressive performance on a wide variety of tasks, modern language models remain susceptible to distribution shifts, exhibiting brittle behavior when evaluated on data that differs in distribution from their training data. In this paper, we describe how distribution shifts in language models can be separated into observable and unobservable components, and we discuss how established approaches for dealing with distribution shift address only the former. Importantly, we identify that the resulting omitted variable bias from unobserved variables can compromise both evaluation and optimization in language models. To address this challenge, we introduce a framework that maps the strength of the omitted variables to bounds on the worst-case generalization performance of language models under distribution shift. In empirical experiments, we show that using these bounds directly in language model evaluation and optimization provides more principled measures of out-of-distribution performance, improves true out-of-distribution performance relative to standard distribution shift adjustment methods, and further enables inference about the strength of the omitted variables when target distribution labels are available.

Language models often struggle or behave unpredictably when they are tested on data that differs from what they were originally trained on—a challenge known as distribution shift. Current methods try to correct this by adjusting for the differences in the data that we can easily observe and measure. However, they do not account for differences resulting from hidden factors, including factors that are observable to humans but inherently unobservable to models. This creates a critical blind spot known as omitted variable bias. We find that failure to account for these hidden factors can make models appear more capable than they actually are in evaluations, leading to poorly trained systems. To address this, we introduce a new framework that computes a "worst-case scenario" bound based on how strong these unseen variables might be. We find that explicitly evaluating models for their worst-case scenario provides a more realistic measure of how well they may actually perform in the real world, and optimizing models for their worst-case scenario helps them perform better when deployed on new, unfamiliar data.