Large Language Models Develop Novel Social Biases Through Adaptive Exploration

As large language models (LLMs) are adopted into frameworks that grant them the capacity to make real decisions, it is increasingly important to ensure that they are unbiased. In this paper, we argue that the predominant approach of simply removing existing biases from models is not enough. Using a paradigm from the psychology literature, we demonstrate that LLMs can spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist. These biases result in highly stratified task allocations, which are less fair than assignments by human participants and are exacerbated by newer and larger models. In humans, emergent biases like these have been shown to result from exploration-exploitation trade-offs, where the decision-maker explores too little, allowing early observations to strongly influence impressions about entire demographic groups. To alleviate this effect, we examine a series of interventions targeting model inputs, problem structure, and explicit steering. We find that explicitly incentivizing exploration most robustly reduces stratification, highlighting the need for better multifaceted objectives to mitigate bias. These results reveal that LLMs are not merely passive mirrors of human social biases, but can actively create new ones from experience, raising urgent questions about how these systems will shape societies over time.

Bias in large language models (LLMs) is becoming an even more urgent problem as these models are put into agents that make decisions. Our paper shows that the current way that we focus on removing biases from models is not enough. We do this by showing how LLMs can develop new previously unseen biases for demographic groups, even when there are no differences between groups in the first place! The way LLMs do that is through a multi-step interaction with the world, where they make a decision, learn about the result, and use that result to change their beliefs. However, when the result comes from a random process, these models try so hard to learn from it that they end up developing incorrect beliefs, and this only ends up strengthening because it affects the LLM's future decisions. The end result is that demographics are very unequally treated, even worse than when humans are put in the same decision making situation. Our paper then tries to fix this by exploring a bunch of options, from changing the model's inputs to telling it to be more fair. The only change that actually worked is one where we change the goal that we give the model entirely, forcing it to think about equality by giving it a reward. Our paper shows how important it is to give AI systems the right instructions, so that they can follow the goals that we give them without harming others. It also raises awareness for this new type of bias, which can have a large negative impact if we don't pay attention to it.