Timezone: »

 
Poster
Whose Opinions Do Language Models Reflect?
Shibani Santurkar · Esin Durmus · Faisal Ladhak · Cinoo Lee · Percy Liang · Tatsunori Hashimoto

Thu Jul 27 01:30 PM -- 03:00 PM (PDT) @ Exhibit Hall 1 #510
in Poster Session 5 »

Language models (LMs) are increasingly being used in open-ended contexts, where the opinions they reflect in response to subjective queries can have a profound impact, both on user satisfaction, and shaping the views of society at large. We put forth a quantitative framework to investigate the opinions reflected by LMs -- by leveraging high-quality public opinion polls. Using this framework, we create OpinionQA, a dataset for evaluating the alignment of LM opinions with those of 60 US demographic groups over topics ranging from abortion to automation. Across topics, we find substantial misalignment between the views reflected by current LMs and those of US demographic groups: on par with the Democrat-Republican divide on climate change. Notably, this misalignment persists even after explicitly steering the LMs towards particular groups. Our analysis not only confirms prior observations about the left-leaning tendencies of some human feedback-tuned LMs, but also surfaces groups whose opinions are poorly reflected by current LMs (e.g., 65+ and widowed individuals).

[ Poster] 

Author Information

Shibani Santurkar (OpenAI)
Esin Durmus (Stanford University)
Faisal Ladhak (Columbia University)
Cinoo Lee (Stanford University)
Percy Liang (Stanford University)
Tatsunori Hashimoto (Stanford)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors