Evaluating Pluralism in LLMs through Latent Perspectives

Pluralistic representation and generation in LLMs is becoming increasingly relevant because of the importance of showcasing the diversity of opinion. Models are known to reduce the diversity of training data and to exhibit homogeneity when generating. However, this issue has been demonstrated primarily on multiple-choice questionnaires or using aggregated, high-level characteristics for free-form text. Such approaches do not adequately capture the variety of human perspectives expressed in opinionated text, nor do they identify the missing aspects of human text driving the pluralistic gap in LLM-generated text. In this paper, we aim to analyze model pluralism by extracting and comparing latent perspectives from human and LLM-generated text. We propose a two-tiered framework which identifies which aspects of perspectives are underrepresented in LLM generation. We evaluate an instance of our framework on on highly opinionated data from book reviews and find that while LLMs reduce pluralism in human text across levels of abstraction, simple prompting techniques such as persona prompting can alleviate the pluralistic gap in subjective aspects.