Repertoires, Not Scores: Instability as Signal in Cultural Evaluation of LLMs

Cultural evaluation of large language models typically scores agreement between model outputs and national-average responses on survey instruments like Hofstede's dimensions or the World Values Survey. This rests on a values paradigm that contemporary sociology has substantially revised: cultural competence is now widely viewed as the contextual deployment of repertoires rather than the expression of stable values. We argue that observed LLM "value instability" across paraphrases and framings is, under this view, what cultural competence should look like — and that current benchmarks cannot distinguish competent contextual adaptation from random prompt sensitivity. We propose two evaluation criteria: persistence, the ratio of preference shift across substantive contexts to shift across incidental perturbations, and repertoire alignment, the divergence between model and human reference distributions on matched item-context cells. We formalize both, work through an example on cross-cultural pragmatics of indirect requests, and audit existing benchmarks against the criteria. Our framework recasts what cultural evaluation should measure and predicts that current cultural alignment methods systematically reduce the contextual variation they should preserve.