Measuring Meta-Cultural Competency: A Spectral Framework for LLM Knowledge Structures

Most existing cultural evaluation frameworks for large language models (LLMs) focus on matching model outputs to ground-truth answers, primarily measuring factual cultural awareness. This overlooks whether models internalize broader cultural structure and pluralism. We introduce a spectral-analysis-based framework that captures large-scale macrostructural patterns in models' cultural knowledge and evaluate eight LLMs across nine cultural domains spanning all five of Newmark's cultural dimensions and 170 countries. Comparing with human data, we find that instruction-tuned models align more closely with human cultural structure than older models, while increased model size does not consistently improve performance. Finally, simulation-based experiments show that our proposed spectral metric better predicts a model's ability to serve users from unfamiliar cultural backgrounds than existing ones.

Assume you want to assess the health of a banyan tree. You could inspect its individual leaves. However, checking only a few leaves would give a partial picture, while checking every leaf would be impossible. Alternatively, you could examine how strong and deep the trunk and roots are: the very structures that support all the leaves. Testing whether a language model understands culture is similar. Existing evaluations often query models for factual correctness, such as whether they know the currencies of different countries or what people eat in different regions, which is like checking the leaves. Although such facts are useful, there can be countless questions to test exhaustively, and even if we ask many of them, they may still not reveal whether the model has a deeper understanding. Instead, our work proposes a way to examine the "trunk and roots" of cultural knowledge. Rather than asking only whether a model knows isolated facts, we study whether it captures the broader structures that connect those facts. For example, a model should not only know individual national dishes, but also recognize that countries often form meaningful clusters because they share ingredients, cooking styles, and eating habits, while still retaining local differences. We evaluate whether models capture such real-world patterns of similarity and variation across 170 countries and nine cultural domains, including food, religion, language, holidays, and currency. We find that newer instruction-tuned models capture these human-like patterns better than older models, but larger models are not always better. Our measure also better predicts whether a model can help users from unfamiliar cultural backgrounds, such as in recipe recommendation.