AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems

While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis (CFA) and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance and establish: (1) structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; (2) evidence of local dependence among leaderboard items, undermining uses of benchmarks as measurement instruments under current scoring systems; (3) contributor metadata explains more rank-relevant variance ($\approx9\%$) than architecture or deployment categories in this context; (4) a manifest-score ``scaling law'' slope has low reliability ($R_{\beta}=0.53$); by contrast, the latent general-factor size slope is highly stable across ecosystem controls ($R_g=0.97$). We are able to provide unique insights into benchmark dynamics, such as which benchmarks are a function of LLM size and which can be oppositely impacted by post-training practices. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.

Problem: We often judge AI models by their scores on competitive leaderboards, but these rankings can be deceptive. A high score doesn’t always mean an AI is genuinely smarter; the results can be skewed by the model’s size, its specific training, or even quirks in the test itself. This makes it difficult to separate true progress from noise, leading researchers to chase misleading metrics. Solution: Our paper builds an “AI cartography” map that separates broad ability from benchmark-specific quirks and ecosystem effects (such as model size and common training choices). Using measurement tools designed to tease signal from noise, we estimate how much of each benchmark reflects general capability versus idiosyncrasies. This allows us to map the hidden landscape of AI capabilities, disentangling a model’s general intelligence from other influences so tiny leaderboard gaps aren’t over-interpreted. Impact: This new map provides a much clearer picture of AI progress. The map suggests knowledge-heavy tests keep improving strongly with scale, while gains on other reasoning-focused tests show much smaller returns once you control for general ability. Overall, the work helps the community make fairer, more meaningful comparisons—and reduces the risk of chasing misleading leaderboard wins instead of real advances.