Learning More from Less: Unlocking Internal Representations for Benchmark Compression

The prohibitive cost of evaluating Large Language Models (LLMs) necessitates efficient alternatives to full-scale benchmarking. Prevalent approaches address this by identifying a small coreset of items to approximate full-benchmark performance. However, existing methods must estimate a reliable item profile from response patterns across many source models, which becomes statistically unstable when the source pool is small. This dependency is particularly limiting for newly released benchmarks with minimal historical evaluation data. We argue that discrete correctness labels are a lossy view of the model's decision process and fail to capture information encoded in hidden states. To address this, we introduce \textsc{RepCore}, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets. Using these subsets for performance extrapolation, \textsc{RepCore} achieves precise estimation accuracy with as few as ten source models. Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy. Spectral analysis further indicates that the aligned representations contain separable components reflecting broad response tendencies and task-specific reasoning patterns.

Evaluating large AI models often means testing them on thousands of questions, which can be slow and expensive. Researchers therefore look for much smaller question sets that still show how models would perform on the full test. This is especially hard for newly released tests, where only a few models have been evaluated and simple right-or-wrong answers reveal little about what makes each question different. We introduce RepCore, a method for choosing representative test questions using information produced inside models while they answer. Rather than looking only at final answers, RepCore maps this internal information from different models onto a common map, where questions that challenge models in similar ways are placed close together. It then selects a compact set of representative questions and uses them to estimate performance on the full test. Across text and image-based tests, RepCore gives more reliable estimates of model scores and rankings than methods based only on final answers. This can make AI evaluation cheaper, faster, and more practical when many models need to be compared.