Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation

Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce \emph{semantic stratification}, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes. Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics.

Modern AI systems often rely on retrieval systems to gather relevant information before generating answers. These systems are commonly evaluated using average benchmark scores, such as Recall@k or nDCG, which summarize performance across an entire dataset. While useful, these averages can hide important weaknesses: a retrieval system may perform very well overall while consistently failing on certain kinds of information. In this work, we introduce a new evaluation framework called semantic stratification. Instead of treating all benchmark queries as a single group, we organize datasets into semantically coherent regions based on the structure of the underlying document corpus. We then measure retrieval quality separately within each semantic region. This approach reveals failure modes and uneven system behavior that traditional aggregate metrics often miss. For example, we show that retrievers with similar overall benchmark scores can behave very differently across different semantic areas of a corpus. Some regions may be consistently difficult for all systems, while others expose weaknesses in particular retrieval methods. By focusing on coverage rather than averages alone, our framework provides a more trustworthy and interpretable view of retrieval system performance. We believe this can help researchers and practitioners build retrieval systems that are more robust, reliable, and better aligned with real-world enterprise applications.