Detecting Errors in AI-Generated Annotations: When and Why Semantic Neighbors Help

Large language models (LLMs) and vision-language models (VLMs) have emerged as efficient annotators for tasks such as generation and classification. While these models offer significant cost and speed advantages over human annotation, a critical challenge remains: existing self-evaluation methods, such as LLM-as-judge, often lack reliable reference-based calibration for error detection. We address this limitation by introducing **SAGE** (**S**emantic-**A**nchored Jud**G**m**E**nt), a method that leverages semantically similar samples retrieved via $k$-nearest-neighbor as references for annotation verification. We provide a theoretical framework that derives a closed-form expression for the error detection AUROC, which can be decomposed into three factors: intrinsic separability, reference-induced mean shift, and noise reduction through averaging. This decomposition reveals *when* semantic neighbors help (when references are both semantically matched and correct) and *why* (by providing reference-based calibration that raises scores for correct annotations and lowers scores for incorrect ones). Experiments on LLM generation, VLM captioning, and classification tasks validate our theoretical framework: SAGE improves error detection when semantic neighbors provide reliable reference-based calibration, and our decomposition offers insights into when direct scoring or alternative strategies may be preferred. Our code is available at https://github.com/dina-1205/SAGE.

When we use large language models (LLMs) to label vast amounts of data (sorting reviews, captioning photos, classifying documents), they work far faster than humans can. But how do we catch the mistakes these AI labelers inevitably make? The standard trick of asking the AI to grade itself ('LLM-as-judge') often fails: the model tends to be confidently wrong exactly where errors hide. Our paper introduces SAGE, which shows the AI judge a few semantically similar examples before scoring each new label, like a teacher checking answers against model solutions for similar problems. The familiar context helps the AI catch its mistakes far more reliably. We also prove mathematically why this works: similar examples raise scores for correct answers and lower them for wrong ones. We tested SAGE on text generation, image captioning, and classification tasks; it consistently improves error detection. This matters for anyone using AI to label data at scale (researchers cleaning training datasets, companies moderating content, organizations validating AI-generated reports): catching errors becomes much cheaper, letting people fix mistakes before they propagate downstream.