Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews

How can we distinguish whether a peer review was written by a human or generated by an AI model? We argue that, in this setting, authorship should not be attributed solely from the textual features of a review, but also from the ideas, judgments, and claims it expresses. To this end, we propose Sem-Detect, an authorship detection method for peer reviews that operationalizes this principle by combining textual features with claim-level semantic analysis. Sem-Detect compares a target review against multiple AI-generated reviews of the same paper, leveraging the observation that different AI models tend to converge on similar points, while human reviewers introduce more unique and diverse ones. As a result, Sem-Detect is able to distinguish fully AI-generated reviews from authentic human-written ones, including those that have been refined using an LLM but still reflect human judgment. Across a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest prior detector by 36.5% in TPR@1% FPR in the binary setting. More importantly, in the three-class scenario, we empirically show that LLM refinement preserves the semantic signals of human reviews, which remain distinct from the patterns exhibited by fully AI-generated text; as a result, fewer than 3.5% of LLM-refined human reviews are misclassified as AI-generated.

Peer review is foundational to science, but as Large Language Models become widely available, there is growing concern that some reviews may no longer reflect genuine human judgment. The challenge is nuanced. A reviewer who drafts an assessment and uses an LLM to polish it is fundamentally different from one who prompts an LLM to write the entire review. ICML 2026 reflects this: it permits LLM editing but bars fully AI-generated reviews. What matters is not whether an LLM touched the text, but whether the ideas came from a human. Motivated by this, we developed Sem-Detect: an approach that focuses on the content of a review rather than only its writing style. Our insight: different AI models reviewing the same paper tend to converge on similar claims, while human reviewers bring more unique judgments. By comparing a target review against multiple AI-generated reviews at the claim level, Sem-Detect distinguishes fully AI-generated reviews from authentic human ones, including those polished by an LLM. Evaluated on over 20,000 ICLR and NeurIPS reviews, Sem-Detect improves over the strongest prior detector by 25.5%, while misclassifying fewer than 3.5% of LLM-refined human reviews. We believe it offers a path toward preserving peer review integrity in the LLM era.