REVIEW
Auditing AI peer reviewers: a benchmark on real scientific papers
Computation and Language · Submitted 12 Nov 2026 · 61 references
Pith's verdict: The paper's central result—model choice dominates detection by ~50 pp—is robust; the SKEPTHICAL advantage and false-positive rankings rest on an LLM adjudication instrument validated on one of thirty papers.
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
Given the same prompt, three frontier LLMs differ by up to about 50 percentage points in detecting injected paper errors, and enforced citation retrieval only helps where the model does not already know the literature.
The reading
The main result is that the choice of model dominates everything else. Given the same one-sentence prompt, three frontier models detect between 12 and 60 percent of errors on the astrophysics corpus, a gap of roughly 50 percentage points. A longer structured prompt narrows that gap by lifting the weaker models. Tools help only where they supply something the model lacks: on astrophysics, running code catches numerical and mathematical errors, and forcing the reviewer to download and read the cited papers gives the new system, Skepthical, a 20-point lead on citation errors. On machine learning, where the models already know the literature, forced retrieval adds nothing.
The honest caveat sits in the measurement itself. Whether a system detected an error is decided mostly by a panel of three LLM judges, and a human checked only one paper of thirty. The false-positive panel is noisier still, with low agreement between judges. So the relative ordering of systems is the most robust part of the paper, while the absolute rates should be read as approximate.
Extended reading notes
Core claim
The load-bearing assertion is the benchmark result: "given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora," and, on astrophysics, "enforced citation retrieval gives SKEPTHICAL the highest citation error detection rate of all systems by a large margin" (46.7 percent versus at most 26.7 percent). The paper also claims the first controlled comparison of AI reviewers on a real non-ML corpus combining severity grading, false-positive control, mechanical verifiers, and cross-family adjudication. If correct, these claims establish that reviewer detection is dominated by model choice, and that enforced verification steps help only when they add information the model lacks.
Load-bearing premise
The measured detection and false-positive rates treat the LLM adjudication panels' verdicts as ground truth. Detection matching is done by a three-judge LLM panel (Section 2.4) and false-positive classification by another LLM pipeline, with a human check covering one of thirty papers (132 of 4,084 adjudications, Appendix F) and no human check at all on the false-positive pipeline, whose pairwise judge correlation is only r=0.32 to 0.58 (Appendix D). If the panels systematically over- or under-credit detections in a way that correlates with a system's prose style or error-type focus, every ranking in Table 2 and Figure 2 shifts. The paper's own one-paper check found all five disagreements were the panel crediting detections the human rejected, inflating the detection rate on that paper by 3.7 points.
Original abstract
LLM-based peer review is now used at production scale, but its evaluation remains limited. We introduce a benchmark that injects objective, localisable errors of four types (general issues, mathematics, numerics, citations) at three severity levels into real papers and measures each reviewer's detection rate by error type and severity, together with its false-positive rate on clean copies. We also present Skepthical, a multi-agent reviewer that verifies citation-backed claims by retrieving the cited papers, mathematics with computer algebra, and numerical claims with executed code. We apply the benchmark to two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning (ML), injecting a total of 357 errors, and evaluate twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available reviewers, and Skepthical. The choice of model has the largest impact on detection rate: given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora. Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves on a single Claude call in mathematics and numerics detection, and enforced citation retrieval gives Skepthical the highest citation error detection rate of all systems by a large margin. This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical's citation detection rate is statistically indistinguishable from that of the best single LLM. False-positive rates are below 10% for most systems, but vary with prompt, model, and scaffolding, with mathematics flags the main source of false positives. The evaluation pipeline, the injected errors, and the adjudication logs are released as a benchmark.
Figures
Figures reproduced from arXiv:2611.17052 by the authors.
References
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.