{"id":"6b8e695f-18ba-4ec1-96ae-dd994a723380","arxiv_id":"2608.03581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-generated reviews on real pre-revision submissions are longer, more positive, and less score-calibrated than human reviews, and aggregate quality scores alone overestimate their quality.","lead":"This paper maps how 111 AI/ML conferences and medical journals regulate reviewers' use of AI, then tests three language models as reviewers on original submissions from ICLR 2026 and Nature Communications. It finds AI reviews are fluent and receive high automated quality scores, yet are overly positive, poorly calibrated, and unevenly grounded, so single quality scores overstate their usefulness.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Overlap metric's LLM-based concern extraction/matching was iteratively tuned on the same corpus and lacks human gold-standard validation; coverage numbers (e.g., GPT 46% Essential) and the 'breadth not judgment' conclusion may be matcher artifacts.","rationale":"The reader's weakest assumption pins the conditional verdict on the overlap metric, and I agree that is the single most load-bearing concern. The central claim about systematic weaknesses is robust: score inflation is measured directly against human scores (Section 5.2), generic criticism and evidence-grounding deficits are demonstrated with phrase counts and reference counts (Sections 5.6, I.2-I.4), and Llama's degeneration is shown with examples (I.3). The LLM-as-a-Judge self-preference is real but explicitly acknowledged in Limitations and not load-bearing for the core claim; it actually illustrates the paper's point that aggregate scores alone overestimate. The overlap metric, by contrast, is the only evidence for the paper's distinctive positive conclusion ('breadth rather than superior judgment', Section 6) and for the seriousness-stratified coverage table (Table 3). The pipeline was tuned on the same corpus without a human gold standard, and the provided sensitivity check cannot rule out extraction bias. A human-annotated gold-standard test would settle whether the coverage numbers are genuine. The verdict should remain CONDITIONAL: the paper is valuable and the main claims survive, but the overlap-based sub-claims need external validation before full acceptance.","tokens_in":23303,"tokens_out":9338,"duration_ms":116733,"concrete_test":"Build a gold standard on a random sample of ~30 ICLR papers: have two expert annotators (blind to source) (a) judge whether each GPT-5.5-extracted concern is genuinely present in the review and assign seriousness, and (b) label which AI/human concern pairs express the same underlying point. Recompute raw, length-normalized, and Essential-only coverage using only gold-validated concerns/matches. If GPT's Essential coverage drops from 0.46 toward the human baseline or the AI-vs-human ordering reverses by more than ~10 percentage points, the overlap-based conclusions should be revised. As a cheaper check, rerun the full pipeline with a model not used during prompt development (e.g., a different LLM as extractor/matcher) and compare coverage distributions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline coverage results rest on a GPT-5.5 pipeline that (1) decomposes reviews into concerns and labels seriousness, and (2) matches AI concerns to human concerns using the paper text. Appendix E.1 states that the prompts were 'iteratively refined until the inspected outputs aligned with our judgments' on the same data; no held-out validation, inter-annotator agreement, or human gold-standard is reported. The only sensitivity check (Appendix E.5) swaps the matcher for sentence embeddings but reuses the same extracted concerns, so it cannot detect extraction bias; and while aggregate coverage is similar (0.36 vs 0.37), per-paper correlation is only 0.50 and pair-level agreement 0.39. The specific matches feeding the seriousness-stratified numbers (GPT covers 46% of Essential human concerns vs 12% Minor) are therefore matcher-dependent. If GPT-5.5 preferentially recognizes AI-phrased critiques, or if extraction paraphrases human critiques into LLM-like language, the AI-vs-human coverage gap and the 'breadth rather than judgment' conclusion (Section 6) could be inflated. This is load-bearing because the paper's distinctive positive claim—that LLMs add coverage especially for Essential concerns—and the recommendation to use AI as a second-pass assistant rest on this metric. The other core claims (score inflation, generic criticism, evidence-grounding gaps, generation degeneracy) are supported by direct human-auditable observations and would survive even if this concern lands.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies AI-assisted peer review from two angles. First, it surveys reviewer-facing AI policies across 111 venues (63 AI/NLP conferences and 48 medical journals), finding that AI/NLP venues are predominantly permissive with 'Partial' policies while medical journals mostly prohibit AI use. Second, it constructs a dataset of original manuscripts and corresponding human and machine-generated reviews for ICLR 2026 (50 papers) and Nature Communications (31 papers), using GPT-5, Llama-4 Scout, and Qwen3-VL to generate reviews. The evaluation combines score alignment, an LLM-based overlap metric, GranuScore, and LLM-as-a-Judge. The reported findings are that LLM reviews are longer, more positive, and less calibrated than human reviews; GPT shows moderate correlation with human scores (r=0.62) while Llama and Qwen show none; AI reviews achieve high aggregate judge scores but exhibit generic criticism and uneven evidence grounding; and AI reviews cover a nontrivial fraction of human concerns, especially 'Essential' ones, which the authors interpret as breadth rather than superior judgment.","tokens_in":23605,"tokens_out":3304,"duration_ms":39260,"significance":"If the results hold, the paper makes a useful contribution to the current debate on AI-assisted peer review. The venue-level policy survey is novel and valuable, and the release of a dataset with original submission versions and human/AI review pairs addresses a real gap in the literature. The multi-metric evaluation, including direct qualitative evidence of positive bias, generic critique, and generation degeneracy, is a strength: these findings are human-auditable and do not depend on the more fragile parts of the pipeline. The paper is also commendable for making code and data available and for transparently discussing limitations. However, two load-bearing evaluation components—the LLM-based overlap metric and the GPT-5-as-judge—have serious validity concerns that directly affect the headline conclusions about coverage and about aggregate scores overestimating review quality. These concerns need to be addressed before the paper's central claims can be accepted as stated.","major_comments":[{"comment":"The overlap score, which supports the 'breadth not judgment' conclusion and the seriousness-stratified coverage numbers (e.g., GPT covers 46% of Essential concerns vs. 12% of Minor concerns), rests entirely on a GPT-5.5-based concern extraction and matching pipeline. Appendix E.1 states that prompts were 'iteratively refined until the inspected outputs aligned with our judgments' on the same corpus, with no held-out validation, human gold standard, or inter-annotator agreement reported. The sensitivity check in Appendix E.5 only varies the matcher (embedding vs. LLM) while reusing the same extracted concerns; it reports per-paper correlation of only 0.50 and pairwise agreement of 0.39, which is not strong evidence of robustness. Without independent human annotation of extracted concerns and matched pairs, the specific coverage numbers and the related conclusion in §6 that 'the main value","section":"§5.3, Appendix E.1/E.5"},{"comment":"The LLM-as-a-Judge evaluation uses GPT-5 to rate reviews that include GPT-5's own output. This creates a same-model self-preference bias, which the authors acknowledge in the Limitations. The judge scores are not merely incidental: they directly feed the paper's claim that 'aggregate quality scores alone can overestimate review quality' and the observation that GPT and Qwen receive near-maximum scores on most dimensions. The acknowledgment that the paper 'does not rely on LLM-as-a-Judge as the only evaluation' mitigates the concern but does not remove it for the aggregate-quality claim. I recommend either using a judge from a different model family, adding a human evaluation of a subset of reviews, or explicitly reporting a sensitivity analysis that quantifies the self-preference effect (e.g., by also running the judge on human reviews of varying style).","section":"§5.5, Table 4, Limitations"},{"comment":"The Nature Communications component includes only accepted papers, because rejected submissions are not public. The paper acknowledges this in the Limitations, but the cross-venue comparison in Appendix E.4—coverage drops by roughly one third to one half from ICLR to Nature Communications for all sources—is affected by this selection bias: human reviews of accepted papers may differ systematically from reviews of a mixed set, and the AI reviews are generated for the same accepted papers. This makes the venue comparison difficult to interpret as a pure effect of review setting. The authors should either restrict cross-venue claims to within-venue relative comparisons or explicitly discuss the direction of the bias for each metric.","section":"§4.1.2, Appendix E.4"},{"comment":"The recommendation that AI should be used as a 'second-pass assistant' is reasonable and well-hedged, but the manuscript goes further in saying AI reviews 'may also support the formulation of more actionable feedback' based on the recommendation-category analysis in Appendix E.3. That analysis is also produced by the same unvalidated GPT-5.5 extraction pipeline, and the categories are assigned by the extractor rather than by human readers. Since this is a secondary point, it can be fixed by softening the language or by providing a small human-coded verification of the recommendation categories.","section":"§6, 'AI as Review Support'"}],"minor_comments":[{"comment":"There is inconsistent capitalization and naming: 'GranuScore' vs. 'Granuscore' in the text; 'Qwen3-VL-235B' is sometimes referred to as 'Qwen3'; and 'Nature Comm.' vs. 'Nature Communications' are used interchangeably. Please standardize.","section":"General"},{"comment":"Typo: 'indiviual' should be 'individual'.","section":"Figure 6 caption"},{"comment":"The sentence 'Every main-track submission at AAAI 2026 received one clearly identified AI review...' cites Biswas et al. (2026), but the reference entry does not clarify whether this was an official pilot or an external experiment. Please add a sentence describing the nature of the deployment, since it is used to motivate the paper.","section":"§1"},{"comment":"The standard deviations for GPT dimensions in Table 4 are reported as 0.0 for many entries (e.g., Understanding, Coverage, Evidence Support, Constructiveness for ICLR). This suggests a ceiling effect. It would be informative to report the full distribution or the number of papers at the maximum score, so readers can judge the discriminative power of the metric.","section":"Table 4"},{"comment":"The explanation for excluding strengths is clear and sensible, but the example with paper OWHKdYwYiF would benefit from a citation to the specific review excerpt so that the reader can verify the point.","section":"Appendix E.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important topic and contains valuable descriptive results, especially the policy survey and the qualitative evidence of LLM review weaknesses. The two main concerns—the unvalidated overlap matcher and the same-model LLM-as-a-Judge—are acknowledged but not resolved; they affect the paper's central claims about coverage and aggregate quality overestimation. I believe the paper can be made acceptable with additional validation or substantial hedging, but the current version is not yet ready for publication. The venue-level policy survey alone is likely to be of significant interest to the community, and the dataset release is a concrete contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time, but read the appendices before trusting the overlap numbers.\n\nWhat is actually new: the venue-level survey of reviewer-facing AI policies across 111 AI/NLP conferences and medical journals. That is a real gap, and the two-community contrast (permissive AI/NLP vs. prohibitive medical, with different partial-policy flavors) is a solid empirical contribution. The dataset of pre-revision submissions with matched human and AI reviews is also a genuine step forward; existing corpora mostly give you post-revision papers, which undercuts review evaluation. The descriptive findings are well supported and honestly reported: all three LLMs inflate scores, Llama and Qwen scores are essentially uncorrelated with human scores at ICLR, Llama generates generic and sometimes degenerate praise, and the LLM-as-a-judge gives GPT and Qwen near-ceiling ratings that do not survive contact with other metrics.\n\nNow the soft spots, in proportion. The overlap metric is the load-bearing piece for the claim that LLMs add breadth, especially on Essential concerns (GPT 46% vs. 12% Minor). That metric depends on a GPT-5.5 extraction-and-matching pipeline that was iteratively refined on the same corpus, with no held-out validation or human gold standard. The sensitivity check swaps the matcher but reuses the same extracted concerns, so it cannot detect extraction bias. Per-paper correlation between the two matchers is only 0.50, and pair-level agreement 0.39. So the headline coverage numbers, and the \"breadth not judgment\" conclusion, are matcher-dependent. This is a real weakness, not a manufactured one. Also worth noting: the LLM-as-a-judge uses GPT-5 to rate GPT-5 reviews, which the authors acknowledge in Limitations; the Nature Communications sample is accepted papers only; and the promised code/data release has no URL or commit hash. None of these sink the paper, but they cap how far the absolute quality comparisons can be trusted.\n\nThe good news is that the central claims that do not depend on the overlap metric—score inflation, lack of calibration, generic criticism, generation degeneracy—are supported by direct, human-auditable evidence and would survive even if the overlap analysis were removed entirely. The policy survey stands on its own.\n\nBottom line: send it to peer review. The referee should require either external validation of the extraction/matching pipeline or a softened interpretation of the coverage claims, and a working code/data link. The paper is for editors, program chairs, and anyone building or evaluating AI review assistants; they will get a useful dataset and a well-motivated warning against single-metric evaluation.","headline":"A genuinely useful venue-level policy survey and a carefully documented LLM-review evaluation whose headline coverage numbers rest on a matcher tuned on the same data; the core descriptive findings survive, the coverage claims are fragile.","tokens_in":24144,"tokens_out":1505,"would_cite":true,"duration_ms":19115,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated peer reviews are fluent but systematically over-positive and poorly calibrated.","keywords":["peer review","large language models","AI-assisted reviewing","reviewer AI policy","review quality evaluation","score alignment","concern overlap","LLM-as-a-Judge"],"falsifier":"Take the same 50 ICLR papers and have independent expert reviewers, blind to source, judge whether each AI-identified concern expresses the same point as a pooled human concern. If their agreement falls well below the reported coverage (e.g., GPT's 46% for Essential concerns), the overlap-based conclusion loses support. A complementary check: run a GPT variant prompted to match the human score distribution; if it still receives near-maximum LLM-judge scores while failing to rank papers, that confirms aggregate quality scores ignore calibration.","tokens_in":23180,"feed_emoji":"🤖","tokens_out":11906,"duration_ms":132289,"temperature":0.7,"pith_summary":"Using 50 original ICLR 2026 submissions and 31 Nature Communications papers with matched human and machine reviews, the paper asks whether current LLMs can produce genuinely useful peer reviews. It finds that GPT-5, Llama-4, and Qwen3 write long, fluent, detailed reviews that receive high LLM-judge quality scores, but all three are systematically over-positive: they average 6.8–7.9 on a 10-point scale where human reviewers average 4.3, and only GPT's scores track human rankings (correlation 0.62; Llama and Qwen are statistically indistinguishable from zero). The central contention is that review quality is multi-dimensional, so aggregate quality scores alone overestimate AI review usefulness; the real value of current AI reviews is breadth — surfacing more candidate concerns — not judgment. Separately, a policy survey of 111 venues finds AI/NLP conferences mostly permit partial AI assistance while medical journals mostly prohibit reviewer AI use, so regulation is community-specific and still unsettled.","feed_headline":"AI reviews read well but can't match human review scores","feed_subtitle":"On 50 ICLR papers, only GPT tracked human ratings (0.62); Llama and Qwen did not, and all scored too high.","key_machinery":"The argument is carried by a four-part evaluation suite applied to identical manuscript-and-review pairs. The centerpiece is the overlap score: reviews are decomposed into atomic concerns by GPT-5.5, each tagged as strength or weakness with a seriousness level, and a paper-grounded LLM matcher links AI concerns to pooled human concerns for the same manuscript, producing raw and length-normalized coverage. Around this sit score alignment (correlating model and human paper scores on ICLR), Granuscore (a reference-free measure of semantic granularity, where lower means more specific feedback), and LLM-as-a-Judge (GPT-5 ratings on understanding, coverage, evidence support, constructiveness, conc","core_discovery":"On the paper's own terms, current LLMs are useful reviewer assistants but not autonomous reviewers. The evidence has three parts. First, calibration fails: mean LLM scores are 6.8–7.9 out of 10 versus 4.3 for humans; GPT separates accepted from rejected papers by only about half the human gap (0.87 vs 1.53 points), and Llama and Qwen show no separation at all. Second, depth of critique is uneven: decomposing reviews into atomic concerns and matching them to pooled human concerns shows AI reviews cover more human concerns than a single reviewer in raw terms (GPT 0.34 vs human baseline 0.16), but this advantage vanishes when normalized for review length (all sources about 0.14–0.16), meaning t","pith_inferences":["Going beyond the paper: the overlap matcher was tuned until its outputs matched the authors' judgments, so the headline coverage numbers deserve an independent re-check; if blind expert raters call fewer AI-human pairs the same point, the breadth advantage shrinks.","Going beyond the paper: the Nature Communications sample contains only accepted papers, so the large overlap drop from ICLR to Nature may partly reflect selection, not just review format; rejected manuscripts would separate the two explanations.","Going beyond the paper: the paper's Limitations section notes that GPT-5 is both generator and judge; an independent human rating of the same reviews would test how much of GPT's near-maximum judge scores is stylistic affinity rather than quality.","Going beyond the paper: GPT's higher coverage of Essential concerns suggests a testable prompt intervention — asking models to name the single most important weakness — that might lift open-weight models without fixing score calibration."],"forward_implications":["A single LLM-as-a-Judge score is not a valid gate for deploying AI review tools; venues should require calibration, grounding, and overlap metrics alongside it.","Using current open-weight LLM review scores for accept/reject decisions would be close to random at the ranking level; even the best model tested is upward-biased and only moderately aligned.","AI reviews can serve as a second-pass checklist: they surface additional candidate concerns and attach proposed actions to almost every weakness, including more Essential concerns in raw coverage.","Reviewer-AI policy should be written for specific uses, not blanket rules: AI/NLP venues tend to permit language assistance while medical journals restrict manuscript uploads and emphasize confidentiality.","Human reviewers should form their own assessment before seeing AI output to avoid anchoring; this workflow matches the paper's evidence and its recommended role for AI as an assistant."],"supporting_citations":[{"why":"Supplies the closest prior evaluation pipeline and human-AI feedback overlap baselines (30.85% Nature, 39.23% ICLR) that this paper extends with a multi-model, multi-metric comparison.","marker":"[Liang et al., 2024b]"},{"why":"Provides the bias-aware review-generation prompting strategy and the LLM-as-a-Judge rubric the paper adapts for ICLR and Nature Communications.","marker":"[Tyser et al., 2024]"},{"why":"Prior evidence that AI-assisted reviews boost paper scores and acceptance rates, which the paper's positivity-bias finding replicates.","marker":"[Russo Latona et al., 2025]"},{"why":"Existing ICLR dataset contains post-revision manuscripts, motivating the paper's collection of original submissions as reviewers saw them.","marker":"[González-Márquez and Kobak, 2024]"},{"why":"Baseline claim that GPT-4 is only slightly helpful and generic on deeper critique, which the paper qualifies by showing GPT covers Essential concerns at higher rates.","marker":"[Robertson, 2023]"},{"why":"Supplies the atomic-decomposition idea (FactScore) that the overlap concern-extraction pipeline adapts.","marker":"[Min et al., 2023]"},{"why":"Publisher-level AI policy survey that the venue-level policy taxonomy extends.","marker":"[Zhuang et al., 2025]"},{"why":"Documents the first large-scale field deployment of AI-assisted peer review at AAAI-26, the deployment context the evaluation addresses.","marker":"[Biswas et al., 2026]"}],"fun_headline_variants":["AI peer reviews flatter authors, miss critical gaps","LLM reviews too generous to pass as human peer review","AI reviews score high but fail to judge like humans","Study: AI reviews fluent but overly positive, uneven","AI can't yet replicate human peer review judgment"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The result that AI reviews add breadth rather than better judgment rests on trusting the model that matches AI criticisms to human criticisms; if that matcher is biased toward AI phrasing, the coverage numbers overstate real overlap.","fun_headline_variants_meta":{"raw":{"variants":["AI peer reviews flatter authors, miss critical gaps","LLM reviews too generous to pass as human peer review","AI reviews score high but fail to judge like humans","Study: AI reviews fluent but overly positive, uneven","AI can't yet replicate human peer review judgment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2241,"prompt_tokens":739,"completion_tokens":1502,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":1436}},"tokens_in":483,"tokens_out":1502,"duration_ms":13599,"temperature":1.0,"reasoning_tokens":1436,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:31:54.262689+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 50 ICLR papers and have independent expert reviewers, blind to source, judge whether each AI-identified concern expresses the same point as a pooled human concern. If their agreement falls well below the reported coverage (e.g., GPT's 46% for Essential concerns), the overlap-based conclusion loses support. A complementary check: run a GPT variant prompted to match the human score distribution; if it still receives near-maximum LLM-judge scores while failing to rank papers, that confirms aggregate quality scores ignore calibration.","supporting_citations":[],"review_version":1}