Pith. sign in

REVIEW 5 major objections 3 minor 33 references

The paper claims that the choice of reviewer guideline — official conference criteria, especially NeurIPS's, versus LLM-distilled imitation of human reviewers — determines how closely LLM review scores match human judgment, and that strict

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:49 UTC pith:QSHPRK5J

load-bearing objection Useful comparative study on guideline prompts for LLM review; the core rankings are plausible but the missing human-human baseline and significance tests keep the magnitudes from being interpretable. the 5 major comments →

arxiv 2607.22553 v1 pith:QSHPRK5J submitted 2026-05-16 cs.CL cs.AI

Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review

Haowen Li , Yoichi Ishibashi , Masafumi Oyamada This is my paper
classification cs.CL cs.AI
keywords automated peer reviewreviewer guidelineslarge language modelsprompt designrubric scoringreview qualityhuman-AI agreementRMSE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the kind of reviewer guideline fed to a large language model changes how well the model's review scores agree with human judgment. The authors report that official conference guidelines — in particular NeurIPS's — produce the closest match to human average scores, clearly better than no guideline at all. Guidelines distilled by an LLM from high-quality and low-quality human reviews are generally less effective than official conference guidelines. They also find that forcing a strict rubric-style point system consistently raises error compared with allowing free-form holistic scoring. The practical upshot: for LLM-based review assistants, community-refined criteria with subjective scoring appear to be the better design.

Core claim

On a set of 1,500 ICLR 2024 submissions, three open-weight LLMs assigned 1–10 ratings under different prompt conditions, and the paper measured root-mean-squared error against the average of the human reviews for each paper. Its central finding is that official conference reviewer guidelines, especially the NeurIPS guideline, give the lowest RMSE (best agreement), while guidelines generated by imitating human reviewers — even those distilled from 'Good' reviews — generally underperform official guidelines. A rubric version that scores each checklist item +1/-1 and normalizes the sum to 1–10 performs consistently worse than letting the model judge holistically. The paper interprets this as ev

What carries the argument

The reviewer guideline is the manipulated variable: official venue guidelines (ICLR, NeurIPS, ARR), LLM-generated reviewer-imitating checklists (from 'Good', 'Middle', and 'Bad' human-review groups), and a rubric variant with explicit +1/-1 point allocation normalized to the 1–10 scale. The comparison metric is RMSE between LLM score and average human score. The design isolates what instructions alone do to an LLM's scoring behavior, and the qualitative analysis contrasts the structured, actionable review produced under NeurIPS guidelines with the vague, poorly calibrated review produced under an imitative 'Bad reviewer' guideline.

Load-bearing premise

The load-bearing premise is that the average of the human review scores is a stable, meaningful ground truth, so that RMSE between an LLM's single score and that average measures review quality.

What would settle it

Compute the RMSE between one individual human reviewer's score and the average of the remaining reviewers on the same papers. If that baseline is around 2 points on the 1–10 scale, the reported LLM differences (2.07–3.18) fall within normal human disagreement noise, and the rankings would not establish guideline quality.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Adding official conference guidelines as prompts lowers RMSE versus a bare no-guideline prompt across all three tested LLMs.
  • Among official guidelines, NeurIPS yields the best alignment for all three models; ICLR and ARR are close but worse, and rewriting the same guidelines as imperative instruction prompts can flip which venue wins.
  • Reviewer-imitating guidelines distilled from human review text do not reliably beat the no-guideline baseline and consistently lose to official conference guidelines.
  • Enforcing rubric-style additive scoring (+1 for 'good' items, -1 for 'bad', normalized) increases error relative to free-form holistic scoring in every model and review-quality group tested.
  • When given structured official guidelines, LLM score distributions spread across the 1–10 scale instead of collapsing around the mean, so guideline quality also counters shortcut 'mean score' behavior.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable implication the paper leaves open: part of the official-guideline benefit may come from forcing score spread rather than from deeper understanding; rank-correlation or calibration metrics would separate those mechanisms.
  • If the rubric result generalizes, then decomposing scientific quality into additive checklist points is a poor design for LLM reviewers — a caution for review-assistant systems being built on point-based scorecards.
  • Because reviewer-imitating guidelines encode whatever habits dominate the training reviews, distilling guidelines from 'Bad' reviews may actually teach an LLM to write positive-sounding text that disagrees with its own score, which the paper's single qualitative case illustrates; a larger systematic text-score consistency measurement would test this.
  • The English-only ICLR data means the superiority of conference-refined guidelines over imitation is untested in other fields and languages; cross-venue and cross-language replication would show whether the ranking is a property of these guidelines or of guideline design in general.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper investigates how different types of reviewer guidelines affect LLM-based automated peer review. Using 1,500 ICLR 2024 submissions and three open-weight LLMs, it compares a no-guideline baseline, official conference guidelines from ICLR/NeurIPS/ARR (both verbatim and rewritten as instruction-style prompts), and 'reviewer-imitating' guidelines distilled from filtered human reviews, with and without rubric-style scoring. The central claim is that official conference guidelines, especially NeurIPS, yield scores most consistent with human judgments, that reviewer-imitating guidelines are generally less effective, and that enforcing strict rubric-style scoring consistently degrades performance. The evaluation metric is RMSE between the LLM's single overall score and the average human overall score.

Significance. If established, the paper's practical recommendation—use official conference guidelines, especially NeurIPS, with free-form scoring, when building LLM review assistants—would be useful and actionable. The study has genuine strengths: it uses open-weights models, a reasonably large evaluation set, fully specified prompt templates in the appendices, and a qualitative comparison of review justifications. The official-guideline comparison is anchored to externally written documents and is not circular. However, the current evidence does not yet support the central claim. The missing human-human agreement baseline, the absence of uncertainty quantification, and several internal inconsistencies in the reported results and rubric design mean that the comparative rankings are not yet interpretable as measures of review quality. The stress-test concern about the absent human baseline lands: without it, an LLM RMSE of 2.07 cannot be called 'most consistent with human judgments.'

major comments (5)
  1. [§2 Evaluation; Tables 1–2] The central measure is RMSE between one LLM score and the average human score, but no human-human agreement baseline is reported (e.g., leave-one-human-out RMSE against the mean of the remaining reviewers on the same papers), and no confidence intervals, error bars, or paired significance tests accompany the point estimates. Without a baseline, RMSE magnitudes are uninterpretable as quality: if human reviewers disagree by RMSE ~1.5, then even the best result (NeurIPS, 2.07) is far from human-level agreement; if human-human RMSE is ~2.5, it is more credible. Several differences that motivate the ranking are small (e.g., Table 2: DeepSeek Good 3.13 vs Middle 3.12). The Limitations section acknowledges reliance on RMSE but does not supply the needed baseline. Please report human-human agreement and uncertainty/paired comparisons for the headline differences.
  2. [§3, Q1; Table 1] The claim that 'RMSE consistently decreases across all models when guidelines are provided' is contradicted by Table 1: for Qwen3-30B-A3B, official ARR RMSE is 3.18 versus 3.06 for No-Guideline. The following sentence, 'the instruction-style prompts outperform the baseline for all tested models,' is also contradicted by the same model's instruction-style NeurIPS result (3.08 vs 3.06). This is not a wording issue: RQ1's answer explicitly states that providing guidelines improves performance, but the table shows some guideline presentations can degrade performance. Please correct the summary and qualify RQ1 to specific guideline/prompt combinations.
  3. [Appendix C, Listing 13; §4 Q3] The rubric scoring procedure is internally inconsistent. The formula normalized_score = round(((temporary_score + 10) / 20) * 9 + 1) maps temporary_score = 0 to 5.5, which rounds to 6 under the usual round-half-up rule; the listing, however, states the mapping '0 -> 5'. If the implementation used round, rubric scores at the center of the scale are shifted by +1 relative to the stated design; if it used floor, the formula should say floor, not round. This condition is the entire basis for RQ3's claim that strict rubric-style scoring degrades performance. Please clarify the exact rounding rule and verify that the implementation matches the stated mapping.
  4. [§4 STEP1–STEP2; §3 Setup] The reviewer-imitating guidelines are generated from 30 reviews sampled from the ICLR 2024 dataset, while the evaluation uses 1,500 submissions from the same dataset. The paper does not state that the seed reviews are disjoint from the evaluation papers. If any seed review belongs to a paper in the evaluation sample, the guideline already encodes information correlated with that paper's human ground truth, confounding RQ2 and RQ3. Please either confirm disjointness and explain how it was ensured, or re-run the analysis with an explicitly held-out seed set.
  5. [§4 STEP1; Appendix A.1] The Good/Middle/Bad grouping is derived from a composite of five structural proxies with arbitrary thresholds (submission-date deviation bins, length bins, composite cutoffs at 17, 11, and ≤6). Appendix A.1 states that 'we empirically analyze whether these assumptions hold true and investigate which criteria ... correlate with automated review performance,' but no such analysis appears in the manuscript. Since the group labels are the independent variable for RQ2/RQ3, this missing support is load-bearing. Please either include the promised analysis or explicitly reframe the results as conditional on these specific filtering choices.
minor comments (3)
  1. [§4 Overview] Typo: 'Good,Middle, andBad Review' should be 'Good, Middle, and Bad Review.'
  2. [Appendix C, Listing 11] The final bullet under 'methodological_soundness' in the Middle guideline appears truncated: 'periments, theory, or statistically significant results (e.g., overlapping confidence intervals).' The intended sentence is incomplete.
  3. [§2; Appendix C] The text refers to 'the prompt template in Appendix (Listing 6)' without naming the appendix; it should be Appendix C. Also, the claim about rubric-style scoring in the abstract should be scoped to the specific binary equal-weight rubric in Listing 13, since other rubric designs (e.g., partial credit, weighted dimensions) are not tested.

Circularity Check

0 steps flagged

No significant circularity: the paper's main comparisons are empirical, with official conference guidelines serving as external anchors; no prediction reduces to a fitted input by construction.

full rationale

The paper's central claim is an empirical ranking of RMSE between LLM-generated scores and human average scores under different guideline conditions. The official conference guidelines (ICLR, NeurIPS, ARR) are external, human-written documents and are not derived from the evaluation labels, so the RQ1/RQ2 comparisons do not reduce to their inputs by construction. The statement that NeurIPS guidelines are most effective is not self-definitional because the human labels come from ICLR 2024 reviews, not from NeurIPS guidelines. The reviewer-imitating guideline arm is generated from 30 reviews sampled from the same ICLR 2024 pool that supplies the ground-truth labels, and the paper does not document exclusion of those seed reviews/papers from the 1,500 evaluation papers; this is a potential in-distribution leakage concern in §3 Setup / §4 STEP2 / §A.2, but it is not a formal equation equating the prediction to the fitted input, and any such leakage would likely bias toward the reviewer-imitating arm, against the paper's headline claim that official guidelines are better. The Limitations section acknowledges reliance on RMSE, but the lack of a human-human agreement baseline is a validity/interpretability limitation, not a circularity. No load-bearing self-citations are present: the cited 'Li et al., 2025' refers to a different author group (Ruochi Li et al.), and no other references overlap with the authors of this paper. Therefore no circular step is exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper's empirical contribution rests on hand-chosen filtering thresholds and several domain assumptions about what makes a review good and what metric validates automated review. The most consequential free choices are the Good/Middle/Bad split and the rubric arithmetic. These do not undermine the official-guideline comparison but do bound the generality of the reviewer-imitating conclusions.

free parameters (5)
  • Good/Middle/Bad review-quality thresholds = Good=17, Middle=11, Bad≤6 (composite 3-17)
    Thresholds chosen by hand to split review pools; they determine which human reviews seed the reviewer-imitating guidelines (§A.2).
  • Review length score bins = 0.5, 0.8, 1.2, 1.5 × median
    Hand-chosen cutoffs in the length-filter criterion; different cutoffs change the Good/Middle/Bad pools.
  • Submission-date deviation bins = 2, 5, 10, 20 days
    Hand-chosen bins for the promptness proxy in the review-quality filter (§A.2).
  • Seed-review sample size = 30 per quality pool
    Arbitrary sample size for guideline generation (§4 Step 2); affects the content of generated guidelines.
  • Rubric normalization formula = round(((temp+10)/20)*9+1)
    Fixed but hand-specified mapping from -10..10 checklist score to 1..10 rating; if changed, rubric results change.
axioms (5)
  • domain assumption Review-quality filtering proxies (length, prompt submission, citation engagement, consistency) correlate with actual review quality.
    Appendix A.1 asserts this intuition; if false, 'Good' and 'Bad' guidelines are mislabeled and §4 comparisons lose meaning.
  • domain assumption RMSE between LLM score and average human score measures automated review quality.
    Section 2 defines the metric; the paper never validates it against human-human agreement.
  • domain assumption The 1,500-paper sample from ICLR 2024 is representative and the human average is a stable reference.
    Section 3 setup; no stratification or confidence intervals reported.
  • domain assumption Guidelines generated by Gemini 2.5 Pro from 30 reviews capture stable reviewer behavior rather than idiosyncrasies of the sample.
    Section 4 Step 2; no cross-model or multi-sample validation.
  • domain assumption The three open-weight LLMs follow the provided guidelines and produce scores comparable to human averages.
    Required for all RMSE comparisons; no inspection of instruction-following rate.

pith-pipeline@v1.3.0-alltime-deepseek · 18219 in / 14190 out tokens · 116998 ms · 2026-08-02T13:49:15.051559+00:00 · methodology

0 comments
read the original abstract

Peer review is an essential process in scientific research, yet the growing workload has made its automation increasingly necessary. In this study, we analyze how different types of reviewer guidelines, such as official conference guidelines and reviewer-imitating ones generated from high-quality human reviews using LLMs, affect automated peer review. Our experiments show that official conference guidelines produce review results most consistent with human judgments, suggesting that evaluation criteria refined through conference practice serve as effective guidance for automated reviewing as well. In contrast, reviewer-imitating guidelines were generally less effective than official conference guidelines. Furthermore, enforcing strict rubric-style scoring consistently degraded performance, highlighting the importance of allowing subjective and holistic scoring.

Figures

Figures reproduced from arXiv: 2607.22553 by Haowen Li, Masafumi Oyamada, Yoichi Ishibashi.

Figure 1
Figure 1. Figure 1: Scatter plots comparing LLM scores to human scores for the Qwen3-30B-A3B model across various [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Histograms comparing the score distributions of the No-Guideline baseline against the NeurIPS guideline [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 7 linked inside Pith

  1. [1]

    A Dataset of Peer Reviews (PeerRead): Collection, Insights and

    Kang, Dongyeop and Ammar, Waleed and Dalvi, Bhavana and van Zuylen, Madeleine and Kohlmeier, Sebastian and Hovy, Eduard and Schwartz, Roy , booktitle =. A Dataset of Peer Reviews (PeerRead): Collection, Insights and. 2018 , address =

    2018

  2. [2]

    2023 , url =

    Dycke, Nils and Kuznetsov, Ilia and Gurevych, Iryna , booktitle =. 2023 , url =

    2023

  3. [3]

    2023 , doi =

    Lin, Jialiang and Song, Jiaxin and Zhou, Zhangping and Chen, Yidong and Shi, Xiaodong , journal =. 2023 , doi =

    2023

  4. [4]

    2024 , eprint =

    ReviewMT: A Multi-Turn Scholarly Peer Review Corpus and Benchmark , author =. 2024 , eprint =

    2024

  5. [5]

    Proceedings of INLG 2020 , year =

    ReviewRobot: Explainable Paper Review Generation Based on Knowledge Graphs , author =. Proceedings of INLG 2020 , year =

    2020

  6. [6]

    Journal of Artificial Intelligence Research , volume =

    Can We Automate Scientific Reviewing? , author =. Journal of Artificial Intelligence Research , volume =. 2022 , doi =

    2022

  7. [7]

    Zhu, Axiang and others , booktitle =. Is. 2024 , url =

    2024

  8. [8]

    2025 , eprint =

    Automatically Evaluating the Paper Reviewing Capability of Large Language Models , author =. 2025 , eprint =

    2025

  9. [9]

    Proceedings of ACL 2025 (Long Papers) , year =

    DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process , author =. Proceedings of ACL 2025 (Long Papers) , year =

    2025

  10. [10]

    Is Your Paper Being Reviewed by an

    Yu, Sungduk and Luo, Man and Madasu, Avinash and Lal, Vasudev and Howard, Phillip , year =. Is Your Paper Being Reviewed by an. 2410.03019 , archivePrefix =

    Pith/arXiv arXiv

  11. [11]

    and Zou, James Y

    Liang, Weixin and Izzo, Zachary and Zhang, Yaohui and Lepp, Haley and Cao, Hancheng and Zhao, Xuandong and Chen, Lingjiao and Ye, Haotian and Liu, Sheng and Huang, Zhi and McFarland, Daniel A. and Zou, James Y. , year =. Monitoring. 2403.07183 , archivePrefix =

    Pith/arXiv arXiv

  12. [12]

    NEJM AI , volume =

    Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis , author =. NEJM AI , volume =. 2024 , doi =

    2024

  13. [13]

    2408.10365 , archivePrefix =

    Tyser, Keith and Segev, Ben and Longhitano, Gaston and Zhang, Xin-Yu and Meeks, Zachary and Lee, Jason and Garg, Uday and Belsten, Nicholas and Shporer, Avi and Udell, Madeleine and Te'eni, Dov and Drori, Iddo , year =. 2408.10365 , archivePrefix =

    Pith/arXiv arXiv

  14. [14]

    2023 , eprint =

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models , author =. 2023 , eprint =

    2023

  15. [15]

    and others , year =

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and others , year =. Judging. 2306.05685 , archivePrefix =

    Pith/arXiv arXiv

  16. [16]

    Scientometrics , volume=

    How do journals of different rank instruct peer reviewers? Reviewer guidelines in the field of management , author=. Scientometrics , volume=. 2020 , publisher=

    2020

  17. [17]

    CoRR , volume =

    Chhavi Kirtani and Madhav Krishan Garg and Tejash Prasad and Tanmay Singhal and Murari Mandal and Dhruv Kumar , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2502.11736 , eprinttype =. 2502.11736 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.11736 2025

  18. [18]

    CoRR , volume =

    OpenAI , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2303.08774 , eprinttype =. 2303.08774 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774 2023

  19. [19]

    2020 , eprint=

    A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences , author=. 2020 , eprint=

    2020

  20. [20]

    2025 , eprint=

    The AI Imperative: Scaling High-Quality Peer Review in Machine Learning , author=. 2025 , eprint=

    2025

  21. [21]

    2025 , eprint=

    Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards , author=. 2025 , eprint=

    2025

  22. [22]

    Proceedings of the 23rd ACM Conference on Economics and Computation , pages =

    Zhang, Yichi and Yu, Fang-Yi and Schoenebeck, Grant and Kempe, David , title =. Proceedings of the 23rd ACM Conference on Economics and Computation , pages =. 2022 , isbn =. doi:10.1145/3490486.3538235 , abstract =

    arXiv 2022

  23. [23]

    2025 , eprint=

    Large language models for automated scholarly paper review: A survey , author=. 2025 , eprint=. doi:https://doi.org/10.1016/j.inffus.2025.103332 , url=

    arXiv 2025

  24. [24]

    2025 , eprint=

    Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews , author=. 2025 , eprint=

    2025

  25. [25]

    Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work , articleno =

    Chen, Shiping and Brumby, Duncan and Cox, Anna , title =. Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work , articleno =. 2025 , isbn =. doi:10.1145/3729176.3729196 , abstract =

    arXiv 2025

  26. [26]

    2025 , eprint=

    Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attacks , author=. 2025 , eprint=

    2025

  27. [27]

    2025 , eprint=

    Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Papers , author=. 2025 , eprint=

    2025

  28. [28]

    2025 , eprint=

    CycleResearcher: Improving Automated Research via Automated Review , author=. 2025 , eprint=

    2025

  29. [29]

    2025 , eprint=

    AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition , author=. 2025 , eprint=

    2025

  30. [30]

    ICLR 2024 Conference — Reviews and Submissions (OpenReview) , howpublished =

    2024

  31. [31]

    2025 , eprint=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2025 , eprint=

    2025

  32. [32]

    2025 , journal =

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author =. 2025 , journal =

    2025

  33. [33]

    CoRR , volume =

    An Yang and Anfeng Li and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Gao and Chengen Huang and Chenxu Lv and Chujie Zheng and Dayiheng Liu and Fan Zhou and Fei Huang and Feng Hu and Hao Ge and Haoran Wei and Huan Lin and Jialong Tang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jian Yang and Jiaxi Yang and Ji...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025