Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a 7B reasoning model trained with multi-objective reinforcement learning can generate peer reviews that score more than twice as high as human reviews and existing review-generation systems on a human-aligned reward…

desk verdict Useful dataset and training recipe, but the headline '>2x better than humans' conflates optimizing the reward with measuring review quality; the paper deserves engagement, not acceptance as-is. read the letter →

arxiv 2505.11718 v2 pith:L3IWBGBF submitted 2025-05-16 cs.AI

Pawin Taechoyotin , Daniel Acuna This is my paper · ORCID
classification cs.AI
keywords AIpeerreviewgenerationmulti-objectivereinforcementlearningGRPOreasoninglanguagemodelsqualityrewardRTdatasetHPRRautomatedevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that an open 7B reasoning language model can be trained to write scientific peer reviews that score far above human reviews on an automated review-quality metric. The authors build a reward function, HPRR, from eight sentence-level aspects of review quality plus a METEOR relevance score, then supervise-fine-tune a reasoning model on a new corpus of ICLR reviews augmented with thinking traces, and finally apply GRPO reinforcement learning under two weightings. They report that the two resulting models, REMOR-U and REMOR-H, achieve more than twice the average rewards of human reviewers, non-reasoning agentic review systems, and commercial LLM baselines, and that the advantage comes mainly from avoiding the long tail of low-quality human reviews. The authors state the key caveat themselves: the evaluation is based on the constructed reward function, not on blinded human judgment, so the headline result stands or falls with that function.

What carries the argument

The load-bearing object is the Human-aligned Peer Review Reward (HPRR): a scalar score formed as a weighted sum of eight sentence-level review-quality aspects, normalized per sentence, plus a METEOR relevance score between the generated review and the manuscript text. HPRR does double duty: it is the reward signal used during GRPO training and the metric used to compare models against humans and baselines. The weights for the human-aligned variant are estimated from pairwise human votes with a constrained Bradley-Terry model, and the optimization itself is carried out with Group Relative Policy Optimization on a 7B reasoning model that was first supervised-fine-tuned on PeerRT, a corpus of ICLR reviews augmented with thinking traces.

What would settle it

Compute HPRR on a held-out set of real human reviews that have independent expert quality ratings; if HPRR correlates weakly or negatively with those ratings, the reported two-to-one advantage over humans does not establish better reviews. A blinded randomized comparison in which domain experts rate REMOR and human reviews for usefulness, actionability, and correctness would settle the question directly.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a 7B reasoning model can be turned into a peer-review generator whose outputs score more than twice as high as human-written reviews on a purpose-built quality metric. The authors construct HPRR, a weighted sum of eight sentence-level aspects (criticism, example, importance and relevance, materials and methods, praise, presentation and reporting, results and discussion, suggestion and solution) plus a METEOR relevance score between the review and the manuscript. They fine-tune DeepSeek-R1-Distill-Qwen-7B on PeerRT, a corpus of ICLR reviews augmented with synthetic reasoning traces, then apply GRPO under either uniform weights (REMOR-U) or weights derived from human pairwise preferences (REMOR-H). They report that both models exceed the average rewards of human reviewers, agentic review systems, and commercial LLMs by a factor of two, and that the advantage comes from eliminating the long tail of low-quality human reviews rather than surpassing the best human reviews. The authors also report that the uniform model produces qualitatively more substantive feedback, while the human-aligned model's reviews become long and redundant, an effect they attribute to the weight placed on the METEOR relevance component.

Load-bearing premise

The load-bearing premise is that HPRR, the paper's automated review-quality score, actually tracks what expert humans mean by a useful review, since the model is trained to maximize it and then judged by it.

Editorial extensions

If this is right

  • A peer-review assistant built this way could run in about a minute, where the agentic baselines it beats take roughly 20 to 30 minutes.
  • Since the largest gain is against the low-quality tail of human reviews, such a model could raise the floor of review quality rather than just matching the best human reviewers.
  • HPRR itself can be reused as a self-assessment score for human reviewers, giving a quick automated signal of which review aspects a draft covers.
  • A reasoning model optimized with GRPO can improve several review aspects at once without needing multi-agent coordination, figure understanding, or external knowledge retrieval.
  • The choice of reward weighting matters for style: human-vote-derived weights favor suggestions and relevance but discourage criticism, so the weighting is a policy choice, not just a technical detail.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The human-aligned weights are estimated from only about 130 pairwise votes; a larger preference dataset could shift which review style is rewarded, so the specific REMOR-H results should be read as tied to that sample.
  • Because the METEOR relevance component rewards overlap with the manuscript, REMOR-H's long and redundant reviews suggest that HPRR may need a length penalty or a saturating cap before it is used as a general evaluation metric.
  • That human-vote-derived weights down-weight criticism suggests preference-based rewards may capture politeness norms rather than critical value; a system aimed at scientific rigor might want to invert those weights.
  • If HPRR tracks expert judgment, the same supervised-fine-tuning-plus-GRPO recipe could transfer to other expert writing tasks with multi-dimensional rubrics, such as grant reviews or clinical assessments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents REMOR, a pipeline for automated peer-review generation that combines supervised fine-tuning of DeepSeek-R1-Distill-Qwen-7B on a new PeerRT dataset (ICLR 2017-2020 reviews augmented with synthetic reasoning traces) with GRPO reinforcement learning against a newly proposed HPRR reward function. Two model variants are trained: REMOR-U with uniform reward weights and REMOR-H with weights fitted to human preference judgments. The paper reports that both variants achieve more than twice the average reward of human reviews and of several baseline systems, with lower variance, and it offers a qualitative comparison as well as released code, datasets, and models.

Significance. If the empirical claims were supported, the paper would be a significant contribution: it would show that a 7B reasoning model fine-tuned against a well-designed reward can generate peer-review reports comparable to the best human reviews while running in about a minute, and the released PeerRT dataset and HPRR function would support further research. The paper is transparent about its limitations and ships reproducible artifacts. However, the central claim is conditional on HPRR being a valid proxy for human-judged review quality, and that validity is not established; the paper's own qualitative analysis suggests the opposite. Consequently, the headline 'more than twice the average rewards' claim is not currently supported as a statement about review quality.

major comments (4)
  1. [§3.2, Fig. 2, Table 4] The central quantitative claim is circular. HPRR is the reward optimized by GRPO in §3.2, and the same function is then used as the evaluation metric in Fig. 2 and Table 4; the 'more than twice the average rewards' result is therefore largely a training-curve statement. The comparison is further weakened because Table 4 scores the same 5.5k ICLR papers from which the 90th-percentile training samples in §4.1 were selected, so the human-review baseline and the test set overlap the training set. An independent evaluation would require a held-out set and a measure that was not part of the training objective.
  2. [Appendix Table 3, §4.2, §4.7] The human-aligned reward does not behave like a proxy for review quality even by the authors' own analysis. Appendix Table 3 assigns weight 8.67 of 9 to METEOR relevance and 0.01–0.16 to all substantive aspects; optimizing this axis rewards long manuscript-paraphrasing text. Section 4.7 reports exactly this failure mode: REMOR-H reviews are 'too long and redundant,' and the authors judge REMOR-U as qualitatively better even though REMOR-H has the higher human-aligned reward (0.670 vs 0.470 on ICLR; 1.438 vs 0.731 on ACL/NeurIPS). Thus the evaluation axis on which REMOR-H is declared the winner contradicts the qualitative judgment, which undermines the claim that HPRR-H measures human-aligned quality.
  3. [§3.2.1, Table 3] The 'human-aligned' weights are fitted to only 130 human preference votes and reach cross-validated F1 = 0.57 on a three-outcome prediction task, which is modest predictive performance; these same fitted weights then define the human-aligned evaluation axis used to declare REMOR-H the winner. With such a small fit, the paper's central premise that HPRR is aligned with human judgments is not established. The weight fitting also appears to drive the outcome: selecting the model that maximizes the fitted objective and then evaluating on that same objective is not an independent validation of human alignment.
  4. [§6] The paper explicitly concedes that the evaluation is not based on direct human feedback but on a constructed reward function, and it defers a blinded human evaluation to future work. Since the abstract's 'more than twice the average rewards' claim is intended to support the conclusion that REMOR produces higher-quality reviews than humans, the absence of any human validation is a load-bearing gap: no amount of optimization against a synthetic reward can, by itself, establish that claim. This gap cannot be closed by local edits; it requires a new evaluation study.
minor comments (4)
  1. [§4.2] The word 'Bradly' in §4.2 is a typo and should be 'Bradley'.
  2. [Fig. 3] The text refers to 'hard,' 'moderate,' and 'easy' difficulty groups, but the figure contains no group boundaries or labels; please add them.
  3. [§4.3] Reporting only that all SEM values were within 0.001 is not informative; report the per-model SEM or confidence intervals for each metric.
  4. [Appendix Table 1] Appendix Table 1 includes unadjusted weights with negative values; the text should explain why the ABT linear regression variant is preferred over the other models beyond the reported F1 score.

Circularity Check

3 steps flagged · score 7.0 of 10

HPRR is simultaneously the GRPO training objective and the evaluation yardstick; the human-aligned weights are fitted to the authors' own MAMORX preference data, so the '>2x superhuman reward' claim is largely a trained-objective curve rather than an independent quality prediction.

  1. self definitional [Abstract; Sec 3.1.3; Sec 3.2; Sec 4.3]
    "3.1.3 Reward Model: Our reward model combines a sentence-level review assessment model and a relevance score ... The final reward is the weighted sum of the eight normalized quality metrics and METEOR. ... Abstract: Our results show that REMOR-U and REMOR-H achieve more than twice the average rewards of human reviews, non-reasoning state-of-the-art agentic multi-modal AI review systems, and general commercial LLM baselines."

    REMOR-U and REMOR-H are trained by GRPO to maximize exactly this weighted sum (HPRR), and the headline 'more than twice the average rewards' is then measured on the same weighted sum. For the trained models, the reported superiority is therefore a training curve: increasing the training objective is the optimization target, not an independent prediction of review quality. The comparison against humans and non-REMOR baselines is an out-of-sample contrast on the metric, but the 'superhuman' quality claim reduces to the assumption that HPRR equals quality, which is assumed rather than derived.

  2. fitted input called prediction [Sec 3.2.1; Sec 4.2; Sec 4.3; Appendix Table 3]
    "We use the dataset of human preferences described in [41] ... The Adapted Bradly-Terry obtained the best five-fold cross-validation performance for predicting human preferences (Appendix Table 3). ... REMOR-H loses to REMOR-U regarding uniform reward but has the highest human-aligned reward at 0.670."

    The 'human-aligned' evaluation axis HPRR-H is a weighted sum whose weights are fit to 130 human preference votes, with cross-validated F1 = 0.57. REMOR-H is then trained with GRPO to maximize that same fitted weighted sum, and the paper's central comparison declares REMOR-H the winner on that axis. The winner is thus selected by a parameter fitted to the same preference data that is re-used as the outcome measure; the 'human-aligned' label is a fitted input rather than an independently validated ground truth.

1 more flagged steps
  1. self citation load bearing [Sec 3.2.1; References [41]; Sec 6]
    "We use the dataset of human preferences described in [41], which results from an arena-style competition across multiple human and AI-generated reviews. ... our human-aligned reward model is based on a relatively small sample of votes."

    The load-bearing human-preference dataset comes from [41], the authors' own prior MAMORX paper, and it is used both to define HPRR-H and to declare REMOR-H the best on that human-aligned axis. No external, machine-checked, or independently reproduced source is supplied for these 130 votes, and the paper itself concedes the sample is small. The 'human-aligned' claim is therefore supported by a self-citation chain that also provides the fitted weights used in the evaluation.

full rationale

The central quantitative claim is not an independent test of review quality: HPRR is the GRPO objective and the outcome measure. For the two REMOR models, 'more than twice the rewards' is the value of the training objective, so the claimed superiority over humans is substantially a training curve, though the comparisons against systems not trained on HPRR retain some empirical content as a metric benchmark. The 'human-aligned' variant is worse in this respect: its weights are fit to 130 preference votes from the authors' own MAMORX dataset, and the same fitted weighted sum is then used to declare REMOR-H the winner. The paper's own Section 4.7 contradicts the metric: REMOR-H has the highest human-aligned reward but is described as 'too long and redundant,' and the authors 'judge REMOR-U as qualitatively better,' while Section 6 concedes that 'our evaluation is not based on direct human feedback but on a constructed human-aligned reward function.' Additionally, the main ICLR evaluation (Table 4) is computed on the same 5.5k-paper pool from which the 1.7k training samples were selected, adding leakage risk. The headline 'superhuman' result therefore reduces substantially to the definition and fitting of the reward, although the release of HPRR and PeerRT and the metric-level comparison against external baselines are independent contributions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the validity of HPRR as a quality metric and on the fitted human-aligned weights. Because HPRR is both the training objective and the outcome measure, the number of genuinely independent evidence sources is low.

free parameters (3)
  • Human-aligned reward weights = ReME=8.67, SuSo=0.16, ImRe=0.11, others=0.01 after Laplace smoothing
    Fitted via Adapted Bradley-Terry linear regression on 130 human preference votes from the MAMORX arena, then min-max scaled and reweighted to sum to nine.
  • 90th percentile selection threshold = Score above the 90th percentile of uniform reward
    Used to select the 1.7k SFT and RL training samples; the threshold is arbitrary and affects which reviews are seen during training.
  • Laplace smoothing alpha = 0.01
    Applied to reward weights to prevent zero weights; chosen ad hoc.
assumptions (4)
  • domain assumption The Severin et al. sentence-level assessment model accurately measures review quality along eight aspects.
    HPRR uses this model's outputs for criticism, example, importance, materials, praise, presentation, results, and suggestion; the paper does not validate this model against human judgments of review quality.
  • domain assumption METEOR overlap between review and manuscript text is a valid relevance measure.
    METEOR is the ninth reward component and receives weight 8.67 of 9 in the human-aligned reward, yet the paper provides no evidence that METEOR tracks review helpfulness.
  • domain assumption Bradley-Terry weights estimated from 130 arena votes generalize to the review-generation task.
    The preference dataset is small and the cross-validated F1 is around 0.55, so the estimated weights are noisy.
  • ad hoc to paper Synthetic reasoning traces from Claude Sonnet 3.7 are suitable supervision for SFT and improve review generation.
    The traces are generated without human verification, and the paper does not ablate their contribution to the final performance.
invented entities (1)
  • HPRR reward function
    purpose: Quantifies review quality across eight sentence aspects plus METEOR relevance; used both as the GRPO training reward and as the evaluation metric for all comparisons.
    No external validation shows that HPRR predicts expert judgments of review quality. The paper itself calls for a future blinded human evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning." pith.science (2026). https://pith.science/paper/L3IWBGBF

@misc{pith2026250511718,
  author       = {Pith},
  title        = {Pith review of: REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3IWBGBF}},
  note         = {Machine review of arXiv:2505.11718}
}
read the original abstract

AI-based peer review systems tend to produce shallow and overpraising suggestions compared to human feedback. Here, we evaluate how well a reasoning LLM trained with multi-objective reinforcement learning (REMOR) can overcome these limitations. We start by designing a multi-aspect reward function that aligns with human evaluation of reviews. The aspects are related to the review itself (e.g., criticisms, novelty) and the relationship between the review and the manuscript (i.e., relevance). First, we perform supervised fine-tuning of DeepSeek-R1-Distill-Qwen-7B using LoRA on PeerRT, a new dataset of high-quality top AI conference reviews enriched with reasoning traces. We then apply Group Relative Policy Optimization (GRPO) to train two models: REMOR-H (with the human-aligned reward) and REMOR-U (with a uniform reward). Interestingly, the human-aligned reward penalizes aspects typically associated with strong reviews, leading REMOR-U to produce qualitatively more substantive feedback. Our results show that REMOR-U and REMOR-H achieve more than twice the average rewards of human reviews, non-reasoning state-of-the-art agentic multi-modal AI review systems, and general commercial LLM baselines. We found that while the best AI and human reviews are comparable in quality, REMOR avoids the long tail of low-quality human reviews. We discuss how reasoning is key to achieving these improvements and release the Human-aligned Peer Review Reward (HPRR) function, the Peer Review Reasoning-enriched Traces (PeerRT) dataset, and the REMOR models, which we believe can help spur progress in the area.

Figures

Figures reproduced from arXiv: 2505.11718 by the authors.

Figure 1
Figure 1. Illustrative diagram of how REMOR works. A. AI-review generation system with our proposed multi-dimensional reward function (HPRR), which evaluates the quality of the peer review. A sample review is shown and color-coded to indicate the relevance between the analysis metric and the review. For this instance, purple indicates content related to praise, and blue is content related to criticism. B. The training is perf… view at source ↗
Figure 2
Figure 2. The left panel shows the average reward across the metrics for each model. We can see [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Reward for each aspect per each optimization step. The left panel depicts the reward trend [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The comparison shows that REMOR outperforms non-reasoning models across all metrics. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The left panel shows the normalized mean across metrics. We can see the REMOR-U [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: A distribution of Reward (U) of human and REMOR-U reviews. The plot shows that [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AI for Auto-Research: Roadmap & User Guide

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.

Reference graph

Works this paper leans on

55 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  2. [2]

    Your paper has been accepted, rejected, or whatever: Automatic generation of scientific paper reviews

    Alberto Bartoli, Andrea De Lorenzo, Eric Medvet, and Fabiano Tarlao. Your paper has been accepted, rejected, or whatever: Automatic generation of scientific paper reviews. InAvailability, Reliability, and Security in Information Systems: IFIP WG 8.4, 8.9, TC 5 International Cross- Domain Conference, CD-ARES 2016, and Workshop on Privacy Aware Machine Lear...

    work page 2016

  3. [3]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URLhttp://www.jstor.org/stable/2334029

    arXiv 1952

  4. [4]

    Reviewer fatigue? why scholars decline to review their peers’ work.PS: Political Science & Politics, 48(4):595–600, 2015

    Marijke Breuning, Jeremy Backstrom, Jeremy Brannon, Benjamin Isaak Gross, and Michael Widmeier. Reviewer fatigue? why scholars decline to review their peers’ work.PS: Political Science & Politics, 48(4):595–600, 2015

    work page 2015

  5. [5]

    A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run.Water Resources Research, 49(6):3476–3486, 2013

    Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run.Water Resources Research, 49(6):3476–3486, 2013

    work page 2013

  6. [6]

    Ai-assisted peer review

    A Checco, L Bracciale, P Loreti, S Pinfield, and G Bianchi. Ai-assisted peer review. humanities and social sciences communications, 8 (1), 1-11, 2021

    work page 2021

  7. [7]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. InForty-first International Conference on Machine Learning, 2024

    2024

  8. [8]

    Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

Show all 55 references
  1. [9]

    Marg: Multi-agent review generation for scientific papers.arXiv preprint arXiv:2401.04259, 2024

    Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. Marg: Multi-agent review generation for scientific papers.arXiv preprint arXiv:2401.04259, 2024

    2024 arXiv

  2. [10]

    Can editors save peer review from peer reviewers? PloS one, 12(10):e0186111, 2017

    Rafael D’Andrea and James P O’Dwyer. Can editors save peer review from peer reviewers? PloS one, 12(10):e0186111, 2017

    2017

  3. [11]

    The critical role of peer reviewers: Challenges and future steps.Nordic Studies on Alcohol and Drugs, 40(1):14–21, 2023

    Nady El-Guebaly, John Foster, Anees Bahji, and Matilda Hellman. The critical role of peer reviewers: Challenges and future steps.Nordic Studies on Alcohol and Drugs, 40(1):14–21, 2023

    2023

  4. [12]

    Risk evaluation in peer review of grant applications.Environment Systems and Decisions, 38:216–229, 2018

    Stephen Gallo, Lisa Thompson, Karen Schmaling, and Scott Glisson. Risk evaluation in peer review of grant applications.Environment Systems and Decisions, 38:216–229, 2018

    2018

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Fundamental problems in the peer-review process and stakeholders’ perceptions of potential suggestions for improvement.Learned Publishing, 38(1):e1637, 2025

    Cigdem Kadaifci, Erkan Isikli, and Y Ilker Topcu. Fundamental problems in the peer-review process and stakeholders’ perceptions of potential suggestions for improvement.Learned Publishing, 38(1):e1637, 2025. 10

    2025

  7. [15]

    A dataset of peer reviews (peerread): Collection, insights and nlp applications.arXiv preprint arXiv:1804.09635, 2018

    Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine Van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A dataset of peer reviews (peerread): Collection, insights and nlp applications.arXiv preprint arXiv:1804.09635, 2018

    2018 arXiv

  8. [16]

    On peer review–the cornerstone of scientific publication, 2015

    Ingvar Karlberg. On peer review–the cornerstone of scientific publication, 2015

    2015

  9. [17]

    Peer review in scientific publications: benefits, critiques, & a survival guide.Ejifcc, 25(3):227, 2014

    Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. Peer review in scientific publications: benefits, critiques, & a survival guide.Ejifcc, 25(3):227, 2014

    2014

  10. [18]

    Revieweval: An evaluation framework for ai-generated reviews.arXiv preprint arXiv:2502.11736, 2025

    Chhavi Kirtani, Madhav Krishan Garg, Tejash Prasad, Tanmay Singhal, Murari Mandal, and Dhruv Kumar. Revieweval: An evaluation framework for ai-generated reviews.arXiv preprint arXiv:2502.11736, 2025

    2025 arXiv

  11. [19]

    Deep reinforcement learning for multiobjective opti- mization.IEEE transactions on cybernetics, 51(6):3103–3114, 2020

    Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective opti- mization.IEEE transactions on cybernetics, 51(6):3103–3114, 2020

    2020

  12. [20]

    Can large language models provide useful feedback on research papers? a large-scale empirical analysis.NEJM AI, 1(8): AIoa2400196, 2024

    Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas V odrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis.NEJM AI, 1(8): AIoa2400196, 2024

    2024

  13. [21]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  14. [22]

    Reviewergpt? an exploratory study on using large language models for paper reviewing.arXiv preprint arXiv:2306.00622, 2023

    Ryan Liu and Nihar B Shah. Reviewergpt? an exploratory study on using large language models for paper reviewing.arXiv preprint arXiv:2306.00622, 2023

    2023 arXiv

  15. [23]

    Grobid: Combining automatic bibliographic data recognition and term extraction for scholarship publications

    Patrice Lopez. Grobid: Combining automatic bibliographic data recognition and term extraction for scholarship publications. InInternational conference on theory and practice of digital libraries, pages 473–474. Springer, 2009

    2009

  16. [24]

    Now more than ever: reflections on the state and importance of peer review.PRiMER: Peer-review reports in medical education research, 5:36, 2021

    Christopher P Morley and Sam Grammer. Now more than ever: reflections on the state and importance of peer review.PRiMER: Peer-review reports in medical education research, 5:36, 2021

    2021

  17. [25]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  18. [26]

    Use of politeness strategies in signed open peer review

    Syavash Nobarany and Kellogg S Booth. Use of politeness strategies in signed open peer review. Journal of the Association for Information Science and Technology, 66(5):1048–1064, 2015

    2015

  19. [27]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  20. [28]

    Zero: Memory optimiza- tions toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020

    2020

  21. [29]

    Perceptions of ethical problems with scientific journal peer review: an exploratory study.Science and engineering ethics, 14:305–310, 2008

    David B Resnik, Christina Gutierrez-Ford, and Shyamal Peddada. Perceptions of ethical problems with scientific journal peer review: an exploratory study.Science and engineering ethics, 14:305–310, 2008

    2008

  22. [30]

    Grobid-information extraction from scientific publications

    Laurent Romary and Patrice Lopez. Grobid-information extraction from scientific publications. ERCIM News, 100, 2015

    2015

  23. [31]

    Journal peer review: Meaning, types, and processes.Journal Peer Review: Meaning, 2023

    Abu-Ubaida Sani. Journal peer review: Meaning, types, and processes.Journal Peer Review: Meaning, 2023. 11

    2023

  24. [32]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  25. [33]

    Relationship between journal impact factor and the thorough- ness and helpfulness of peer reviews.PLOS Biology, 21(8):1–18, 08 2023

    Anna Severin, Michaela Strinzel, Matthias Egger, Tiago Barros, Alexander Sokolov, Julia Vil- strup Mouatt, and Stefan Müller. Relationship between journal impact factor and the thorough- ness and helpfulness of peer reviews.PLOS Biology, 21(8):1–18, 08 2023. doi: 10.1371/journ...

    2023 doi

  26. [34]

    Challenges, experiments, and computational solutions in peer review.Communi- cations of the ACM, 65(6):76–87, 2022

    Nihar B Shah. Challenges, experiments, and computational solutions in peer review.Communi- cations of the ACM, 65(6):76–87, 2022

    2022

  27. [35]

    Survey of different large language model architectures: Trends, benchmarks, and challenges.IEEE Access, 2024

    Minghao Shao, Abdul Basit, Ramesh Karri, and Muhammad Shafique. Survey of different large language model architectures: Trends, benchmarks, and challenges.IEEE Access, 2024

    2024

  28. [36]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  29. [37]

    Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017

    2017

  30. [38]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv. org/abs/2408.03314

    2024 arXiv

  31. [39]

    Tools used to assess the quality of peer review reports: a methodological systematic review.BMC Medical Research Methodology, 19(1):48, Mar 2019

    Cecilia Superchi, José Antonio González, Ivan Solà, Erik Cobo, Darko Hren, and Isabelle Boutron. Tools used to assess the quality of peer review reports: a methodological systematic review.BMC Medical Research Methodology, 19(1):48, Mar 2019. ISSN 1471-2288. doi: 10.1186/s1287...

    2019 doi

  32. [40]

    Development of arcadia: a tool for assessing the quality of peer-review reports in biomedical research.BMJ Open, 10(6), 2020

    Cecilia Superchi, Darko Hren, David Blanco, Roser Rius, Alessandro Recchioni, Isabelle Boutron, and José Antonio González. Development of arcadia: a tool for assessing the quality of peer-review reports in biomedical research.BMJ Open, 10(6), 2020. ISSN 2044-6055. doi: 10.1136...

    2020 doi

  33. [41]

    Mamorx: Multi-agent multi-modal scientific review generation with external knowledge

    Pawin Taechoyotin, Guanchao Wang, Tong Zeng, Bradley Sides, and Daniel Acuna. Mamorx: Multi-agent multi-modal scientific review generation with external knowledge. InNeurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024

    2024

  34. [42]

    Is novel research worth doing? evidence from peer review at 49 journals.Proceedings of the National Academy of Sciences, 119(47):e2118046119, 2022

    Misha Teplitskiy, Hao Peng, Andrea Blasco, and Karim R Lakhani. Is novel research worth doing? evidence from peer review at 49 journals.Proceedings of the National Academy of Sciences, 119(47):e2118046119, 2022

    2022

  35. [43]

    What have we learned from openreview?World Wide Web, 26(2):683–708, Mar 2023

    Gang Wang, Qi Peng, Yanfeng Zhang, and Mingyang Zhang. What have we learned from openreview?World Wide Web, 26(2):683–708, Mar 2023. ISSN 1573-1413. doi: 10.1007/ s11280-022-01109-z. URLhttps://doi.org/10.1007/s11280-022-01109-z

    2023 doi

  36. [44]

    Reviewrobot: Explainable paper review generation based on knowledge synthesis.arXiv preprint arXiv:2010.06119, 2020

    Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. Reviewrobot: Explainable paper review generation based on knowledge synthesis.arXiv preprint arXiv:2010.06119, 2020

    2010 arXiv

  37. [45]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in N...

    2022

  38. [46]

    Rude reviews are pervasive and sometimes harmful, study finds.Science, 366 (6472):1433–1433, 2019

    Christie Wilcox. Rude reviews are pervasive and sometimes harmful, study finds.Science, 366 (6472):1433–1433, 2019. 12

    2019

  39. [47]

    Can we automate scientific reviewing?Journal of Artificial Intelligence Research, 75:171–212, 2022

    Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can we automate scientific reviewing?Journal of Artificial Intelligence Research, 75:171–212, 2022

    2022

  40. [48]

    Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2019

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  41. [49]

    Benchmarking large language models for news summarization.Transactions of the Association for Computational Linguistics, 12:39–57, 2024

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization.Transactions of the Association for Computational Linguistics, 12:39–57, 2024

    2024

  42. [50]

    A novel multi-objective deep q-network: Addressing immediate and delayed rewards in multi-objective q-learning.IEEE Access, 12:144932–144949, 2024

    Youming Zhang. A novel multi-objective deep q-network: Addressing immediate and delayed rewards in multi-objective q-learning.IEEE Access, 12:144932–144949, 2024

    2024

  43. [51]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  44. [52]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024 arXiv

  45. [53]

    Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks

    Ruiyang Zhou, Lu Chen, and Kai Yu. Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks. InProceedings of the 2024 Joint International Con- ference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), p...

    2024

  46. [54]

    relevance METEOR

    Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. Large lan- guage models for automated scholarly paper review: A survey.arXiv preprint arXiv:2501.10326, 2025. 13 A Metric names Throughout the Appendix, we will use the following short names for each m...

    2025 arXiv

  47. [55]

    For any interpretability technique, passing the sanity check is a must, but just because a saliency technique passes the sanity checks, it doesn’t mean that these maps explain the network’s decision well. 2. Lack of any quantitative evaluation (such as localization or pointing...

    2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.