{"id":"71b48007-6b89-4180-9bd3-8ce5063647a4","arxiv_id":"2505.11718","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A reasoning LLM trained with GRPO on a multi-aspect reward (REMOR) produces peer reviews that score higher on that same reward than human reviews, but the evaluation is circular and partly in-sample.","lead":"The paper trains a 7-billion-parameter reasoning model with reinforcement learning to generate peer reviews, then reports that it scores more than twice as high as human reviews on a custom review-quality metric. The catch is that the same metric was used as the training reward, so the headline gain is partly built into the setup.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HPRR is both the training objective and the evaluation yardstick; the paper's own Section 4.7 shows the human-aligned reward winner is qualitatively worse, so '>2x rewards' does not establish better reviews.","rationale":"The reader's weakest assumption—HPRR validity—is also my primary concern. It is the most load-bearing because every quantitative comparison in Figures 2 and 4 uses HPRR as the yardstick, and the central 'more than twice' statement is only an interesting result if HPRR approximates human judgment. The strongest evidence that it does not is internal: Section 4.7 and Appendix Table 6 show the highest-HPRR-H model, REMOR-H, is judged by the authors themselves as qualitatively worse than REMOR-U. That is a direct failure of the reward's 'human-aligned' label, not merely a disagreement with an external standard. The tiny 130-sample preference fit (F1=0.57) and the 8.67/9 METEOR weight make this failure mechanistically plausible. The train/evaluation overlap on ICLR is an additional confound, but it is secondary: fixing the overlap would not fix the metric-validity problem. I therefore concur with the reader's REJECT; no verdict change is needed.","tokens_in":15724,"tokens_out":5533,"duration_ms":56478,"concrete_test":"Run a blinded expert evaluation on the held-out ACL/NeurIPS set: sample 40 reviews from each of REMOR-U, REMOR-H, human reviewers, and a strong baseline (e.g., MAMORX); have at least three expert reviewers rate each review on a validated instrument (ARCadia or the eight HPRR aspects plus overall helpfulness) without knowing the source. Compute (i) mean expert ratings and pairwise preference win rates between REMOR and human reviews, and (ii) Spearman correlation between expert 'overall helpfulness' and the HPRR score for the same review. If REMOR does not significantly outperform humans on expert ratings, or if HPRR correlates weakly (<0.5) with expert ratings, then the 'more than twice' result is an artifact of optimizing the evaluation metric and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that REMOR-U and REMOR-H produce reviews more than twice as good as human reviews—rests on HPRR's validity as a proxy for human-judged quality. That premise is contradicted inside the paper. HPRR is the objective optimized by GRPO (Section 3.2), and the same reward is used for evaluation. More importantly, Appendix Table 3 assigns weight 8.67 of 9 to METEOR relevance and 0.01–0.16 to all substantive aspects in the human-aligned reward; optimizing this objective can be satisfied by long, manuscript-paraphrasing text. Section 4.7 reports exactly this: REMOR-H, the model with the highest human-aligned reward (0.670 vs REMOR-U's 0.470 on ICLR; 1.438 vs 0.731 on ACL/NeurIPS), produces reviews that are 'too long and redundant,' and the authors judge REMOR-U qualitatively better. Thus the model with lower HPRR-H is judged better by the authors, so HPRR-H does not track the quality it claims to align with. The human-preference weight fitting also uses only 130 samples with cross-validated F1 = 0.57 (Table 3), and the main ICLR evaluation overlaps the source of training data. These issues are compounded by Section 6's admission that no blinded human evaluation was performed. The literal claim about HPRR scores may hold, but the implied claim about review quality does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents REMOR, a pipeline for automated peer-review generation that combines supervised fine-tuning of DeepSeek-R1-Distill-Qwen-7B on a new PeerRT dataset (ICLR 2017-2020 reviews augmented with synthetic reasoning traces) with GRPO reinforcement learning against a newly proposed HPRR reward function. Two model variants are trained: REMOR-U with uniform reward weights and REMOR-H with weights fitted to human preference judgments. The paper reports that both variants achieve more than twice the average reward of human reviews and of several baseline systems, with lower variance, and it offers a qualitative comparison as well as released code, datasets, and models.","tokens_in":15970,"tokens_out":6589,"duration_ms":69192,"significance":"If the empirical claims were supported, the paper would be a significant contribution: it would show that a 7B reasoning model fine-tuned against a well-designed reward can generate peer-review reports comparable to the best human reviews while running in about a minute, and the released PeerRT dataset and HPRR function would support further research. The paper is transparent about its limitations and ships reproducible artifacts. However, the central claim is conditional on HPRR being a valid proxy for human-judged review quality, and that validity is not established; the paper's own qualitative analysis suggests the opposite. Consequently, the headline 'more than twice the average rewards' claim is not currently supported as a statement about review quality.","major_comments":[{"comment":"The central quantitative claim is circular. HPRR is the reward optimized by GRPO in §3.2, and the same function is then used as the evaluation metric in Fig. 2 and Table 4; the 'more than twice the average rewards' result is therefore largely a training-curve statement. The comparison is further weakened because Table 4 scores the same 5.5k ICLR papers from which the 90th-percentile training samples in §4.1 were selected, so the human-review baseline and the test set overlap the training set. An independent evaluation would require a held-out set and a measure that was not part of the training objective.","section":"§3.2, Fig. 2, Table 4"},{"comment":"The human-aligned reward does not behave like a proxy for review quality even by the authors' own analysis. Appendix Table 3 assigns weight 8.67 of 9 to METEOR relevance and 0.01–0.16 to all substantive aspects; optimizing this axis rewards long manuscript-paraphrasing text. Section 4.7 reports exactly this failure mode: REMOR-H reviews are 'too long and redundant,' and the authors judge REMOR-U as qualitatively better even though REMOR-H has the higher human-aligned reward (0.670 vs 0.470 on ICLR; 1.438 vs 0.731 on ACL/NeurIPS). Thus the evaluation axis on which REMOR-H is declared the winner contradicts the qualitative judgment, which undermines the claim that HPRR-H measures human-aligned quality.","section":"Appendix Table 3, §4.2, §4.7"},{"comment":"The 'human-aligned' weights are fitted to only 130 human preference votes and reach cross-validated F1 = 0.57 on a three-outcome prediction task, which is modest predictive performance; these same fitted weights then define the human-aligned evaluation axis used to declare REMOR-H the winner. With such a small fit, the paper's central premise that HPRR is aligned with human judgments is not established. The weight fitting also appears to drive the outcome: selecting the model that maximizes the fitted objective and then evaluating on that same objective is not an independent validation of human alignment.","section":"§3.2.1, Table 3"},{"comment":"The paper explicitly concedes that the evaluation is not based on direct human feedback but on a constructed reward function, and it defers a blinded human evaluation to future work. Since the abstract's 'more than twice the average rewards' claim is intended to support the conclusion that REMOR produces higher-quality reviews than humans, the absence of any human validation is a load-bearing gap: no amount of optimization against a synthetic reward can, by itself, establish that claim. This gap cannot be closed by local edits; it requires a new evaluation study.","section":"§6"}],"minor_comments":[{"comment":"The word 'Bradly' in §4.2 is a typo and should be 'Bradley'.","section":"§4.2"},{"comment":"The text refers to 'hard,' 'moderate,' and 'easy' difficulty groups, but the figure contains no group boundaries or labels; please add them.","section":"Fig. 3"},{"comment":"Reporting only that all SEM values were within 0.001 is not informative; report the per-model SEM or confidence intervals for each metric.","section":"§4.3"},{"comment":"Appendix Table 1 includes unadjusted weights with negative values; the text should explain why the ABT linear regression variant is preferred over the other models beyond the reported F1 score.","section":"Appendix Table 1"}],"recommendation":"reject","confidential_remarks":"To the editor: the manuscript's central claim rests on an evaluation metric that is also the training objective, and the paper's own Section 4.7 and Section 6 undercut the metric's validity. A major revision would need a blinded expert evaluation and a non-circular benchmark; without those, I do not think the claim can be supported. I recommend rejection, though I note the released artifacts are potentially useful for future work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid engineering paper with an honest limitations section, but the central claim outruns the evidence. The new artifacts — PeerRT (ICLR reviews with reasoning traces and per-sentence quality scores from Severin's taxonomy), the HPRR reward function, and the GRPO/LoRA pipeline on a 7B reasoning model — are real contributions. The held-out comparison on ACL/NeurIPS is the right kind of generalization test, and the result that REMOR-U gets high uniform reward with low variance is genuinely interesting.\n\nThe problem is that HPRR is the objective GRPO optimizes and also the yardstick. So \"more than twice the average rewards\" is a training curve, not an independent quality measure. The paper's own Section 4.7 shows REMOR-H, the model with the best human-aligned reward, producing reviews the authors judge too long and redundant, and they call REMOR-U qualitatively better. That is direct evidence that HPRR-H does not track what the authors themselves consider good review quality. The human-aligned weights come from 130 preference votes with cross-validated F1 = 0.57, and eight of nine weight mass sits on METEOR relevance, which rewards long paraphrases. So the \"human-aligned\" axis is close to a length/relevance proxy.\n\nThe main ICLR evaluation also overlaps the training selection pool: the same 5.5k papers, with top-90th-percentile samples used for SFT/GRPO. That inflates the in-house numbers. The limitation section concedes the absence of blinded human evaluation, which is exactly the missing piece. Code and model links are promised but no working URL appears in the text, so independent replication is not currently possible.\n\nWhat is actually established: a small reasoning model can be trained with GRPO to score very high on a particular reward function, and it avoids the low-quality tail of human reviews on that metric. What is not established: that those reviews are better than human reviews. The authors mostly say this in Section 6, but the abstract and introduction state it more strongly, and the mismatch is load-bearing rather than cosmetic.\n\nFor a revision: retrain or at least re-evaluate on a disjoint paper split, report HPRR's correlation with expert human judgments on a sample, and temper the abstract. The PeerRT dataset and HPRR are useful for people working on AI-assisted review, so this deserves a serious referee — but not acceptance in this form.","headline":"Useful dataset and training recipe, but the headline '>2x better than humans' conflates optimizing the reward with measuring review quality; the paper deserves engagement, not acceptance as-is.","tokens_in":16579,"tokens_out":1753,"would_cite":true,"duration_ms":16818,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a 7B reasoning model trained with multi-objective reinforcement learning can generate peer reviews that score more than twice as high as human reviews and existing review-generation systems on a human-aligned reward…","keywords":["AI peer review generation","multi-objective reinforcement learning","GRPO","reasoning language models","review quality reward","PeerRT dataset","HPRR","automated review evaluation"],"falsifier":"Compute HPRR on a held-out set of real human reviews that have independent expert quality ratings; if HPRR correlates weakly or negatively with those ratings, the reported two-to-one advantage over humans does not establish better reviews. A blinded randomized comparison in which domain experts rate REMOR and human reviews for usefulness, actionability, and correctness would settle the question directly.","tokens_in":15421,"feed_emoji":"📝","tokens_out":8246,"duration_ms":80691,"temperature":0.7,"pith_summary":"The paper sets out to show that an open 7B reasoning language model can be trained to write scientific peer reviews that score far above human reviews on an automated review-quality metric. The authors build a reward function, HPRR, from eight sentence-level aspects of review quality plus a METEOR relevance score, then supervise-fine-tune a reasoning model on a new corpus of ICLR reviews augmented with thinking traces, and finally apply GRPO reinforcement learning under two weightings. They report that the two resulting models, REMOR-U and REMOR-H, achieve more than twice the average rewards of human reviewers, non-reasoning agentic review systems, and commercial LLM baselines, and that the advantage comes mainly from avoiding the long tail of low-quality human reviews. The authors state the key caveat themselves: the evaluation is based on the constructed reward function, not on blinded human judgment, so the headline result stands or falls with that function.","feed_headline":"RL-trained 7B reasoning model beats human review scores 2-to-1","feed_subtitle":"Trained on a multi-aspect review-quality reward, the models also outscore agentic baselines and cut the low-quality tail.","key_machinery":"The load-bearing object is the Human-aligned Peer Review Reward (HPRR): a scalar score formed as a weighted sum of eight sentence-level review-quality aspects, normalized per sentence, plus a METEOR relevance score between the generated review and the manuscript text. HPRR does double duty: it is the reward signal used during GRPO training and the metric used to compare models against humans and baselines. The weights for the human-aligned variant are estimated from pairwise human votes with a constrained Bradley-Terry model, and the optimization itself is carried out with Group Relative Policy Optimization on a 7B reasoning model that was first supervised-fine-tuned on PeerRT, a corpus of ICLR reviews augmented with thinking traces.","core_discovery":"On its own terms, the paper's central claim is that a 7B reasoning model can be turned into a peer-review generator whose outputs score more than twice as high as human-written reviews on a purpose-built quality metric. The authors construct HPRR, a weighted sum of eight sentence-level aspects (criticism, example, importance and relevance, materials and methods, praise, presentation and reporting, results and discussion, suggestion and solution) plus a METEOR relevance score between the review and the manuscript. They fine-tune DeepSeek-R1-Distill-Qwen-7B on PeerRT, a corpus of ICLR reviews augmented with synthetic reasoning traces, then apply GRPO under either uniform weights (REMOR-U) or weights derived from human pairwise preferences (REMOR-H). They report that both models exceed the average rewards of human reviewers, agentic review systems, and commercial LLMs by a factor of two, and that the advantage comes from eliminating the long tail of low-quality human reviews rather than surpassing the best human reviews. The authors also report that the uniform model produces qualitatively more substantive feedback, while the human-aligned model's reviews become long and redundant, an effect they attribute to the weight placed on the METEOR relevance component.","pith_inferences":["The human-aligned weights are estimated from only about 130 pairwise votes; a larger preference dataset could shift which review style is rewarded, so the specific REMOR-H results should be read as tied to that sample.","Because the METEOR relevance component rewards overlap with the manuscript, REMOR-H's long and redundant reviews suggest that HPRR may need a length penalty or a saturating cap before it is used as a general evaluation metric.","That human-vote-derived weights down-weight criticism suggests preference-based rewards may capture politeness norms rather than critical value; a system aimed at scientific rigor might want to invert those weights.","If HPRR tracks expert judgment, the same supervised-fine-tuning-plus-GRPO recipe could transfer to other expert writing tasks with multi-dimensional rubrics, such as grant reviews or clinical assessments."],"forward_implications":["A peer-review assistant built this way could run in about a minute, where the agentic baselines it beats take roughly 20 to 30 minutes.","Since the largest gain is against the low-quality tail of human reviews, such a model could raise the floor of review quality rather than just matching the best human reviewers.","HPRR itself can be reused as a self-assessment score for human reviewers, giving a quick automated signal of which review aspects a draft covers.","A reasoning model optimized with GRPO can improve several review aspects at once without needing multi-agent coordination, figure understanding, or external knowledge retrieval.","The choice of reward weighting matters for style: human-vote-derived weights favor suggestions and relevance but discourage criticism, so the weighting is a policy choice, not just a technical detail."],"supporting_citations":[{"why":"Supplies the sentence-level review aspects (criticism, example, importance, etc.) that HPRR scores.","marker":"[33]"},{"why":"Provides the human pairwise preference votes used to estimate the human-aligned reward weights, and the MAMORX system used as a benchmark.","marker":"[41]"},{"why":"Supplies METEOR, the relevance component of the HPRR reward.","marker":"[1]"},{"why":"Supplies DeepSeek-R1-Distill-Qwen-7B, the base reasoning model that is fine-tuned and then optimized with GRPO.","marker":"[13]"},{"why":"Introduces GRPO, the reinforcement-learning algorithm used to optimize the review policy from reward values alone.","marker":"[36]"},{"why":"Provides the Bradley-Terry paired-comparison model adapted to derive human-aligned weights from votes.","marker":"[3]"},{"why":"Supplies the ICLR review corpus that underlies the PeerRT dataset.","marker":"[15]"},{"why":"Provides the MARG-S multi-agent review generation system used as a non-reasoning baseline in the comparison.","marker":"[9]"}],"fun_headline_variants":["7B RL reasoning model doubles human review scores","RL-trained 7B outscores human reviewers 2:1","7B LLM doubles peer-review rewards vs humans","7B RL model: 2x human review scores, cuts low-quality tail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that HPRR, the paper's automated review-quality score, actually tracks what expert humans mean by a useful review, since the model is trained to maximize it and then judged by it.","fun_headline_variants_meta":{"raw":{"variants":["7B RL reasoning model doubles human review scores","RL-trained 7B outscores human reviewers 2:1","7B LLM doubles peer-review rewards vs humans","7B RL model: 2x human review scores, cuts low-quality tail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001367,"raw_usage":{"total_tokens":5625,"prompt_tokens":1110,"completion_tokens":4515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":4444}},"tokens_in":726,"tokens_out":4515,"duration_ms":31817,"temperature":1.0,"reasoning_tokens":4444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:49:13.179664+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute HPRR on a held-out set of real human reviews that have independent expert quality ratings; if HPRR correlates weakly or negatively with those ratings, the reported two-to-one advantage over humans does not establish better reviews. A blinded randomized comparison in which domain experts rate REMOR and human reviews for usefulness, actionability, and correctness would settle the question directly.","supporting_citations":[{"cited_title":"Mamorx: Multi-agent multi-modal scientific review generation with external knowledge","cited_arxiv_id":null,"evidence_quote":"Provides the human pairwise preference votes used to estimate the human-aligned reward weights, and the MAMORX system used as a benchmark."}],"review_version":1}