Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

ReviewRL: Towards Automated Scientific Review with RL

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ReviewRL claims that reinforcement learning, layered on retrieval and supervised warm-up, produces automated scientific reviews with more accurate ratings and deeper, more factual critique than prior LLM reviewers.

desk verdict A credible rating-consistency result and a clean ablation story, but the quality-difference claims are only as strong as the LLM judge that shares criteria with the training reward. read the letter →

arxiv 2508.10308 v1 pith:GZMISO6P submitted 2025-08-14 cs.CL cs.AI

Sihang Zeng , Kai Tian , Kaiyan Zhang , Yuru wang , Junqi Gao , Runze Liu , Sa Yang , Jingxuan Li
show 4 more authors
Xinwei Long Jiaheng Ma Biqing Qi Bowen Zhou
This is my paper
classification cs.CLcs.AI
keywords automatedpeerreviewreinforcementlearningretrieval-augmentedgenerationratingconsistencygenerativerewardmodelLLM-as-a-judgescientificcritiqueICLR2025
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReviewRL claims that the quality of automatically generated peer reviews is limited less by the underlying language model than by how it is trained, and that a three-stage recipe—retrieval of related literature, supervised fine-tuning on long chain-of-thought reviews, then reinforcement learning with a composite reward—fixes the main weaknesses. The composite reward pairs a rule-based term for rating accuracy with a generative judge model that decides which of two reviews is better, which the authors say is necessary because pure rule rewards produce generic reviews. On 472 ICLR 2025 papers, ReviewRL reports the best rating alignment among the compared systems (MSE 2.585, Spearman 0.634) and the highest scores on all seven LLM-judged quality dimensions, with the largest margins in analytical depth and factual correctness. If correct, this is a concrete recipe for applying RL to a non-verifiable writing task where quality must be judged rather than computed. The practical stakes are high because conference review loads are growing, but the same reliance on judge models is what a skeptical reader should watch.

What carries the argument

The load-bearing machinery is the composite reward $R_{\mathrm{final}} = \gamma R_{\mathrm{rule}} + (1-\gamma)R_{\mathrm{judge}}$ (with $\gamma=0.5$), where $R_{\mathrm{rule}}$ combines a Gaussian rating-consistency reward $R_{rc}=\exp(-(s-\hat{s})^2/2\sigma^2)$ with a format penalty, and $R_{\mathrm{judge}}$ is a GenRM—a judge model that reads the paper and a reference review and outputs which review is better on six criteria. Around this reward sits the pipeline: Qwen3-8B generates three natural-language queries, ArXiv-MCP (a Model Context Protocol server for arXiv search) retrieves related papers, and Qwen-Agent consolidates the context; the policy is SFT-warmed on DeepReview-13k and trai

What would settle it

Collect blind human pairwise comparisons between ReviewRL and DeepReviewer reviews on a random sample of the 472 ICLR 2025 papers. If humans do not prefer ReviewRL's reviews at a rate comparable to the Llama-3.3-70B judge's reported margins (e.g., analytical depth 4.18 vs 3.80), the model-based quality claim fails. A second decisive check: replace the GenRM judge's preferences with human preferences in training; if following human reward does not reproduce the reported MSE and quality gains, the composite reward is capturing judge style rather than review quality.

Watch

Extended reading notes

Core claim

The paper's discovery is that an RL-trained reviewer, initialized by SFT and fed retrieved arXiv context, can simultaneously improve numerical rating consistency and judged review quality. Using Qwen2.5-7B-Instruct as the base, the authors report MSE against averaged human scores falling from 2.829 (SFT only) to 2.585 after RL, and Spearman correlation rising from 0.335 to 0.634; the same model also outscores previous SFT-based reviewers like DeepReviewer on all evaluator dimensions (e.g., analytical depth 4.18 vs 3.80). Ablations identify two load-bearing pieces: the retrieval context, whose removal degrades factual-correctness scores, and the GenRM reward, without which RL gains plateau ne

Load-bearing premise

The whole quality story rests on LLM judges: a Qwen2.5-14B judge supplies the training reward and a Llama-3.3-70B judge measures the outcome, so if these judges prefer longer or more fluent reviews rather than more accurate ones, the reported quality gains are partly an artifact of the judge.

Editorial extensions

If this is right

  • RL with a judge-based composite reward can be applied to non-verifiable generation tasks beyond review, such as scientific critique, legal analysis, or clinical commentary, whenever a reference output and rating anchor exist.
  • Retrieval-augmented reviewers should be expected to hallucinate less and to ground critiques in related work, making the factual-accuracy dimension of automated review more trustworthy.
  • Because SFT cold-start prevents rating collapse, any RL recipe for judgment-heavy text generation should keep an SFT warm-up and a rating-distribution balancing step.
  • Rule-only reward in non-verifiable tasks is insufficient; the paper's ablation implies judge-model rewards are required for analytical depth and actionable feedback.
  • Rating alignment with averaged human scores improves with RL, so automated triage systems that rank submissions could inherit this alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because both the training reward and the evaluation metrics come from LLM judges, the same recipe could overfit to judge preferences; a natural test is to retrain with human pairwise preferences as the reward and see whether the Llama-3.3-70B-quality scores and MSE improve or degrade.
  • Editorial extension: the Gaussian rating reward treats the averaged human score as a single point, ignoring reviewer disagreement; rewarding calibrated rating distributions instead might generalize better to papers where human reviewers genuinely split.
  • Editorial extension: the retrieval pipeline only searches arXiv, so papers in fields with weak arXiv coverage would need a different corpus; the reported hallucination reductions may not transfer to those settings.
  • Editorial extension: if RL-trained reviewers are used to triage submissions, the improved rating alignment could systematically shift acceptance decisions, so the ethics section's 'support, not replace' framing deserves an empirical study of decision impact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ReviewRL, an RL-based framework for automated scientific paper review. The pipeline combines (1) an ArXiv-MCP retrieval-augmented context generation module, (2) supervised fine-tuning on long-CoT review data, and (3) a reinforcement learning stage using Reinforce++ with a composite reward that mixes a rule-based rating-consistency reward, a format penalty, and a generative reward model (GenRM) that compares the generated review against a reference review. Experiments are conducted on a balanced sample of 472 ICLR 2025 papers, with rule-based metrics (MSE, Spearman, pairwise ranking metrics, concordance) and LLM-as-a-judge quality scores across seven dimensions. The paper reports that ReviewRL outperforms open-source instruct/reasoning models and SFT-trained baselines on both metric families, and presents ablations showing contributions from RL, retrieval, and the GenRM reward.

Significance. If the rating-consistency result holds, this is a useful contribution: it demonstrates that RL with a composite reward can improve numerical score alignment with averaged human ratings, and the cold-start mitigation via SFT and data balancing is a practical insight for non-verifiable generation tasks. The paper also provides a clean ablation structure for retrieval, RL, and reward components, and releases training-dynamics plots. However, the subjective quality claim is currently weakened by the overlap between the GenRM reward used during training and the LLM judge used for evaluation; the rule-based metrics, being anchored to external human scores, are the more credible evidence and should carry the main claim.

major comments (4)
  1. [§4.2.2 vs §3.4] The evaluation judge and the training reward use substantially overlapping criteria. The GenRM prompt (Table 10) asks for factual accuracy, completeness, detail, comparison with existing work, constructiveness, and clarity; the ReviewEval judge in §4.2.2 scores topic coverage, semantic similarity, correctness, absence of hallucinations, analytical depth, actionability, and adherence. The policy is optimized against the GenRM, so evaluating it with a judge that scores nearly the same dimensions risks rewarding exactly the behavior RL was trained to produce. The 'Semantic Similarity' dimension is especially problematic because it rewards agreement with a plausible human review, while RL training compares against reference reviews. The claim in §5.2 that ReviewRL 'consistently outperforms all baselines' on quality is not established without a held-out judge with non-overlapping criteria or
  2. [§5.1, Table 2] The rule-based results are reported as point estimates without confidence intervals, standard deviations, or significance tests. The abstract and §5.1 use 'significantly outperforms,' but no statistical support is provided. For instance, the MSE gap between ReviewRL (2.585) and its SFT-only counterpart (2.829) may be real, but without multiple runs or bootstrap CIs over the 472-paper sample, it is impossible to assess whether the differences are noise. Please report at least three independent runs per condition or bootstrap confidence intervals, and perform a paired test (e.g., Wilcoxon) for MSE/Spearman.
  3. [§4.1] The evaluation set is deliberately rebalanced to have an approximately uniform distribution of average human ratings. This is a reasonable design choice for coverage, but it means the reported Spearman and MSE values are computed on a uniform distribution, which is not directly comparable to naturalistic-distribution results in prior work. The relative comparison among models on the same balanced sample is fair, but the absolute numbers should not be quoted without this caveat. Please also report results on the un-rebalanced ICLR 2025 sample, or explicitly state that the balanced set is the only evaluation and interpret numbers accordingly.
  4. [§5.3, Figure 4] The cold-start analysis is qualitative. The text claims that data balancing 'partially mitigates' the rating-collapse issue, but no numeric evidence is given for the w/o SFT and w/o SFT+balancing conditions. Provide quantitative metrics such as rating variance, MSE, or distributional divergence for these conditions to support the claim, not just a density plot.
minor comments (4)
  1. [Title/Abstract area] Typo: 'a ArXiv-MCP' should be 'an ArXiv-MCP'.
  2. [§3.3] The phrase 'the best mode' in DeepReview-13k is undefined. Please clarify what 'best mode' refers to in that dataset or provide a citation to a definition.
  3. [Appendix C.3] The text uses 'REVIEW RL' in uppercase, inconsistent with the paper's 'ReviewRL' notation.
  4. [§5.4] The retrieval effectiveness evaluation mentions 'three independent LLM judges' but does not specify which models were used. For reproducibility, name the judge models or state that they are the same as the GenRM/evaluation judges.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: rule-based rating metrics are externally anchored to human scores; model-based quality evaluation, while sharing criteria with the GenRM reward, uses a different judge model and is not definitionally forced.

full rationale

ReviewRL's central derivations are not circular. The rating-consistency path is anchored to averaged human ratings: the reward Rrc = exp(−(s−ŝ)^2/2σ^2) is trained on data excluding ICLR 2025 and evaluated on a held-out ICLR 2025 sample using the same human-average ground-truth; this is a standard held-out evaluation, not a fitted-input prediction. The MSE/Spearman improvements are a direct consequence of optimizing rating consistency, but they are measured on unseen papers, so no reduction-by-construction occurs. The model-based quality evaluation (Llama-3.3-70B on ReviewEval dimensions) does share several criteria with the frozen Qwen2.5-14B GenRM reward used during RL (e.g., factual accuracy, completeness/detail, constructiveness, clarity), which raises a legitimate validity concern about judge-criterion overlap or reward hacking. However, this is not a circular derivation: the evaluation judge is a different model, and the reported scores are not computed from the reward function by any equation. The only self-citations (e.g., Qi et al., 2024 for LLM analysis capabilities) are contextual and not load-bearing. The Limitations section also acknowledges that the seven evaluation dimensions are not exhaustive, which is a coverage limitation rather than a circular step. Under the stated rules, the paper exhibits no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the validity of LLM judge evaluations, the reliability of averaged human ratings as ground truth, and a fair evaluation setup. These assumptions are not independently validated in the paper.

free parameters (5)
  • α (rating-consistency weight)
    Coefficient in Rrule = clip(α·Rrc + β·Rf, 0, 1); balances rating accuracy against format adherence; not specified in the paper.
  • β (format-penalty weight)
    Coefficient for missing-format penalties in the rule-based reward; not reported.
  • σ (Gaussian kernel width)
    Width of the Gaussian in Rrc = exp(-(s - ŝ)^2/(2σ^2)); controls how sharply rating error is punished; not reported.
  • γ (composite reward weight) = 0.5
    Reported in Appendix C.1: Rfinal = γ·Rrule + (1-γ)·Rjudge. Chosen by hand, not swept.
  • Data balancing ratios (down/upsampling)
    Papers with mid-range ratings are downsampled and extreme-ratings papers upsampled; the exact ratios are not quantified.
assumptions (5)
  • domain assumption LLM-as-a-judge produces a valid measure of review quality
    Both the GenRM reward (§3.4) and the ReviewEval evaluation (§4.2.2) rely on judge models without validation against human judgments on this task.
  • domain assumption Averaged human reviewer scores are a reliable ground truth for paper quality
    Rating rewards and metrics (MSE, Spearman) treat the mean of multiple human ratings as the target, despite known noise and inter-rater disagreement.
  • domain assumption ArXiv-MCP retrieval returns sufficient relevant context
    The retrieval module (§3.2) is assumed to provide the related-work context needed for factual, comparative reviews; only judge-based win rates are reported, not downstream citation precision.
  • domain assumption Reference reviews (meta-reviews and summarized human reviews) are appropriate comparators for GenRM preference
    The GenRM reward compares against r derived from DeepReview-13k meta-reviews and DeepSeek-R1 summarization of Reviewer2 reviews; the quality of these references is taken as given.
  • domain assumption Uniform sampling of the evaluation set preserves the validity of the reported metrics
    §4.1 rebalances the ICLR 2025 sample to be approximately uniform across the rating scale, altering the distribution that real-world deployment would see; metric values depend on this distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReviewRL: Towards Automated Scientific Review with RL." pith.science (2026). https://pith.science/paper/GZMISO6P

@misc{pith2026250810308,
  author       = {Pith},
  title        = {Pith review of: ReviewRL: Towards Automated Scientific Review with RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZMISO6P}},
  note         = {Machine review of arXiv:2508.10308}
}
read the original abstract

Peer review is essential for scientific progress but faces growing challenges due to increasing submission volumes and reviewer fatigue. Existing automated review approaches struggle with factual accuracy, rating consistency, and analytical depth, often generating superficial or generic feedback lacking the insights characteristic of high-quality human reviews. We introduce ReviewRL, a reinforcement learning framework for generating comprehensive and factually grounded scientific paper reviews. Our approach combines: (1) an ArXiv-MCP retrieval-augmented context generation pipeline that incorporates relevant scientific literature, (2) supervised fine-tuning that establishes foundational reviewing capabilities, and (3) a reinforcement learning procedure with a composite reward function that jointly enhances review quality and rating accuracy. Experiments on ICLR 2025 papers demonstrate that ReviewRL significantly outperforms existing methods across both rule-based metrics and model-based quality assessments. ReviewRL establishes a foundational framework for RL-driven automatic critique generation in scientific discovery, demonstrating promising potential for future development in this domain. The implementation of ReviewRL will be released at GitHub.

Figures

Figures reproduced from arXiv: 2508.10308 by the authors.

Figure 1
Figure 1. Evaluation results of ReviewRL under the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of ReviewRL, including Arxiv-MCP, SFT warm up, and RL optimization. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training Dynamics of RL 0 5 10 Rating 0.0 0.5 1.0 1.5 2.0 Density Methods Ground Truth w/o SFT and Balancing w/o SFT ReviewRL [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Win/Loss percentages of the retrieval answer [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions

    cs.CL 2026-06 conditional novelty 7.0 of 10

    Presentation-only revisions guided by AI feedback can boost AI reviewer scores by over 1 point on average with 75% success rate across tested systems.

  2. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  3. LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    LLMs overrate weak papers, diverge from humans on criteria like clarity and reproducibility, write longer less diverse reviews, and remain vulnerable to prompt injection attacks that can boost low-scoring papers to ac...

Reference graph

Works this paper leans on

14 extracted references · 7 canonical work pages · cited by 3 Pith papers

  1. [1]

    Factual Accuracy & Soundness: Does the review accurately understand the paper’s contributions and limitations? Is the critique based on sound reasoning?

    work page

  2. [2]

    Completeness & Coverage: Does the review address the core aspects of the paper (e.g., methodology, results, signifi- cance)?

    work page

  3. [3]

    Level of Detail & Specificity: Does the review provide specific examples and detailed comments rather than vague statements?

    work page

  4. [4]

    Comparison with Existing Work: Does the review appropriately contextualize the paper within the existing literature and compare it to relevant methods?

    work page

  5. [5]

    5: Scaling reinforcement learning with llms

    Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. Qwen Team. 2024. Qwen2.5: A party of foundation models. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elic- its reasoning in large language models. Advances in neural inform...

    arXiv 2024

  6. [6]

    arXiv preprint arXiv:2411.00816

    Cycleresearcher: Improving automated research via automated review. arXiv preprint arXiv:2411.00816. Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. 2025. J1: Incentivizing thinking in llm-as- a-judge via reinforcement learning. arXiv preprint arXiv:2505.10320. Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Li...

    arXiv 2025

  7. [7]

    arXiv preprint arXiv:2505.09388

    Qwen3 technical report. arXiv preprint arXiv:2505.09388. Sungduk Yu, Man Luo, Avinash Madusu, Vasudev Lal, and Phillip Howard. 2025. Is your paper being re- viewed by an llm? a new benchmark dataset and approach for detecting ai text in peer review. arXiv preprint arXiv:2502.19614. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, ...

    arXiv 2025

  8. [8]

    arXiv preprint arXiv:2503.08569

    Deepreview: Improving llm-based paper re- view with human-like deep thinking process. arXiv preprint arXiv:2503.08569. 10 A Prompts The prompts for both the Generation, Evaluation, and GenRM are presented in Tables 6, 7, 8, 9 and 10. B Pairwise Metrics Prelation =    1.0, if sgn(s1 − s2) = sgn(s∗ 1 − s∗ 2), 0, otherwise. (6) Pabsolute =    1.0, ...

    arXiv

Show all 14 references
  1. [13]

    Constructiveness: Is the feedback helpful for the authors to improve the paper? Is the tone professional and constructive?

  2. [14]

    A TIE IS NOT ALLOWED

    Clarity & Organization: Is the review well-structured and easy to understand? Paper Context (Abstract/Content): {paper_context} Review 1: {review1} Review 2: {review2} Which peer review is of higher quality based on the criteria above? Respond with EXACTLYone of these options:...

  3. [2022]

    arXiv preprint arXiv:2211.06651

    Nlpeer: A unified resource for the com- putational study of peer review. arXiv preprint arXiv:2211.06651. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims

    arXiv

  4. [2023]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    Direct preference optimization: Your lan- guage model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728– 53741. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. ...

    2017 arXiv

  5. [2024]

    arXiv preprint arXiv:2402.10886

    Reviewer2: Optimizing review genera- tion through prompt generation. arXiv preprint arXiv:2402.10886. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoni...

    2025 arXiv

  6. [2025]

    arXiv preprint arXiv:2503.01785

    Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785. 9 Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foer- ster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific dis- covery. arXiv preprint arXiv:2408.062...

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.