{"id":"f1f8e719-8b38-41af-9cad-e60942b8523b","arxiv_id":"2508.10308","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ReviewRL combines arXiv retrieval, supervised fine-tuning, and reinforcement learning with a composite reward to generate paper reviews that better match human ratings and judged quality.","lead":"This paper presents ReviewRL, a system that automatically writes peer reviews of research papers by retrieving related literature from arXiv and then training a large language model with reinforcement learning to produce better review text and more accurate scores. On ICLR 2025 papers it reports lower rating error and higher judged review quality than several existing review-generation models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quality gains may reflect reward hacking: the RL reward and the evaluation judge share criteria, and no human validation anchors the subjective quality claims.","rationale":"The reviewer's CONDITIONAL verdict is appropriate. The rule-based rating improvements are credible due to their anchoring to human scores; however, the model-based quality claims are the most load-bearing part of the paper's central claim ('strongest performance across the board'). The concern is not that the authors are dishonest, but that the evaluation methodology is circular in an important sense: the reward function and the evaluation judge share the same qualitative criteria, and no external ground truth (e.g., human quality judgments) is provided. This does not refute the paper, but it blocks the strong comparative claim. The proposed human-evaluation test would directly settle whether the quality gains are real or an artifact of judge alignment. The reader's weakest_assumption identifies the same issue, so I agree. Verdict remains CONDITIONAL pending that validation.","tokens_in":13577,"tokens_out":4099,"duration_ms":45992,"concrete_test":"Conduct a blinded human evaluation on a random sample of 50 papers from the ICLR 2025 evaluation set. Have three expert reviewers independently rate ReviewRL-7B, DeepReviewer-7B, and (if available) the original human meta-review on the seven ReviewEval dimensions, or on a single overall quality scale. Compute (a) the inter-annotator agreement (e.g., ICC or Kendall's W), (b) the correlation between human scores and the Llama-3.3-70B judge scores, and (c) the relative ranking of ReviewRL vs. DeepReviewer by human raters. If ReviewRL does not significantly outperform DeepReviewer on human ratings, or if the human–judge correlation is low (Spearman < 0.5), then the reported quality advantage is not established and the Section 5.2 claims should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quality claim (Section 5.2, Table 5) that ReviewRL 'consistently outperforms all baselines' on seven review-quality dimensions is supported only by an LLM-as-a-judge evaluation (Llama-3.3-70B) whose criteria substantially overlap with the GenRM reward used during RL training (Section 3.4). The GenRM prompt (Table 10) asks the judge to prefer reviews based on factual accuracy, completeness, detail, comparison with existing work, constructiveness, and clarity—nearly the same dimensions assessed by the evaluation judge (topic coverage, correctness, hallucinations, depth, actionability, etc.). Moreover, the evaluation includes 'Semantic Similarity,' which explicitly rewards agreement with a plausible human review; the RL policy is trained against reference reviews, so it may be optimizing for exactly the stylistic features this dimension rewards. Because no human evaluation or held-out judge with non-overlapping criteria is reported, the observed quality improvements could arise from the policy learning to exploit the judge's preferences (e.g., producing longer, more assertive, or reference-shaped text) rather than from genuinely better reviews. The rule-based rating metrics (MSE 2.585, Spearman 0.634) are anchored to external human scores and are therefore more credible, but they do not validate the subjective quality dimensions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReviewRL, an RL-based framework for automated scientific paper review. The pipeline combines (1) an ArXiv-MCP retrieval-augmented context generation module, (2) supervised fine-tuning on long-CoT review data, and (3) a reinforcement learning stage using Reinforce++ with a composite reward that mixes a rule-based rating-consistency reward, a format penalty, and a generative reward model (GenRM) that compares the generated review against a reference review. Experiments are conducted on a balanced sample of 472 ICLR 2025 papers, with rule-based metrics (MSE, Spearman, pairwise ranking metrics, concordance) and LLM-as-a-judge quality scores across seven dimensions. The paper reports that ReviewRL outperforms open-source instruct/reasoning models and SFT-trained baselines on both metric families, and presents ablations showing contributions from RL, retrieval, and the GenRM reward.","tokens_in":13804,"tokens_out":4319,"duration_ms":48747,"significance":"If the rating-consistency result holds, this is a useful contribution: it demonstrates that RL with a composite reward can improve numerical score alignment with averaged human ratings, and the cold-start mitigation via SFT and data balancing is a practical insight for non-verifiable generation tasks. The paper also provides a clean ablation structure for retrieval, RL, and reward components, and releases training-dynamics plots. However, the subjective quality claim is currently weakened by the overlap between the GenRM reward used during training and the LLM judge used for evaluation; the rule-based metrics, being anchored to external human scores, are the more credible evidence and should carry the main claim.","major_comments":[{"comment":"The evaluation judge and the training reward use substantially overlapping criteria. The GenRM prompt (Table 10) asks for factual accuracy, completeness, detail, comparison with existing work, constructiveness, and clarity; the ReviewEval judge in §4.2.2 scores topic coverage, semantic similarity, correctness, absence of hallucinations, analytical depth, actionability, and adherence. The policy is optimized against the GenRM, so evaluating it with a judge that scores nearly the same dimensions risks rewarding exactly the behavior RL was trained to produce. The 'Semantic Similarity' dimension is especially problematic because it rewards agreement with a plausible human review, while RL training compares against reference reviews. The claim in §5.2 that ReviewRL 'consistently outperforms all baselines' on quality is not established without a held-out judge with non-overlapping criteria or","section":"§4.2.2 vs §3.4"},{"comment":"The rule-based results are reported as point estimates without confidence intervals, standard deviations, or significance tests. The abstract and §5.1 use 'significantly outperforms,' but no statistical support is provided. For instance, the MSE gap between ReviewRL (2.585) and its SFT-only counterpart (2.829) may be real, but without multiple runs or bootstrap CIs over the 472-paper sample, it is impossible to assess whether the differences are noise. Please report at least three independent runs per condition or bootstrap confidence intervals, and perform a paired test (e.g., Wilcoxon) for MSE/Spearman.","section":"§5.1, Table 2"},{"comment":"The evaluation set is deliberately rebalanced to have an approximately uniform distribution of average human ratings. This is a reasonable design choice for coverage, but it means the reported Spearman and MSE values are computed on a uniform distribution, which is not directly comparable to naturalistic-distribution results in prior work. The relative comparison among models on the same balanced sample is fair, but the absolute numbers should not be quoted without this caveat. Please also report results on the un-rebalanced ICLR 2025 sample, or explicitly state that the balanced set is the only evaluation and interpret numbers accordingly.","section":"§4.1"},{"comment":"The cold-start analysis is qualitative. The text claims that data balancing 'partially mitigates' the rating-collapse issue, but no numeric evidence is given for the w/o SFT and w/o SFT+balancing conditions. Provide quantitative metrics such as rating variance, MSE, or distributional divergence for these conditions to support the claim, not just a density plot.","section":"§5.3, Figure 4"}],"minor_comments":[{"comment":"Typo: 'a ArXiv-MCP' should be 'an ArXiv-MCP'.","section":"Title/Abstract area"},{"comment":"The phrase 'the best mode' in DeepReview-13k is undefined. Please clarify what 'best mode' refers to in that dataset or provide a citation to a definition.","section":"§3.3"},{"comment":"The text uses 'REVIEW RL' in uppercase, inconsistent with the paper's 'ReviewRL' notation.","section":"Appendix C.3"},{"comment":"The retrieval effectiveness evaluation mentions 'three independent LLM judges' but does not specify which models were used. For reproducibility, name the judge models or state that they are the same as the GenRM/evaluation judges.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the circularity between the GenRM reward and the LLM-judge evaluation. If the authors add a human-annotation study on a sample of reviews (even 50–100) and/or use a judge with non-overlapping criteria (e.g., focusing on citation correctness, numerical claims, or suggestion implementability), the quality claim would become credible. The rating-consistency results are promising and, if supplemented with significance tests, could support the main contribution. The paper is within scope for cs.CL and the open-sourcing commitment is valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about ReviewRL. The rule-based rating numbers are the real contribution: MSE drops from 2.83 (SFT) to 2.59 with RL, Spearman jumps from 0.335 to 0.634 on an ICLR 2025 sample, and the ablations show each component (retrieval, RL, GenRM) earns its keep. That part is anchored to averaged human scores and is not circular. The second thing: the much-touted quality improvements over baselines (analytical depth 4.18 vs 3.80, etc.) come entirely from a Llama-3.3-70B judge whose criteria (correctness, hallucinations, depth, actionability) largely overlap with the GenRM reward used during RL. No human evaluation, no held-out judge with different criteria. So I read Section 5.2 as showing the policy learns to produce text that this kind of judge prefers, not necessarily better reviews.\n\nWhat's new: the specific combination of SFT warm-up, Reinforce++ with a composite rule+GenRM reward, and ArXiv-MCP retrieval for review generation. Each part is known, but putting them together for this task and documenting the cold-start problem is a reasonable contribution. The paper is well-written and the limitations section is honest.\n\nSoft spots beyond the judge issue. No significance tests or confidence intervals anywhere; the gains could be noise, especially the pairwise metrics. The evaluation set is deliberately rebalanced to a uniform rating distribution; that is fine for measuring discrimination but will not match real submission distributions, so the reported MSE is not a direct estimate of operational error. Also, DeepReviewer-7B was trained on DeepReview-13k, which very likely includes ICLR 2025 reviews; that is not disclosed, and while it makes ReviewRL's win more conservative rather than less, the comparison would be cleaner if the paper stated that explicitly.\n\nWho is this for? People building automated review assistants or applying RL to non-verifiable generation tasks. The framework is a useful template. I'd give it a serious referee: the rating-consistency result is solid enough to merit publication once the quality claims are backed by a human study or an evaluator with disjoint criteria, plus confidence intervals and a sensitivity check on the reward weights.","headline":"A credible rating-consistency result and a clean ablation story, but the quality-difference claims are only as strong as the LLM judge that shares criteria with the training reward.","tokens_in":14423,"tokens_out":2939,"would_cite":false,"duration_ms":30078,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReviewRL claims that reinforcement learning, layered on retrieval and supervised warm-up, produces automated scientific reviews with more accurate ratings and deeper, more factual critique than prior LLM reviewers.","keywords":["automated peer review","reinforcement learning","retrieval-augmented generation","review rating consistency","generative reward model","LLM-as-a-judge","scientific critique generation","ICLR 2025"],"falsifier":"Collect blind human pairwise comparisons between ReviewRL and DeepReviewer reviews on a random sample of the 472 ICLR 2025 papers. If humans do not prefer ReviewRL's reviews at a rate comparable to the Llama-3.3-70B judge's reported margins (e.g., analytical depth 4.18 vs 3.80), the model-based quality claim fails. A second decisive check: replace the GenRM judge's preferences with human preferences in training; if following human reward does not reproduce the reported MSE and quality gains, the composite reward is capturing judge style rather than review quality.","tokens_in":13390,"feed_emoji":"📝","tokens_out":6950,"duration_ms":67697,"temperature":0.7,"pith_summary":"ReviewRL claims that the quality of automatically generated peer reviews is limited less by the underlying language model than by how it is trained, and that a three-stage recipe—retrieval of related literature, supervised fine-tuning on long chain-of-thought reviews, then reinforcement learning with a composite reward—fixes the main weaknesses. The composite reward pairs a rule-based term for rating accuracy with a generative judge model that decides which of two reviews is better, which the authors say is necessary because pure rule rewards produce generic reviews. On 472 ICLR 2025 papers, ReviewRL reports the best rating alignment among the compared systems (MSE 2.585, Spearman 0.634) and the highest scores on all seven LLM-judged quality dimensions, with the largest margins in analytical depth and factual correctness. If correct, this is a concrete recipe for applying RL to a non-verifiable writing task where quality must be judged rather than computed. The practical stakes are high because conference review loads are growing, but the same reliance on judge models is what a skeptical reader should watch.","feed_headline":"Reinforcement learning sharpens AI paper reviews and ratings","feed_subtitle":"ReviewRL's retrieval plus composite rewards cut rating error and lift judged review depth on ICLR 2025 papers.","key_machinery":"The load-bearing machinery is the composite reward $R_{\\mathrm{final}} = \\gamma R_{\\mathrm{rule}} + (1-\\gamma)R_{\\mathrm{judge}}$ (with $\\gamma=0.5$), where $R_{\\mathrm{rule}}$ combines a Gaussian rating-consistency reward $R_{rc}=\\exp(-(s-\\hat{s})^2/2\\sigma^2)$ with a format penalty, and $R_{\\mathrm{judge}}$ is a GenRM—a judge model that reads the paper and a reference review and outputs which review is better on six criteria. Around this reward sits the pipeline: Qwen3-8B generates three natural-language queries, ArXiv-MCP (a Model Context Protocol server for arXiv search) retrieves related papers, and Qwen-Agent consolidates the context; the policy is SFT-warmed on DeepReview-13k and trai","core_discovery":"The paper's discovery is that an RL-trained reviewer, initialized by SFT and fed retrieved arXiv context, can simultaneously improve numerical rating consistency and judged review quality. Using Qwen2.5-7B-Instruct as the base, the authors report MSE against averaged human scores falling from 2.829 (SFT only) to 2.585 after RL, and Spearman correlation rising from 0.335 to 0.634; the same model also outscores previous SFT-based reviewers like DeepReviewer on all evaluator dimensions (e.g., analytical depth 4.18 vs 3.80). Ablations identify two load-bearing pieces: the retrieval context, whose removal degrades factual-correctness scores, and the GenRM reward, without which RL gains plateau ne","pith_inferences":["Editorial extension: because both the training reward and the evaluation metrics come from LLM judges, the same recipe could overfit to judge preferences; a natural test is to retrain with human pairwise preferences as the reward and see whether the Llama-3.3-70B-quality scores and MSE improve or degrade.","Editorial extension: the Gaussian rating reward treats the averaged human score as a single point, ignoring reviewer disagreement; rewarding calibrated rating distributions instead might generalize better to papers where human reviewers genuinely split.","Editorial extension: the retrieval pipeline only searches arXiv, so papers in fields with weak arXiv coverage would need a different corpus; the reported hallucination reductions may not transfer to those settings.","Editorial extension: if RL-trained reviewers are used to triage submissions, the improved rating alignment could systematically shift acceptance decisions, so the ethics section's 'support, not replace' framing deserves an empirical study of decision impact."],"forward_implications":["RL with a judge-based composite reward can be applied to non-verifiable generation tasks beyond review, such as scientific critique, legal analysis, or clinical commentary, whenever a reference output and rating anchor exist.","Retrieval-augmented reviewers should be expected to hallucinate less and to ground critiques in related work, making the factual-accuracy dimension of automated review more trustworthy.","Because SFT cold-start prevents rating collapse, any RL recipe for judgment-heavy text generation should keep an SFT warm-up and a rating-distribution balancing step.","Rule-only reward in non-verifiable tasks is insufficient; the paper's ablation implies judge-model rewards are required for analytical depth and actionable feedback.","Rating alignment with averaged human scores improves with RL, so automated triage systems that rank submissions could inherit this alignment."],"supporting_citations":[{"why":"Supplies DeepReview-13k long CoT reviews used for SFT cold-start and meta-reviews used as RL references.","marker":"(Zhu et al., 2025)"},{"why":"DeepSeek-R1 motivates RL for reasoning; the paper contrasts its rule-based rewards with the judge-based rewards needed for review quality.","marker":"(Guo et al., 2025)"},{"why":"Provides the Reinforce++ optimization algorithm used for RL training.","marker":"(Hu, 2025)"},{"why":"Defines GenRM, the generative reward model approach used as the judge component of the composite reward.","marker":"(Zhang et al., 2024)"},{"why":"Supplies the seven-dimension ReviewEval framework used for model-based evaluation.","marker":"(Kirtani et al., 2025)"},{"why":"Supplies the pairwise evaluation metrics (relation, absolute, confidence) used for ranking assessment.","marker":"(Chen et al., 2025)"},{"why":"Raw Reviewer2 data supplies part of the RL training corpus.","marker":"(Gao et al., 2024)"},{"why":"Justifies LLM-as-a-judge by showing judge models reliably assess relative response quality.","marker":"(Zheng et al., 2023)"}],"fun_headline_variants":["RL sharpens AI reviews and rating accuracy","Reinforcement learning drives deeper paper critiques","Retrieval-augmented RL improves review consistency","ReviewRL combines retrieval and RL for better reviews","AI reviewer trained with RL cuts rating errors"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole quality story rests on LLM judges: a Qwen2.5-14B judge supplies the training reward and a Llama-3.3-70B judge measures the outcome, so if these judges prefer longer or more fluent reviews rather than more accurate ones, the reported quality gains are partly an artifact of the judge.","fun_headline_variants_meta":{"raw":{"variants":["RL sharpens AI reviews and rating accuracy","Reinforcement learning drives deeper paper critiques","Retrieval-augmented RL improves review consistency","ReviewRL combines retrieval and RL for better reviews","AI reviewer trained with RL cuts rating errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1203,"prompt_tokens":713,"completion_tokens":490,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":421}},"tokens_in":457,"tokens_out":490,"duration_ms":5160,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:31:58.874906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect blind human pairwise comparisons between ReviewRL and DeepReviewer reviews on a random sample of the 472 ICLR 2025 papers. If humans do not prefer ReviewRL's reviews at a rate comparable to the Llama-3.3-70B judge's reported margins (e.g., analytical depth 4.18 vs 3.80), the model-based quality claim fails. A second decisive check: replace the GenRM judge's preferences with human preferences in training; if following human reward does not reproduce the reported MSE and quality gains, the composite reward is capturing judge style rather than review quality.","supporting_citations":[],"review_version":1}