Pith. sign in

REVIEW 4 major objections 6 minor 66 references

Making LLM peer review generate explicit paper-specific rubrics first, then judge against them, is what lets automated reviews cover more ground, take clearer stands, and resist injected 'accept' attacks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:47 UTC pith:MCTHXW6U

load-bearing objection A well-engineered three-stage rubric-driven review pipeline with a genuinely plausible design, but the headline gains look inflated by an LLM-inferred gold standard and the fixed general rubrics carry most of the coverage. the 4 major comments →

arxiv 2608.00005 v1 pith:MCTHXW6U submitted 2026-06-04 cs.CL cs.AI

RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review

Shuyu Guo , Wenxiang Hu , Yuyue Zhao , Yougang Lyu , Xiaohui Yan This is my paper
classification cs.CL cs.AI
keywords automated peer reviewrubric-driven generationlatent rubric factorizationreview generationhuman alignmentprompt-injection robustnessaccept/reject predictionLLM-based reviewer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

RubricReviewer's central claim is that the latent evaluation dimensions underlying human reviews — the 'rubrics' — should be made an explicit generative step, not left implicit. The paper formalizes peer review as a sequential factorization: first generate rubrics from the paper, then generate per-rubric reviews, then produce the final verdict conditioned on both. On a real-submission benchmark, this yields strongly broader rubric coverage (80.5 recall, roughly 35 points above the strongest baseline) and the best rating and accept/reject accuracy (MSE 1.365; 71.0%). If right, the rubric intermediate is not a cosmetic rearrangement but a genuine inductive bias: it splits one hard generation problem into two easier ones and anchors the final verdict in many small, checkable judgements.

Core claim

The paper's core discovery is that an explicit, paper-adaptive rubric set — a mix of general, literature-grounded, and cited-review-mined rubrics — is a load-bearing intermediate variable. Generating the rubric set R̄ first, then conditioning per-rubric reviews and the final verdict on it, lifts coverage of human evaluation dimensions to 80.5% (from at most 45.7%), even while producing 53.8 rubrics per paper. Conditioning also improves verdict agreement (R-Lenient 61.4), rating error (MSE 1.365), and accept/reject accuracy (71.0%). Ablations show the largest single contribution is the rubric-conditioned decomposition itself: removing it drops recall by 27.3 points; removing general rubrics d

What carries the argument

The machinery is the factorization p(R̄,R,y|P) = p(R̄|P)·p(R|R̄,P)·p(y|R̄,R,P). Each atomic review instantiates one rubric, and the rubric set partitions the review set. Rubrics are largely shared across papers while the atomic reviews instantiating them are paper-specific, so separating 'what dimensions to judge' from 'how to judge them' splits a high-variance generation problem into two easier subproblems. Two components realize the chain: Scout, a training-free agent that retrieves external evidence and drafts references, and Aligner, a fine-tuned model that consumes those references and outputs human-aligned final results at each stage. The 40 general rubrics form a stable shared backbon

Load-bearing premise

The load-bearing premise is that the rubric set recovered by an LLM decomposition of human reviews faithfully represents the dimensions human reviewers actually used; if that inference drifts, both the training targets and the 80.5% recall metric measure the model's consistency with the rubric-generator rather than with human evaluators.

What would settle it

Annotate a few hundred test papers independently: have human reviewers (who do not see the system) list the evaluation dimensions they actually applied, then measure rubric Recall against that human-listed gold set. If recall drops sharply relative to the 80.5% measured against LLM-inferred rubrics — especially on the paper-specific and domain rubrics — the human-alignment claim is not supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Paper-adaptive rubric factorization yields roughly 4–7× more evaluation dimensions per paper than direct or agentic baselines, while still covering 80.5% of the human rubric set — evidence that breadth and fidelity can coexist.
  • Conditioning each atomic review on its rubric shifts the verdict distribution toward the human consensus: R-Lenient agreement reaches 61.4 and verdict-distribution shift drops to 2.4.
  • The factorization endows the final assessment with better human alignment: lowest rating error (MSE 1.365) and highest accept/reject accuracy (71.0%).
  • Rubric conditioning is also a robustness mechanism: under a prompt-injection payload embedded in the paper, verdict-positive shift is only +1.4 percentage points and average rating shifts +0.02.
  • General rubrics are the dominant coverage source; ablating them collapses recall from 80.5 to 25.1, implying most review value lies in a shared, transferable dimension set.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the paper is right, the rubric intermediate should transfer as an inductive bias to other open-ended expert assessments (grant reviews, code reviews, clinical write-ups), where evaluation dimensions are more stable than their instantiations; a cross-domain replication keeping the same Aligner but re-deriving the rubric vocabulary would test this.
  • Because each atomic review is tied to a named rubric, the output format doubles as an audit trail: authors could trace a final accept/reject verdict to individual rubric-level judgements and rebut specific ones, which a holistic score cannot offer.
  • The ablation pattern suggests a cheaper scaled-down variant may preserve most of the gain: since the 40 general rubrics supply the dominant share of recall, a compact fixed taxonomy plus a light paper-specific extension could recover much of the observed 27-point drop from removing rubric conditioning, at substantially lower inference cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes RubricReviewer, a three-stage LLM-based peer-review pipeline that first generates an explicit rubric set for each manuscript, then generates reviews conditioned on those rubrics, and finally produces an overall score and accept/reject decision. The framework combines a training-free agent (Scout) that retrieves external evidence with a trained model (Aligner) fine-tuned from Phi-4. The central claim is that the factorization p(Rbar,R,y|P) = p(Rbar|P)p(R|Rbar,P)p(y|Rbar,R,P) improves both comprehensiveness and discriminativeness relative to direct paper-to-review mapping. Experiments on DeepReview-13k report rubric Recall of 80.5, per-rubric review agreement of 61.4 R-Lenient, and rating MSE of 1.365, together with ablations and a prompt-injection robustness analysis. The evaluation relies on gold rubrics inferred by GPT-5.2 from human reviews and on recall-style metrics.

Significance. If the empirical claims were established, the paper would make a useful contribution: making the rubric an explicit intermediate variable is a principled way to decompose open-ended review generation, and the combination of agentic evidence gathering with supervised fine-tuning is a promising design. The adversarial robustness analysis is also a welcome addition. However, the central evaluation is currently not convincing. The gold rubrics, training targets, and inference-time rubric generation all involve the same GPT-5.2 model family, so the reported 80.5 recall may reflect self-consistency rather than alignment with human reviewers. The ablation in Table 3 shows that most of the coverage gain comes from a fixed set of 40 general rubrics, not from the paper-adaptive components that the paper emphasizes. With 200 test papers and no uncertainty quantification, the small differences in Table 2 (e.g., MSE 1.365 vs. 1.372) cannot support the claim of best assessment accuracy. The conceptual framework is worth pursuing, but the current evidence requires substantial strengthening.

major comments (4)
  1. [§3.3 and §5.1] The rubric gold standard is constructed by GPT-5.2: Section 3.3 states that 'Using GPT-5.2, every review in R is decomposed into atomic reviews... Each ri is reverse-inferred to its rubric,' and Section 5.1 measures recall against these machine-inferred rubrics. The same model family (GPT-5.2) is used to generate training targets for Aligner and, through Scout, to produce rubrics at inference. This creates a closed evaluation loop: the high rubric Recall (80.5) may reflect the system's ability to reproduce GPT-5.2's rubric decomposition, not genuine agreement with human reviewers' evaluative dimensions. The Limitations section concedes that no large-scale human study was conducted. A human-annotated rubric set (even on a subset of papers), or at minimum an independently constructed rubric taxonomy, is needed to validate the central claim.
  2. [Table 3] The ablation evidence undermines the 'paper-adaptive' framing. Removing the fixed general rubrics collapses Recall from 80.5 to 25.1 (a drop of 55.4 points), whereas removing the model-generated paper-specific rubrics reduces Recall by only 1.9 points and removing cite-derived domain rubrics by 1.7 points. Thus the dominant contribution to the headline recall comes from a static, predefined checklist of 40 general rubrics, not from the adaptive rubric generation that the paper's factorization emphasizes. The authors should report recall separately for each rubric source and demonstrate that the adaptive components add meaningful coverage beyond a fixed generic rubric set.
  3. [§5.1, Tables 1–2] No uncertainty quantification is reported. The test set is only 200 papers, yet Table 2 claims the best assessment results with MSE 1.365 vs. 1.372 for DeepReviewer-14B and ACC 71.0% vs. 66.5%. These differences may be within noise, especially for accept/reject accuracy computed on 200 binary decisions. The paper should report standard errors, confidence intervals, or significance tests, and ideally multiple evaluation runs. The same applies to the R-Lenient/R-Strict metrics in Table 1. Additionally, the computation of 'verdict' and 'supporting reasoning' for these metrics is not specified; if an LLM performs this summarization, another model-in-the-loop bias is introduced.
  4. [§5.1, Table 1] Rubric Recall is reported without precision or false-positive rate. RubricReviewer generates on average 53.8 rubrics per paper, roughly 4–7 times the 7.7–13.5 rubrics of the baselines. With recall defined as the fraction of gold rubrics covered, generating many candidate rubrics can mechanically inflate recall, particularly when the gold vocabulary and the generated vocabulary come from the same GPT-5.2-based pipeline. The paper should report precision, F1, or an evaluation that penalizes spurious/irrelevant rubrics, such as human judgments of whether each generated rubric is useful for the paper at hand.
minor comments (6)
  1. [§3.2, z2] Typo: 'Genrating appropriate rubrics' should be 'Generating appropriate rubrics.'
  2. [Author affiliations] The affiliation email contains 'huaiwei.com' (likely a typo for 'huawei.com'), and 'Y ougang Lyu' has an extra space in the author list.
  3. [Table 3] The table marks '–' for R-Lenient and R-Strict in the rubric-stage ablation rows, but the text says all three metrics are reported. Please clarify whether these metrics are undefined for those ablations or were simply not measured; the current presentation is ambiguous.
  4. [§4.1 and throughout] The dataset name is written inconsistently as 'DeepReview-13k' and 'DeepReview-13K' (e.g., §4.1 vs. §3.3). Please standardize.
  5. [Figure 2 caption] The caption contains spacing artifacts: 'V erdict Positive' and 'A vg. Rating' should be 'Verdict Positive' and 'Avg. Rating.'
  6. [§5.4] The Value Analysis uses GPT-5.2 as the A/B judge with Tie as the default verdict. The paper should state the agreement rate of this judge with human judgments or otherwise justify that the judge is not biased toward RubricReviewer, especially because the generated reviews are longer and rubric-structured.

Circularity Check

2 steps flagged

Rubric Recall is a closed-loop self-consistency metric: the gold rubrics, the training targets, and the inference-time rubric generator all come from GPT-5.2.

specific steps
  1. fitted input called prediction [Section 3.3 (Rubric-generation data) and Section 5.1 (Table 1)]
    "Using GPT-5.2, every review in R is decomposed into atomic reviews {ri}. Each ri is reverse-inferred to its rubric... Canonical rubrics covering more than a fraction τ of D form the general rubrics (40 in total)... Table 1: Recall(%) is the fraction of gold rubrics covered."

    The gold rubrics against which Recall is measured are reverse-inferred by GPT-5.2 from the same human reviews used to build training targets, and the 40 general rubrics are clustered from those same GPT-5.2 rubrics and unconditionally added at inference. Aligner is trained to emit rubrics in this GPT-5.2-derived vocabulary, while Scout (also GPT-5.2) generates rubrics at inference. Thus high Recall reflects self-consistency between the system and the GPT-5.2 rubric extractor, not validated alignment with human evaluative dimensions. No human annotation of rubrics is provided; the Limitations admit 'we have not conducted a large-scale human study.'

  2. self definitional [Section 3.3 and Table 3 (ablation)]
    "General rubrics are excluded from the target since they are always added directly at inference. ... −general rubrics 25.1 ↓55.4"

    The 40 general rubrics are always prepended to every paper's rubric set regardless of content. Removing them drops Recall from 80.5 to 25.1 (a 55.4-point loss). Since the general rubrics are fitted from GPT-5.2's clustering of the training corpus and the gold rubrics are drawn from the same GPT-5.2 decomposition, the dominant share of Recall is structural: the system is rewarded for emitting a pre-fitted, corpus-derived vocabulary rather than for paper-adaptive rubric discovery. The adaptive model-generated rubrics add only 1.9 points.

full rationale

The paper's flagship coverage claim (Rubric Recall 80.5, 34.8 points above the strongest baseline) reduces to a closed evaluation loop. The gold rubrics are not human-annotated but are reverse-inferred by GPT-5.2 from the DeepReview-13k human reviews; the same GPT-5.2 outputs are clustered into the 40 general rubrics that the system always contributes at inference; and the training targets for Aligner are built from the same GPT-5.2 decompositions plus Scout's (GPT-5.2) drafts. Therefore the Recall metric measures how well the system reproduces GPT-5.2's own rubric decomposition, not independent alignment with human reviewers. The ablation confirms this: the fixed general rubric set, fitted from the corpus, accounts for 55.4 of the 80.5 Recall points. This is a clear case of a fitted input being called a prediction. The paper also trains the final-assessment stage partly on pseudo-golden reviews generated by the trained Aligner itself, further embedding self-consistency. On the positive side, the final rating/accept metrics (MSE 1.365, ACC 71.0%) are measured against human meta-reviewer decisions, which is an external target, and the prompt-injection robustness analysis is independent of the rubric gold standard. These independent results keep the paper from being wholly circular, but the central comprehensiveness claim is substantially undermined. Score 7: the core coverage result is forced by construction, while nontrivial independent content remains in the discriminativeness and robustness evaluations.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The factorization in Eq. (1) is a modelling choice, not a theorem. The system's validity rests on three untested premises: that human reviews admit a faithful rubric decomposition by an LLM; that covering the inferred rubric set is a sufficient measure of review quality (no precision constraint); and that GPT-5.2's refinements of human reviews preserve their content. The rubric taxonomy is induced from the training corpus and imposed on evaluation.

free parameters (4)
  • Rubric taxonomy (40 canonical rubrics) = 40 rubrics with category weights
    Induced by clustering GPT-5.2-recovered rubrics from the training corpus; used as predefined general rubrics at inference and as the target vocabulary for training. Not independently grounded.
  • Coverage threshold τ for general rubrics = not reported
    Rubrics covering more than fraction τ of the training corpus form the general set; exact value unreported but controls the rubric vocabulary size.
  • Retrieval budget K and top-k = not reported
    Controls Scout's evidence pooling in z3 and Stage 2; unspecified values affect which evidence is gathered and thus the rubrics and reviews.
  • Round budget for iterative retrieval = not reported
    Terminates retrieval in z3; unreported but influences evidence sufficiency.
axioms (4)
  • domain assumption Human reviews can be decomposed into atomic rubric–review pairs by an LLM.
    Section 3.3: 'Using GPT-5.2, every review in R is decomposed into atomic reviews... Each ri is reverse-inferred to its rubric.' The validity of this decomposition is assumed and never checked against human annotation.
  • domain assumption The rubrics underlying reviews are largely shared across papers.
    Section 3.1: 'although individual atomic reviews vary sharply across papers, the rubrics they instantiate are largely shared.' This motivates the factorization but is not empirically verified independently.
  • ad hoc to paper GPT-5.2 refinement of human reviews is faithful (re-articulates, adds no new opinions).
    Section 3.3: 'The refinement is constrained to re-articulate. Every claim in R⋆k must be supported by at least one atomic review in Rk, and no new opinions are allowed.' No verification that GPT-5.2 respects this constraint.
  • domain assumption External evidence retrieval improves review quality.
    The Scout design (§3.2) assumes that retrieving literature and OpenReview comments yields rubrics and reviews better aligned with human judgment. Ablations suggest a small benefit, but the assumption is not independently motivated.
invented entities (1)
  • Rubric latent variable R̄ no independent evidence
    purpose: Intermediate factorization P → R̄ → R → y; decomposes review generation into rubric generation, rubric-conditioned review generation, and final assessment.
    A modelling construct. Its validity is evaluated only through recall against GPT-5.2-inferred rubrics, which are generated by the same kind of model used in the pipeline, so there is no externally falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 15776 in / 12711 out tokens · 126854 ms · 2026-08-04T04:47:10.526848+00:00 · methodology

0 comments
read the original abstract

Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants. Existing LLM-based reviewers, however, face two structural limitations. First, they map manuscripts directly to reviews, leaving the underlying rubric implicit and entangling its derivation with the judgement. Second, the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but produce undirected critiques, while training-based reviewers inherit human discriminative judgement together with its noise and uneven coverage. We introduce RubricReviewer, a fully rubric-driven framework that addresses both limitations. It makes rubric generation an explicit intermediate step, so that both review generation and the final assessment are conditioned on paper-adaptive rubrics. It further combines a training-free agent (Scout) that gathers external evidence with a human-aligned trained model (Aligner) that consumes this evidence, fusing the strengths of both supervision sources. Experiments on real-world submissions show that RubricReviewer produces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt-injection attacks. Ablation studies further confirm the necessity of each component.

Figures

Figures reproduced from arXiv: 2608.00005 by Shuyu Guo, Wenxiang Hu, Xiaohui Yan, Yougang Lyu, Yuyue Zhao.

Figure 1
Figure 1. Figure 1: Overview of RUBRICREVIEWER’s inference pipeline. Inference unfolds in three stages: rubric generation, review generation, and final assessment. At every stage, SCOUT, a set of training-free LLM agents, retrieves external literature and produces a structured reference (candidate rubrics or draft reviews), while ALIGNER, a human-aligned trained model, consumes this reference and emits the final output. In St… view at source ↗
Figure 3
Figure 3. Figure 3: Per-paper Value verdict against the gold hu￾man review. Each method’s review is judged blind A/B against the gold review (random A/B swap, GPT-5.2 as the judge) and labelled Win/Tie/Lose. authors. We score each review’s value along three axes. Specificity rewards concrete, paper-targeted issues over vague generalities. Decisive stance re￾wards a clear verdict over neutral hedging. Target￾edness asks whethe… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 1 canonical work pages · 1 internal anchor

  1. [18]

    The Thirteenth International Conference on Learning Representations,

    Yixuan Weng and Minjun Zhu and Guangsheng Bao and Hongbo Zhang and Jindong Wang and Yue Zhang and Linyi Yang , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    2025

  2. [20]

    Workshop on Innovative Use of NLP for Building Educational Applications , year=

    Reviewriter: AI-Generated Instructions For Peer Review Writing , author=. Workshop on Innovative Use of NLP for Building Educational Applications , year=

  3. [21]

    ArXiv , year=

    MARG: Multi-Agent Review Generation for Scientific Papers , author=. ArXiv , year=

  4. [23]

    Proceedings of the ACM on Human-Computer Interaction , year=

    MetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review , author=. Proceedings of the ACM on Human-Computer Interaction , year=

  5. [27]

    Openreviewer: A specialized large language model for generating critical scientific paper reviews , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations) , pages=

    2025

  6. [28]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Deepreview: Improving llm-based paper review with human-like deep thinking process , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  7. [30]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    TreeReview: A dynamic tree of questions framework for deep and efficient LLM-based scientific peer review , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  8. [31]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    ReviewRL: Towards Automated Scientific Review with RL , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  9. [34]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Automatic paper reviewing with heterogeneous graph reasoning over llm-simulated reviewer-author debates , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  10. [37]

    2025 IEEE International Conference on Data Mining (ICDM) , pages=

    Unveiling the merits and defects of llms in automatic review generation for scientific papers , author=. 2025 IEEE International Conference on Data Mining (ICDM) , pages=. 2025 , organization=

    2025

  11. [39]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    The good, the bad and the constructive: Automatically measuring peer review’s utility for authors , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  12. [40]

    Scientometrics , pages=

    Large language models in peer review: challenges and opportunities , author=. Scientometrics , pages=. 2025 , publisher=

    2025

  13. [41]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  14. [42]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  15. [43]

    The Twelfth International Conference on Learning Representations,

    Tri Dao , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  16. [44]

    Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen

    Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen. LoRA: Low-Rank Adaptation of Large Language Models , booktitle =. 2022 , url =

    2022

  17. [46]

    Journal of Artificial Intelligence Research , volume=

    Can we automate scientific reviewing? , author=. Journal of Artificial Intelligence Research , volume=

  18. [47]

    NEJM AI , volume=

    Can large language models provide useful feedback on research papers? A large-scale empirical analysis , author=. NEJM AI , volume=. 2024 , publisher=

    2024

  19. [49]

    Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Qui \ n onero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. https://doi.org/10.48550/ARXIV.2505.08775 Healthbench: Evaluating large language models towards improved human health . CoRR, abs/2505.08775

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775 2025

  20. [50]

    Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok-Hay So, Zhijiang Guo, Liya Zhu, and Ngai Wong. 2025. Treereview: A dynamic tree of questions framework for deep and efficient llm-based scientific peer review. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15662--15693

    2025

  21. [51]

    Tri Dao. 2024. https://openreview.net/forum?id=mZn2Xyh9Ec Flashattention-2: Faster attention with better parallelism and work partitioning . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  22. [52]

    Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. https://api.semanticscholar.org/CorpusID:266899693 Marg: Multi-agent review generation for scientific papers . ArXiv, abs/2401.04259

    Pith/arXiv arXiv 2024

  23. [53]

    Nils Dycke and Iryna Gurevych. 2026. https://doi.org/10.1162/TACl.a.642 Automatic reviewers fail to detect faulty reasoning in research papers: A new counterfactual evaluation framework . Transactions of the Association for Computational Linguistics, 14:465--488

    doi:10.1162/tacl.a.642 2026

  24. [54]

    Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023. https://doi.org/10.18653/v1/2023.acl-long.277 NLP eer: A unified resource for the computational study of peer review . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049--5073, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.277 2023

  25. [55]

    Xian Gao, Jiacheng Ruan, Zongyun Zhang, Jingsheng Gao, Ting Liu, and Yuzhuo Fu. 2025. Reviewagents: Bridging the gap between human and ai-generated paper reviews. arXiv preprint arXiv:2503.08506

    Pith/arXiv arXiv 2025

  26. [56]

    Zhaolin Gao, Kiant \'e Brantley, and Thorsten Joachims. 2024. Reviewer2: Optimizing review generation through prompt generation. arXiv preprint arXiv:2402.10886

    Pith/arXiv arXiv 2024

  27. [57]

    Palash Goyal, Mihir Parmar, Yiwen Song, Hamid Palangi, Tomas Pfister, and Jinsung Yoon. 2026. Scholarpeer: A context-aware multi-agent framework for automated peer review. arXiv preprint arXiv:2601.22638

    Pith/arXiv arXiv 2026

  28. [58]

    Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. 2025. https://doi.org/10.48550/ARXIV.2507.17746 Rubrics as rewards: Reinforcement learning beyond verifiable domains . CoRR, abs/2507.17746

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.17746 2025

  29. [59]

    Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.745 Llm-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL ...

    doi:10.18653/v1/2024.acl-long.745 2024

  30. [60]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net

    2022

  31. [61]

    Maximilian Idahl and Zahra Ahmadi. 2025. Openreviewer: A specialized large language model for generating critical scientific paper reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), pages 550--562

    2025

  32. [62]

    Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. 2024. https://doi.org/10.18653/V1/2024.EMNLP-MAIN.70 Agentreview: Exploring peer review dynamics with LLM agents . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 , pages 1208...

    doi:10.18653/v1/2024.emnlp-main.70 2024

  33. [63]

    Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. https://doi.org/10.18653/v1/N18-1149 A dataset of peer reviews ( P eer R ead): Collection, insights and NLP applications . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Ling...

    doi:10.18653/v1/n18-1149 2018

  34. [64]

    Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. https://doi.org/10.18653/v1/2021.naacl-main.393 Hurdles to progress in long-form question answering . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940--4957, Online. Association for Computational L...

    doi:10.18653/v1/2021.naacl-main.393 2021

  35. [65]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  36. [66]

    Bowen Li, Haochen Ma, Yuxin Wang, Jie Yang, Xinchi Chen, Xuanjing Huang, Yining Zheng, and Xipeng Qiu. 2026 a . Beyond rating: A comprehensive evaluation and benchmark for ai reviews. arXiv preprint arXiv:2604.19502

    Pith/arXiv arXiv 2026

  37. [67]

    Ruochi Li, Haoxuan Zhang, Edward Gehringer, Ting Xiao, Junhua Ding, and Haihua Chen. 2025. Unveiling the merits and defects of llms in automatic review generation for scientific papers. In 2025 IEEE International Conference on Data Mining (ICDM), pages 1370--1379. IEEE

    2025

  38. [68]

    Shuaimin Li, Liyang Fan, Yufang Lin, Zeyang Li, Xian Wei, Shiwen Ni, Hamid Alinejad-Rokny, and Min Yang. 2026 b . Automatic paper reviewing with heterogeneous graph reasoning over llm-simulated reviewer-author debates. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 31717--31725

    2026

  39. [69]

    Zhuofeng Li, Yi Lu, Dongfu Jiang, Haoxiang Zhang, Yuyang Bai, Chuan Li, Yu Wang, Shuiwang Ji, Jianwen Xie, and Yu Zhang. 2026 c . https://doi.org/10.48550/ARXIV.2604.14261 Reviewgrounder: Improving review substantiveness with rubric-guided, tool-integrated agents . CoRR, abs/2604.14261

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.14261 2026

  40. [70]

    Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, and 1 others. 2024. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196

    2024

  41. [71]

    Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2025. https://doi.org/10.48550/ARXIV.2510.07743 Openrubrics: Towards scalable synthetic rubric generation for reward modeling and LLM alignment . CoRR, abs/2510.07743

    doi:10.48550/arxiv.2510.07743 2025

  42. [72]

    Ilya Loshchilov and Frank Hutter. 2017. https://api.semanticscholar.org/CorpusID:53592270 Decoupled weight decay regularization . In International Conference on Learning Representations

    2017

  43. [73]

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292

    Pith/arXiv arXiv 2024

  44. [74]

    Changze Lv, Jie Zhou, Wentao Zhao, Jingwen Xu, Zisu Huang, Muzhao Tian, Shihan Dou, Tao Gui, Le Tian, Xiao Zhou, Xiaoqing Zheng, Xuanjing Huang, and Jie Zhou. 2026. https://doi.org/10.48550/ARXIV.2602.03619 Learning query-specific rubrics from human preferences for deepresearch report generation . CoRR, abs/2602.03619

    doi:10.48550/arxiv.2602.03619 2026

  45. [75]

    Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. 2024. https://doi.org/10.48550/ARXIV.2409.16191 Hellobench: Evaluating long text generation capabilities of large language models . CoRR, abs/2409.16191

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.16191 2024

  46. [76]

    Abdelrahman Sadallah, Tim Baumg \"a rtner, Iryna Gurevych, and Ted Briscoe. 2025. The good, the bad and the constructive: Automatically measuring peer review’s utility for authors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28979--29009

    2025

  47. [77]

    Finlayson, David A

    Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David A. Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen - tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, and 2 others. 2025. https://doi.org/10.48550/ARXIV.2511.19399 DR tulu: Re...

    Pith/arXiv arXiv doi:10.48550/arxiv.2511.19399 2025

  48. [78]

    Kanell, Peter Xu, Omar Khattab, and Monica S

    Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. 2024. https://doi.org/10.18653/V1/2024.NAACL-LONG.347 Assisting in writing wikipedia-like articles from scratch with large language models . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    doi:10.18653/v1/2024.naacl-long.347 2024

  49. [79]

    Hendryx, Brad Kenstler, and Bing Liu

    Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, and Bing Liu. 2025. https://doi.org/10.48550/ARXIV.2511.07685 Researchrubrics: A benchmark of prompts and rubrics for evaluating ...

    doi:10.48550/arxiv.2511.07685 2025

  50. [80]

    Leheng Sheng, Wenchang Ma, Ruixin Hong, Xiang Wang, An Zhang, and Tat - Seng Chua. 2026. https://doi.org/10.48550/ARXIV.2602.10885 Reinforcing chain-of-thought reasoning with self-evolving rubrics . CoRR, abs/2602.10885

    doi:10.48550/arxiv.2602.10885 2026

  51. [81]

    Xiaotian Su, Thiemo Wambsganss, Roman Rietsche, Seyed Parsa Neshaei, and Tanja K \"a ser. 2025. https://api.semanticscholar.org/CorpusID:259376592 Reviewriter: Ai-generated instructions for peer review writing . In Workshop on Innovative Use of NLP for Building Educational Applications

    2025

  52. [82]

    Lu Sun, Stone Tao, Junjie Hu, and Steven P. Dow. 2024. https://api.semanticscholar.org/CorpusID:269470548 Metawriter: Exploring the potential and perils of ai writing support in scientific peer review . Proceedings of the ACM on Human-Computer Interaction, 8:1 -- 32

    2024

  53. [83]

    Zhuanlan Sun. 2025. Large language models in peer review: challenges and opportunities. Scientometrics, pages 1--44

    2025

  54. [84]

    Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. 2025. https://doi.org/10.48550/ARXIV.2507.18624 Checklists are better than reward models for aligning language models . CoRR, abs/2507.18624

    doi:10.48550/arxiv.2507.18624 2025

  55. [85]

    Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025. https://openreview.net/forum?id=bjcsVLoHYs Cycleresearcher: Improving automated research via automated review . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

    2025

  56. [86]

    Yixuan Weng, Minjun Zhu, Qiujie Xie, Zhiyuan Ning, Shichen Li, Panzhong Lu, Zhen Lin, Enhao Gu, Qiyao Sun, and Yue Zhang. 2026. Deepreviewer 2.0: A traceable agentic system for auditable scientific peer review. arXiv preprint arXiv:2604.09590

    Pith/arXiv arXiv 2026

  57. [87]

    Shican Wu, Xiao Ma, Dehui Luo, Lulu Li, Xiangcheng Shi, Xin Chang, Xiaoyun Lin, Ran Luo, Chunlei Pei, Changying Du, Zhi-Jian Zhao, and Jinlong Gong. 2025. https://doi.org/10.1093/nsr/nwaf169 Automated literature research and review-generation method based on large language models . National Science Review, 12(6)

    doi:10.1093/nsr/nwaf169 2025

  58. [88]

    Lipeng Xie, Sen Huang, Zhuo Zhang, Anni Zou, Yunpeng Zhai, Dingchao Ren, Kezun Zhang, Haoyuan Hu, Boyin Liu, Haoran Chen, and 1 others. 2025. Auto-rubric: Learning from implicit weights to explicit rubrics for reward modeling. arXiv preprint arXiv:2510.17314

    arXiv 2025

  59. [89]

    Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang. 2026. https://doi.org/10.48550/ARXIV.2602.01511 Alternating reinforcement learning for rubric-based reward modeling in non-verifiable LLM post-training . CoRR, abs/2602.01511

    doi:10.48550/arxiv.2602.01511 2026

  60. [90]

    Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171--212

    2022

  61. [91]

    Sihang Zeng, Kai Tian, Kaiyan Zhang, Yuru Wang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, and 1 others. 2025. Reviewrl: Towards automated scientific review with rl. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16942--16954

    2025

  62. [92]

    Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, and Lifeng Jin. 2025. https://doi.org/10.48550/ARXIV.2509.21500 Chasing the tail: Effective rubric-based reward modeling for large language model post-training . CoRR, abs/2509.21500

    doi:10.48550/arxiv.2509.21500 2025

  63. [93]

    Yinuo Zhang, Dingcheng Huang, Haifeng Suo, Yizhuo Li, Ziya Zhao, Junhao Xu, Zhiying Tu, Dianhui Chu, Deming Zhai, Xianming Liu, and 1 others. 2026. Echoreview: Learning peer review from the echoes of scientific citations. arXiv preprint arXiv:2602.00733

    arXiv 2026

  64. [94]

    Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Jiale Zhao, Jingwen Yang, Jianwei Lv, Kongcheng Zhang, Yihe Zhou, Hengtong Lu, Wei Chen, Yan Xie, and Mingli Song. 2025. https://doi.org/10.48550/ARXIV.2508.16949 Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general LLM reasoning . CoRR, abs/2508.16949

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.16949 2025

  65. [95]

    Changjia Zhu, Junjie Xiong, Renkai Ma, Zhicong Lu, Yao Liu, and Lingyao Li. 2025 a . https://doi.org/10.48550/ARXIV.2509.09912 When your reviewer is an LLM: biases, divergence, and prompt injection risks in peer review . CoRR, abs/2509.09912

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.09912 2025

  66. [96]

    Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025 b . Deepreview: Improving llm-based paper review with human-like deep thinking process. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29330--29355

    2025