What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review

Ming Jin

arxiv: 2604.19998 · v1 · submitted 2026-04-21 · 💻 cs.AI

What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review

Ming Jin This is my paper

Pith reviewed 2026-05-10 01:51 UTC · model grok-4.3

classification 💻 cs.AI

keywords AI peer reviewconcern alignmentmatch graphreview qualitycalibrationpeer review diagnosticsAI-generated reviews

0 comments

The pith

AI review systems detect many official concerns but often miscalibrate their weight, labeling 25-55% of issues on accepted papers as decisive blockers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces concern alignment, a diagnostic framework that evaluates AI-generated reviews by matching individual concerns to official ones rather than relying on final verdict agreement alone. It constructs a match graph that pairs concerns across sources and annotates each pair for type, severity, and post-rebuttal outcome, then derives an evaluation ladder that progresses from simple detection to verdict-stratified behavior, decision-aware calibration, and rebuttal-aware decomposition. In a pilot with four public AI review systems, the analysis finds that detection rates are non-trivial yet most systems still assign decisive status to a quarter to over half of the concerns appearing on papers that were ultimately accepted. Under the same operational definition, official reviews treated none of those concerns on accepted papers as decisive blockers. The work shows that identical verdict accuracy can hide very different underlying patterns of over-rejection or under-recall.

Core claim

The central claim is that concern-level analysis shows detection alone does not determine review quality; calibration is the binding constraint. Systems identify non-trivial fractions of official concerns, yet most mark 25--55% of concerns on accepted papers as decisive, while no official concern on accepted papers was treated as a decisive blocker under the paper's operationalization. Identical overall verdict accuracy can conceal reject-heavy behavior versus low-recall profiles, low full-review false decisive rates can partly reflect concern dilution rather than calibrated prioritization, and most systems do not emit a native accept/reject decision so that inference from tone is method- or

What carries the argument

The match graph, a bipartite alignment between official and AI-generated concerns annotated with match type, severity, and post-rebuttal treatment, which supports deriving the full evaluation ladder from binary accuracy through calibration and rebuttal decomposition.

If this is right

Identical overall verdict accuracy can conceal reject-heavy behavior versus low-recall profiles.
Low full-review false decisive rates can partly reflect concern dilution rather than calibrated prioritization.
Most systems do not emit a native accept/reject, and inferring it from review tone is method-sensitive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Training data for AI reviewers could be constructed by labeling individual concerns according to whether they were decisive in the final human decision.
The same match-graph technique could be used to measure consistency among multiple human reviewers on the same paper.
Hybrid review pipelines might route only high-severity, high-calibration concerns to human experts while letting AI handle lower-stakes items.

Load-bearing premise

The manual annotation process that builds the match graph accurately captures the review rationale that shaped the final acceptance or rejection decision.

What would settle it

A larger evaluation on held-out papers with known final decisions in which the fraction of concerns marked decisive by each AI system on accepted papers falls to zero or near-zero, matching the rate observed in the official reviews.

Figures

Figures reproduced from arXiv: 2604.19998 by Ming Jin.

**Figure 2.** Figure 2: Progressive reveal: the same AI review (System L, Opus, accepted paper) diagnosed [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: False decisive rate on accepted papers. Most single-agent systems mark 25–55% of concerns decisive under the ACaligned operationalization. The calibration failure is visible at the level of individual concerns. On the accepted-paper example in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Recall by AC treatment on rejected papers. Positive gaps indicate greater attention to decisive blockers than to resolved concerns. Level 4 decomposes recall by AC treatment ( [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

**Figure 5.** Figure 5: Model effect slope charts: same method evaluated on Opus vs. GPT-4o. System L (blue) [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗

**Figure 6.** Figure 6: Average concern count by severity (3-run means). Opus systems generate more concerns [PITH_FULL_IMAGE:figures/full_fig_p025_6.png] view at source ↗

**Figure 7.** Figure 7: FDR on accepted papers (left) and decisive recall on rejected papers (right) as [PITH_FULL_IMAGE:figures/full_fig_p026_7.png] view at source ↗

read the original abstract

Evaluating AI-generated reviews by verdict agreement is widely recognized as insufficient, yet current alternatives rarely audit which concerns a system identifies, how it prioritizes them, or whether those priorities align with the review rationale that shaped the final assessment. We propose concern alignment, a diagnostic framework that evaluates AI reviews at the concern level rather than only at the verdict level. The framework's core data structure is the match graph, a bipartite alignment between official and AI-generated concerns annotated with match type, severity, and post-rebuttal treatment. From this artifact we derive an evaluation ladder that moves from binary accuracy to concern detection, verdict-stratified behavior, decision-aware calibration, and rebuttal-aware decomposition. In a pilot study of four public AI review systems evaluated in six configurations, concern-level analysis suggests that detection alone does not determine review quality; calibration is often the binding constraint. Systems detect non-trivial fractions of official concerns yet most mark 25--55% of concerns on accepted papers as decisive, where, under our operationalization, no official concern on accepted papers was treated as a decisive blocker. Identical overall verdict accuracy can conceal reject-heavy behavior versus low-recall profiles, and low full-review false decisive rates can partly reflect concern dilution rather than calibrated prioritization. Most systems do not emit a native accept/reject, and inferring it from review tone is method-sensitive, reinforcing the need for concern-level diagnostics that remain stable across inference choices. The contribution is a reusable evaluation framework for auditing which concerns AI reviewers identify, how they weight them, and whether those priorities align with the review rationale that informed the paper's final assessment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper gives a practical framework for auditing AI reviews at the concern level instead of just verdicts, but the pilot evidence for its calibration claims is too thin on validation.

read the letter

The main thing to know is that the authors move past verdict-agreement checks by building a match graph that lines up official concerns with AI-generated ones, then tagging each pair for match type, severity, and how rebuttals were handled. From there they lay out an evaluation ladder that steps through detection rates, verdict-stratified patterns, decision-aware calibration, and rebuttal decomposition. In the pilot on four systems, they find AI tools often flag 25-55% of concerns on accepted papers as decisive while the official record treated none that way, suggesting calibration is the tighter constraint than raw detection. Identical verdict accuracy can mask very different underlying behaviors, and inferring an accept/reject from tone turns out to be sensitive to the inference rule chosen. That observation is useful and the structure avoids obvious circularity by working from external annotations rather than the authors' own fitted quantities. The framework itself is straightforward to apply and gives a reusable artifact for anyone who wants to dissect what an AI review actually did. The soft spot is the pilot. It reports patterns but gives no sample sizes, no inter-annotator agreement numbers, no adjudication protocol, and no external check against reviewer logs or final outcomes. The central calibration gap therefore rests on how the manual severity and post-rebuttal labels were assigned; a different set of annotators could shrink or widen that gap without changing the raw AI output. The paper is aimed at people building or testing AI review tools and at organizers who need better diagnostics than verdict match rates. Readers who care about automated peer review or meta-research on reviewing will get concrete value from the ladder and the match-graph format. It deserves a serious referee because the problem is timely, the proposed structure is new, and the pilot already surfaces actionable distinctions even if the methods section needs more detail and validation work.

Referee Report

2 major / 1 minor

Summary. The paper claims to introduce a concern-level diagnostic framework called concern alignment for evaluating AI-generated peer reviews. The core is the match graph, a bipartite alignment between official and AI concerns annotated with match type, severity, and post-rebuttal treatment. This enables an evaluation ladder from detection to calibration and rebuttal-aware analysis. In a pilot study involving four AI review systems in six configurations, the authors find that while systems detect non-trivial fractions of concerns, calibration is the binding constraint, with AI systems marking 25-55% of concerns on accepted papers as decisive, whereas no official concerns on accepted papers were decisive blockers under their operationalization.

Significance. Should the operationalization and annotations prove robust, this framework represents a meaningful step forward in AI review evaluation by moving beyond verdict agreement to granular concern analysis. It highlights that calibration, not detection, often limits quality and provides a reusable tool that accounts for rebuttals and is less sensitive to verdict inference methods. This could inform the development of better AI reviewers and more reliable auditing practices.

major comments (2)

[Pilot study] The description of the manual annotation process for constructing the match graph and labeling concerns as decisive (including severity and post-rebuttal treatment) provides no inter-annotator agreement statistics, adjudication protocol, or external validation against reviewer decision logs or rebuttal outcomes. This is load-bearing for the central claim that calibration is the binding constraint, since the reported 25--55% vs. 0% gap on accepted papers depends directly on these annotations faithfully reconstructing the original review rationales.
[Results and evaluation ladder] The pilot reports quantitative patterns on detection rates, verdict-stratified behavior, and calibration without specifying the number of papers or concerns analyzed, without statistical tests, and without sensitivity analysis to annotation choices. This makes it difficult to evaluate the robustness of the conclusion that identical overall verdict accuracy can conceal reject-heavy vs. low-recall profiles.

minor comments (1)

[Abstract] The abstract states that most systems do not emit a native accept/reject verdict and that inference from tone is method-sensitive, but provides no concrete description of the inference procedures used across the six configurations.

Simulated Author's Rebuttal

2 responses · 1 unresolved

Thank you for your constructive feedback on our manuscript introducing the concern alignment framework. We appreciate the focus on annotation robustness and quantitative rigor in the pilot study, as these directly support the central claims about calibration as a binding constraint. We address each major comment below, noting planned revisions where appropriate.

read point-by-point responses

Referee: [Pilot study] The description of the manual annotation process for constructing the match graph and labeling concerns as decisive (including severity and post-rebuttal treatment) provides no inter-annotator agreement statistics, adjudication protocol, or external validation against reviewer decision logs or rebuttal outcomes. This is load-bearing for the central claim that calibration is the binding constraint, since the reported 25--55% vs. 0% gap on accepted papers depends directly on these annotations faithfully reconstructing the original review rationales.

Authors: We agree that the annotation process is load-bearing for the calibration claims and that greater transparency is needed. The match graphs were constructed via a systematic protocol: concerns were extracted from official and AI reviews using explicit criteria for atomicity and actionability, then aligned bipartitely with labels for match type, severity (decisive vs. non-decisive), and post-rebuttal treatment based on language in the reviews. As a pilot, this was performed by the authors with internal consistency reviews rather than independent annotators. In revision we will expand the methods section with the full protocol, report any spot-check agreement metrics, and add a limitations paragraph on the lack of external validation against private reviewer logs. This makes the operationalization reproducible while acknowledging the pilot scope. revision: partial
Referee: [Results and evaluation ladder] The pilot reports quantitative patterns on detection rates, verdict-stratified behavior, and calibration without specifying the number of papers or concerns analyzed, without statistical tests, and without sensitivity analysis to annotation choices. This makes it difficult to evaluate the robustness of the conclusion that identical overall verdict accuracy can conceal reject-heavy vs. low-recall profiles.

Authors: We accept this critique and will strengthen the results reporting. The pilot evaluated four AI systems across six configurations on a fixed set of papers; we will explicitly state the exact counts of papers and concerns analyzed. In the revision we will incorporate basic statistical comparisons (e.g., proportion tests) for the reported detection and calibration gaps where sample sizes permit, plus a sensitivity analysis varying the decisive-label threshold and match criteria. These additions will better substantiate that verdict-level accuracy can mask divergent concern profiles without overclaiming generalizability. revision: yes

standing simulated objections not resolved

External validation of annotations against reviewer decision logs or rebuttal outcomes, which is unavailable in this pilot due to lack of access to private reviewer data.

Circularity Check

0 steps flagged

No circularity: framework and claims derive from independent annotations

full rationale

The paper defines a new concern-alignment framework (match graph with match type/severity/post-rebuttal labels, plus the derived evaluation ladder) from first principles and applies it to external review data via manual annotation in a pilot study. Central claims about detection versus calibration follow directly from comparing AI outputs against this constructed artifact on accepted papers. No equations reduce a prediction to a fitted input by construction, no self-citations are invoked as load-bearing uniqueness theorems, and no ansatz or renaming of prior results occurs. The derivation chain remains self-contained against the provided annotations and does not collapse to author-defined quantities.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

The central claim rests on a newly defined framework whose validity depends on assumptions about concern identification and annotation quality; the pilot study introduces no free parameters but relies on an operationalization of decisive concerns whose details are not supplied.

axioms (1)

domain assumption Official reviewer concerns can be reliably extracted and matched to AI-generated concerns by human annotators
The match graph is constructed from such annotations; the framework's diagnostics depend on this step being accurate and reproducible.

invented entities (2)

match graph no independent evidence
purpose: Bipartite alignment between official and AI-generated concerns annotated with match type, severity, and post-rebuttal treatment
New data structure introduced to support concern-level rather than verdict-level evaluation.
evaluation ladder no independent evidence
purpose: Progressive diagnostic levels from binary accuracy through rebuttal-aware decomposition
Derived construct that organizes the framework's metrics.

pith-pipeline@v0.9.0 · 5585 in / 1482 out tokens · 73520 ms · 2026-05-10T01:51:32.631818+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

[1]

NEJM AI , volume=

Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis , author=. NEJM AI , volume=. 2024 , doi=

work page 2024
[2]

Lu, Chris and Lu, Cong and Lange, Robert Tjarko and Foerster, Jakob and Clune, Jeff and Ha, David , journal=. The

work page
[3]

D'Arcy, Mike and Hope, Tom and Birnbaum, Larry and Downey, Doug , journal=

work page
[4]

Inconsistency in conference peer review: Revisiting the 2014

Cortes, Corinna and Lawrence, Neil D , journal=. Inconsistency in conference peer review: Revisiting the 2014

work page 2014
[5]

Has the machine learning review process become more arbitrary as the field has grown? the

Beygelzimer, Alina and Dauphin, Yann N and Liang, Percy and Vaughan, Jennifer Wortman , journal=. Has the machine learning review process become more arbitrary as the field has grown? the

work page
[6]

and Zou, James Y

Liang, Weixin and Izzo, Zachary and Zhang, Yaohui and Lepp, Haley and Cao, Hancheng and Zhao, Xuandong and Chen, Lingjiao and Ye, Haotian and Liu, Sheng and Huang, Zhi and McFarland, Daniel A. and Zou, James Y. , booktitle=. Monitoring. 2024 , publisher=

work page 2024
[7]

and Veselovsky, Veniamin and West, Robert , journal=

Russo, Giuseppe and Horta Ribeiro, Manoel and Davidson, Tim R. and Veselovsky, Veniamin and West, Robert , journal=. The. 2025 , doi=

work page 2025
[8]

Ma, Qianli and Guo, Chang and Tian, Zhiheng and Wang, Siyu and Xiao, Jipeng and Yue, Yuanhao and Zhang, Zhipeng , journal=

work page
[9]

2020 , address=

Cheng, Liying and Bing, Lidong and Yu, Qian and Lu, Wei and Si, Luo , booktitle=. 2020 , address=. doi:10.18653/v1/2020.emnlp-main.569 , url=

work page doi:10.18653/v1/2020.emnlp-main.569 2020
[10]

2022 , address=

Kennard, Neha Nayak and O’Gorman, Tim and Das, Rajarshi and Sharma, Akshay and Bagchi, Chhandak and Clinton, Matthew and Yelugam, Pranay Kumar and Zamani, Hamed and McCallum, Andrew , booktitle=. 2022 , address=. doi:10.18653/v1/2022.naacl-main.89 , url=

work page doi:10.18653/v1/2022.naacl-main.89 2022
[11]

2019 , address=

Gao, Yang and Eger, Steffen and Kuznetsov, Ilia and Gurevych, Iryna and Miyao, Yusuke , booktitle=. 2019 , address=

work page 2019
[12]

Mind the Blind Spots: A Focus-Level Evaluation Framework for

Shin, Hyungyu and Tang, Jingyu and Lee, Yoonjoo and Kim, Nayoung and Lim, Hyunseung and Cho, Ji Yong and Hong, Hwajung and Lee, Moontae and Kim, Juho , booktitle=. Mind the Blind Spots: A Focus-Level Evaluation Framework for. 2025 , address=. doi:10.18653/v1/2025.emnlp-main.1805 , url=

work page doi:10.18653/v1/2025.emnlp-main.1805 2025
[13]

Ryu, Hyun and Jang, Doohyuk and Lee, Hyemin S. and Jeong, Joonhyun and Kim, Gyeongman and Cho, Donghyeon and Chu, Gyouk and Hwang, Minyeong and Jang, Hyeongwon and Kim, Changhun and Kim, Haechan and Kim, Jina and Kim, Joowon and Kim, Yoonjeon and Lee, Kwanhyung and Park, Chanjae and Yun, Heecheol and Betz, Gregor and Yang, Eunho , journal=. 2025 , doi=

work page 2025
[14]

2025 , address=

Garg, Madhav Krishan and Prasad, Tejash and Singhal, Tanmay and Kirtani, Chhavi and Mandal, Murari and Kumar, Dhruv , booktitle=. 2025 , address=. doi:10.18653/v1/2025.findings-emnlp.1120 , url=

work page doi:10.18653/v1/2025.findings-emnlp.1120 2025
[15]

Xu, Zhijian and Zhao, Yilun and Patwardhan, Manasi and Vig, Lovekesh and Cohan, Arman , booktitle=. Can. 2025 , address=. doi:10.18653/v1/2025.acl-long.1009 , url=

work page doi:10.18653/v1/2025.acl-long.1009 2025
[16]

Gao, Xian and Ruan, Jiacheng and Zhang, Zongyun and Gao, Jingsheng and Liu, Ting and Fu, Yuzhuo , journal=

work page
[17]

2025 , address=

Idahl, Maximilian and Ahmadi, Zahra , booktitle=. 2025 , address=. doi:10.18653/v1/2025.naacl-demo.44 , url=

work page doi:10.18653/v1/2025.naacl-demo.44 2025
[18]

From Replication to Redesign: Exploring Pairwise Comparisons for

Zhang, Yaohui and Zhang, Haijing and Ji, Wenlong and Hua, Tianyu and Haber, Nick and Cao, Hancheng and Liang, Weixin , booktitle=. From Replication to Redesign: Exploring Pairwise Comparisons for. 2025 , url=

work page 2025
[19]

Nature Machine Intelligence , volume=

A Large-Scale Randomized Study of Large Language Model Feedback in Peer Review , author=. Nature Machine Intelligence , volume=. 2026 , doi=

work page 2026
[20]

2025 , publisher=

Lou, Renze and Xu, Hanzi and Wang, Sijia and Du, Jiangshu and Kamoi, Ryo and Lu, Xiaoxin and Xie, Jian and Sun, Yuxuan and Zhang, Yusen and Ahn, Jihyun Janice and Fang, Hongchao and Zou, Zhuoyang and Ma, Wenchao and Li, Xi and Zhang, Kai and Xia, Congying and Huang, Lifu and Yin, Wenpeng , booktitle=. 2025 , publisher=

work page 2025
[21]

Zou, Zhuoyang and Ansari, Abolfazl and Zhang, Delvin Ce and Lee, Dongwon and Yin, Wenpeng , journal=

work page
[22]

Zhang, Daoze and Bao, Zhijian and Du, Sihang and Zhao, Zhiyi and Zhang, Kuangling and Bao, Dezheng and Yang, Yang , journal=

work page
[23]

Wu, Sihong and Ma, Yiling and Zhao, Yilun and Hu, Tiansheng and Jiang, Owen and Patwardhan, Manasi and Cohan, Arman , journal=

work page

[1] [1]

NEJM AI , volume=

Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis , author=. NEJM AI , volume=. 2024 , doi=

work page 2024

[2] [2]

Lu, Chris and Lu, Cong and Lange, Robert Tjarko and Foerster, Jakob and Clune, Jeff and Ha, David , journal=. The

work page

[3] [3]

D'Arcy, Mike and Hope, Tom and Birnbaum, Larry and Downey, Doug , journal=

work page

[4] [4]

Inconsistency in conference peer review: Revisiting the 2014

Cortes, Corinna and Lawrence, Neil D , journal=. Inconsistency in conference peer review: Revisiting the 2014

work page 2014

[5] [5]

Has the machine learning review process become more arbitrary as the field has grown? the

Beygelzimer, Alina and Dauphin, Yann N and Liang, Percy and Vaughan, Jennifer Wortman , journal=. Has the machine learning review process become more arbitrary as the field has grown? the

work page

[6] [6]

and Zou, James Y

Liang, Weixin and Izzo, Zachary and Zhang, Yaohui and Lepp, Haley and Cao, Hancheng and Zhao, Xuandong and Chen, Lingjiao and Ye, Haotian and Liu, Sheng and Huang, Zhi and McFarland, Daniel A. and Zou, James Y. , booktitle=. Monitoring. 2024 , publisher=

work page 2024

[7] [7]

and Veselovsky, Veniamin and West, Robert , journal=

Russo, Giuseppe and Horta Ribeiro, Manoel and Davidson, Tim R. and Veselovsky, Veniamin and West, Robert , journal=. The. 2025 , doi=

work page 2025

[8] [8]

Ma, Qianli and Guo, Chang and Tian, Zhiheng and Wang, Siyu and Xiao, Jipeng and Yue, Yuanhao and Zhang, Zhipeng , journal=

work page

[9] [9]

2020 , address=

Cheng, Liying and Bing, Lidong and Yu, Qian and Lu, Wei and Si, Luo , booktitle=. 2020 , address=. doi:10.18653/v1/2020.emnlp-main.569 , url=

work page doi:10.18653/v1/2020.emnlp-main.569 2020

[10] [10]

2022 , address=

Kennard, Neha Nayak and O’Gorman, Tim and Das, Rajarshi and Sharma, Akshay and Bagchi, Chhandak and Clinton, Matthew and Yelugam, Pranay Kumar and Zamani, Hamed and McCallum, Andrew , booktitle=. 2022 , address=. doi:10.18653/v1/2022.naacl-main.89 , url=

work page doi:10.18653/v1/2022.naacl-main.89 2022

[11] [11]

2019 , address=

Gao, Yang and Eger, Steffen and Kuznetsov, Ilia and Gurevych, Iryna and Miyao, Yusuke , booktitle=. 2019 , address=

work page 2019

[12] [12]

Mind the Blind Spots: A Focus-Level Evaluation Framework for

Shin, Hyungyu and Tang, Jingyu and Lee, Yoonjoo and Kim, Nayoung and Lim, Hyunseung and Cho, Ji Yong and Hong, Hwajung and Lee, Moontae and Kim, Juho , booktitle=. Mind the Blind Spots: A Focus-Level Evaluation Framework for. 2025 , address=. doi:10.18653/v1/2025.emnlp-main.1805 , url=

work page doi:10.18653/v1/2025.emnlp-main.1805 2025

[13] [13]

Ryu, Hyun and Jang, Doohyuk and Lee, Hyemin S. and Jeong, Joonhyun and Kim, Gyeongman and Cho, Donghyeon and Chu, Gyouk and Hwang, Minyeong and Jang, Hyeongwon and Kim, Changhun and Kim, Haechan and Kim, Jina and Kim, Joowon and Kim, Yoonjeon and Lee, Kwanhyung and Park, Chanjae and Yun, Heecheol and Betz, Gregor and Yang, Eunho , journal=. 2025 , doi=

work page 2025

[14] [14]

2025 , address=

Garg, Madhav Krishan and Prasad, Tejash and Singhal, Tanmay and Kirtani, Chhavi and Mandal, Murari and Kumar, Dhruv , booktitle=. 2025 , address=. doi:10.18653/v1/2025.findings-emnlp.1120 , url=

work page doi:10.18653/v1/2025.findings-emnlp.1120 2025

[15] [15]

Xu, Zhijian and Zhao, Yilun and Patwardhan, Manasi and Vig, Lovekesh and Cohan, Arman , booktitle=. Can. 2025 , address=. doi:10.18653/v1/2025.acl-long.1009 , url=

work page doi:10.18653/v1/2025.acl-long.1009 2025

[16] [16]

Gao, Xian and Ruan, Jiacheng and Zhang, Zongyun and Gao, Jingsheng and Liu, Ting and Fu, Yuzhuo , journal=

work page

[17] [17]

2025 , address=

Idahl, Maximilian and Ahmadi, Zahra , booktitle=. 2025 , address=. doi:10.18653/v1/2025.naacl-demo.44 , url=

work page doi:10.18653/v1/2025.naacl-demo.44 2025

[18] [18]

From Replication to Redesign: Exploring Pairwise Comparisons for

Zhang, Yaohui and Zhang, Haijing and Ji, Wenlong and Hua, Tianyu and Haber, Nick and Cao, Hancheng and Liang, Weixin , booktitle=. From Replication to Redesign: Exploring Pairwise Comparisons for. 2025 , url=

work page 2025

[19] [19]

Nature Machine Intelligence , volume=

A Large-Scale Randomized Study of Large Language Model Feedback in Peer Review , author=. Nature Machine Intelligence , volume=. 2026 , doi=

work page 2026

[20] [20]

2025 , publisher=

Lou, Renze and Xu, Hanzi and Wang, Sijia and Du, Jiangshu and Kamoi, Ryo and Lu, Xiaoxin and Xie, Jian and Sun, Yuxuan and Zhang, Yusen and Ahn, Jihyun Janice and Fang, Hongchao and Zou, Zhuoyang and Ma, Wenchao and Li, Xi and Zhang, Kai and Xia, Congying and Huang, Lifu and Yin, Wenpeng , booktitle=. 2025 , publisher=

work page 2025

[21] [21]

Zou, Zhuoyang and Ansari, Abolfazl and Zhang, Delvin Ce and Lee, Dongwon and Yin, Wenpeng , journal=

work page

[22] [22]

Zhang, Daoze and Bao, Zhijian and Du, Sihang and Zhao, Zhiyi and Zhang, Kuangling and Bao, Dezheng and Yang, Yang , journal=

work page

[23] [23]

Wu, Sihong and Ma, Yiling and Zhao, Yilun and Hu, Tiansheng and Jiang, Owen and Patwardhan, Manasi and Cohan, Arman , journal=

work page