Pith. sign in

REVIEW 3 major objections 5 minor 42 references

AutoSupervision, a new benchmark built from 56,000 transparent peer-review records, argues that LLMs can characterize reviewer concerns well but fail at the critical step of verifying, with manuscript evidence, that a revision resolved them

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:20 UTC pith:W7MMBKR6

load-bearing objection New, useful benchmark for revision verification, but the core finding rests on LLM-generated gold labels with weak agreement—human validation needed before the bottleneck claim holds. the 3 major comments →

arxiv 2607.27845 v1 pith:W7MMBKR6 submitted 2026-07-30 cs.CL cs.AI

AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification

Haobo Li , Eunseo Jung , Wenxiao Zhao , Feng Liu , Jiong Wang , Kaiyi Xu , Zijie Guo , Zixin Chen
show 3 more authors
Ben Fei Fenghua Ling Lei Bai
This is my paper
classification cs.CL cs.AI
keywords AutoSupervisionscientific revision verificationpeer reviewLLM evaluationevidence groundingrevision episodeslanguage modelsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a new benchmark, AutoSupervision, can measure whether AI systems can verify that scientific manuscript revisions genuinely address reviewer concerns, using grounded evidence from the revised text. It argues that modern LLMs are already good at extracting and characterizing reviewer concerns (best score 0.754), but fail at the harder capability of verifying resolution and pointing to the exact evidence (best verification score 0.501). The authors build the benchmark from 56,000 article-review records, yielding 8,790 concern instances with resolution labels and evidence block annotations. If the benchmark is trustworthy, the practical consequence is that current AI research assistants can tell you what reviewers want but cannot yet close the feedback loop: they cannot reliably confirm that a revision actually fixed the problem.

Core claim

The central claim is that concern-level, evidence-grounded revision verification is a distinct and currently unsolved capability for LLMs. The paper's main empirical discovery is that the best model reaches only 0.501 on verification while characterization reaches 0.754, and that adding manuscript context and retrieval improves grounding but not verification to the same degree. The benchmark treats a revision episode—a post-revision manuscript paired with its review-response records—as the unit of evaluation, requiring models to output structured predictions for characterization, verification, and grounding.

What carries the argument

The central object is the revision episode: a pair containing the revised manuscript and the set of review-response records for that round. The benchmark defines three scored capabilities: characterization (comment kind, point type, target scope), verification (resolution label, paper status, resolution evidence blocks), and grounding (exact manuscript blocks supporting the concern). The load-bearing design choice is that resolution labels and evidence block IDs are treated as gold, produced by an LLM-assisted pipeline and checked only by inter-model agreement.

Load-bearing premise

The load-bearing premise is that the AI-generated resolution labels and evidence block IDs used as gold are correct; the paper's own agreement check shows low agreement on the key resolution-evidence signal, so if those labels are wrong, the measured 'bottleneck' is an artifact of the annotation, not of the models.

What would settle it

Take a random sample of the benchmark's 8,790 concern instances and have human experts independently label resolution status and evidence blocks; if human labels agree with the LLM-generated labels at low rates, especially on 'resolved' versus 'partially resolved', the central bottleneck conclusion is not a valid measure of model ability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Current LLMs can be used to extract and characterize reviewer concerns at high accuracy in this setup, with the best model reaching 0.754.
  • Evidence-based verification is the primary bottleneck: even the strongest model scores only 0.501, well below the characterization level.
  • Supervised fine-tuning on the benchmark substantially improves grounding (from 0.451 to 0.643) and overall performance (from 0.463 to 0.614).
  • Adding retrieval and agentic decomposition improves grounding for a small model (from 0.085 to 0.336), but verification remains low.
  • A closed-loop case study suggests that stronger verifiers provide more useful author-facing revision feedback in a two-revision cycle.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the benchmark is valid, a natural next step not spelled out in the paper is to use the verification and grounding scores as training signals for research agents, turning public peer-review records into self-supervision for revision quality.
  • The paper's own agreement check shows low inter-model agreement on the key resolution-evidence signal (micro-F1 0.441, kappa 0.261 for the resolution label); if human annotation were added, the definition of 'resolved' might need to be refined before verification scores can be trusted as absolute measures.
  • Because the corpus comes from accepted papers, unresolved concerns are rare (only 91 of 8,790 instances); a testable extension would sample papers with outcome diversity or include rejected manuscripts to see whether the verification bottleneck persists on harder revision trajectories.
  • One could directly test the benchmark's utility by comparing downstream manuscript quality after feeding models different verifier outputs, using human judges; the paper's case study is a small version of this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces AutoSupervision, a benchmark for concern-level, evidence-grounded verification of scientific manuscript revisions. Each instance contains a reviewer concern, the corresponding author response, and the revised manuscript; models must characterize the concern, determine whether it was resolved, and cite the supporting manuscript evidence. The benchmark is constructed from 56,000 Nature Communications transparent-review records, yielding 8,790 episode-level concern instances with resolution labels and evidence annotations. The authors evaluate a range of LLMs, an agentic pipeline, and a supervised fine-tuned model, reporting that characterization is relatively strong (GPT-5.5: 0.754) while verification remains the main bottleneck (best verification score 0.501, Claude-Opus-4.8). The paper concludes that current models struggle with evidence-based verification and grounding.

Significance. If the benchmark is valid, it addresses a genuinely underexplored capability in AI-assisted science: verifying whether feedback has been substantively incorporated into a revised manuscript. The task formulation is clean and the scale (56,000 source papers, 8,790 instances) is substantial. The paper provides a broad model comparison, an ablation study, an agentic baseline, and a supervised fine-tuning experiment, and it states a commitment to releasing data and code. The central weakness is the validity of the gold standard: labels and evidence annotations are produced by GPT-5.5 with no human validation, and inter-model agreement on the key evidence signal is low. Because the main conclusion—verification is the bottleneck—rests on these labels, the benchmark's usefulness depends on additional validation.

major comments (3)
  1. [§4.1, Appendix B.2, Table 8] The gold-standard resolution labels and resolution evidence block IDs are generated by GPT-5.5 without human validation. The only quality check is agreement with Claude Opus 4.8, and this agreement is low for the most load-bearing signal: resolution evidence micro-F1 is 0.441 and mean IoU is 0.308; resolution-label Cohen's kappa is 0.261 (although exact agreement is 0.848, which suggests the low kappa is partly an imbalance artifact). Since the paper's central claim—that evidence-based verification is the primary bottleneck (best verification score 0.501 vs. characterization 0.754)—is computed against these labels, noisy or systematically biased gold evidence would directly undermine that conclusion. The authors should provide human validation on a random subset, or at least analyze how label noise bounds the reported verification scores.
  2. [§4.1, Table 2] There is a circularity risk: GPT-5.5 is used to construct the gold labels (concern extraction, resolution labeling, evidence annotation) and is then evaluated against those same labels. Its high characterization score (0.754) and grounding score (0.702) may partly reflect self-consistency with its own annotation behavior rather than true capability. The agreement study with Claude Opus 4.8 mitigates this concern but does not remove it, because both models may share systematic biases. The concern set is also fixed to GPT-5.5's output, so the coverage and characterization results are conditioned on that model's definition of what counts as a concern. Releasing the annotation prompts and adding at least a small human-annotated verification set would address this.
  3. [§6, §4.1, Table 6] The dataset is constructed exclusively from accepted papers and further filtered to instances with auditable evidence, which likely overrepresents resolved concerns (8,265 of 8,790 instances are labeled resolved) and clean evidence structures. The paper acknowledges this limitation qualitatively, but it does not quantify how the filtering affects the difficulty or the reported bottleneck. For example, if the retained instances are those with explicit follow-up confirmations, the verification task may be artificially restricted, and the low verification scores could partly reflect label ambiguity rather than model deficiency. The authors should report the distribution of follow-up evidence types and assess whether the bottleneck conclusion is robust to excluding easy-to-ground instances.
minor comments (5)
  1. [§6] Typo: 'Natural Communication' should be 'Nature Communications'.
  2. [§5.7] The term 'ReviseBench-style supervisor' is not defined or referenced; please clarify the protocol or provide a citation.
  3. [Appendix A] The evaluation protocol states that concerns with the 'unverifiable' resolution label are excluded, but 'unverifiable' is not listed in Table 5's label options. Please align the schema and the protocol.
  4. [Appendix C] The characterization stage predicts 'response status' while the scored verification fields include 'paper status'; the relationship between these two fields should be clarified.
  5. [Appendix E] The reproducibility statement says the dataset and code are publicly released but no URL is provided in this version. Please include the link.

Circularity Check

2 steps flagged

Gold labels are GPT-5.5's own outputs with no human validation; GPT-5.5's headline scores and the central 'verification bottleneck' conclusion measure agreement with GPT-5.5's self-generated standard, whose resolution-evidence component is only 0.441 micro-F1 reproducible against another LLM.

specific steps
  1. self definitional [Section 4.1 (Concern Extraction), Figure 2 ('Golden information'), Section 4.2, Table 2 / Section 5.3]
    "For each sampled paper, we use GPT-5.5 (OpenAI, 2026) to process the published article and its peer review. ... The model extracts candidate concern threads, links each reviewer concern to its corresponding author response and follow-up discussion, and assigns the resulting review-response record to the revision episode in which the response was made."

    The gold labels for every scored component — resolution label, resolution-evidence block IDs, target paper block IDs, characterization taxonomy — are GPT-5.5's own construction-time outputs (Figure 2 'Golden information'), with no human validation reported anywhere; the only quality check is inter-model agreement (Section 4.2, Appendix B.2). GPT-5.5 is then scored on these same labels (Table 2), so its headline numbers (characterization 0.754, grounding 0.702) partly measure self-consistency between the evaluation run and its own earlier annotation run, not agreement with externally validated truth. Follow-up confirmations used in constructing the gold are withheld at evaluation, so verification is not trivially self-reproduction; still, the yardstick for all three components is one unvali

  2. other [Appendix B.2 (Agreement experiment), Section 5.3 (Verification), Table 8, Table 2]
    "Resolution evidence is harder because it can include several adjacent rebuttal, follow-up, and confirmation blocks in the review record; across all such blocks, agreement reaches 0.441 micro-F1 and 0.308 mean IoU. (Table 8: Resolution label 0.848 0.261 0.837)"

    The central claim — evidence-based verification is the primary bottleneck, with the best model reaching only 0.501 — is scored by C_ver = (F_res + F_status + F_evidence)/3, whose resolution-label and resolution-evidence gold is the least reproducible portion of the annotation scheme (Cohen's kappa 0.261; micro-F1 0.441 against a second LLM). Because that gold consists of GPT-5.5's own unreproducible judgments and no human ground truth exists, the measured 0.501 ceiling — and the characterization-vs-verification gap (0.754 vs 0.501) — is in part an artifact of one model's idiosyncratic evidence-block choices and follow-up interpretation, not purely a measurement of model incapability. The 'hardness' of verification is thereby partially constructed by the self-referential label set rather th

full rationale

The paper constructs a genuinely new benchmark (task definition in Section 3, transparent-review sourcing and episode alignment in Section 4, detailed evaluation protocol in Appendix A), and the construction is documented in enough detail that the derivation chain is inspectable. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via prior work; the related-work comparisons and the agreement experiment are reported transparently. The circularity concern is confined to ground-truth generation: GPT-5.5 both produces the gold annotations (Section 4.1) and is one of the evaluated models (Table 2), and the paper's own agreement study (Appendix B.2) shows the gold is model-dependent, with especially low reproducibility on exactly the resolution evidence/label components that drive the central 'verification is the primary bottleneck' conclusion. The paper deliberately withholds reviewer follow-up evidence from evaluation inputs, which makes the verification result a genuine transfer task and prevents the reduction from being total (hence score 5, not 8). All other models are also scored against the same self-generated gold, so the ranking among models remains meaningful as a relative measure of agreement with GPT-5.5's judgments. Nevertheless, the headline capability numbers and the bottleneck conclusion are not measurements against externally validated truth: they measure agreement with one unvalidated LLM's unreproducible annotation choices, and on the load-bearing signal that agreement between two LLM annotators is only 0.441 micro-F1 / 0.261 kappa (Table 8). The claim that LLMs 'can characterize concerns (0.754) but verification is the bottleneck (0.501)' is therefore in part an artifact of the self-referential, unvalidated gold standard, which is a partial circularity rather than a validated empirical finding.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The benchmark's design choices (scoring weights, taxonomy, retrieval budget) and its annotation assumptions (LLM-as-gold, follow-up-as-evidence, accepted-paper sample, PDF parsing fidelity) are the main load-bearing premises. No new physical or conceptual entities are introduced beyond the benchmark task itself.

free parameters (3)
  • Target-scope scoring weights (I_d, I_s, I_o) = 0.2, 0.4, 0.4
    Hand-chosen coefficients in the hierarchical target-scope score S_scope (Appendix A); no empirical justification; directly affects characterization scores.
  • Equal weights for overall score = 1/3 each
    The overall score is the unweighted average of characterization, verification, and grounding (Section 5.1); the choice is arbitrary and not justified from the data.
  • Agentic retriever candidate budget = top-12 blocks + 1-block neighbor window
    Appendix C sets the retrieval candidate budget; this is a hand-set hyperparameter that changes the agentic baseline's grounding and verification results.
axioms (5)
  • domain assumption Reviewer follow-up statements (e.g., 'This is now clear') are reliable evidence of concern resolution.
    Section 4.1 uses follow-up evidence as gold information during dataset construction. If follow-ups overstate resolution, the labels are wrong.
  • ad hoc to paper GPT-5.5-generated structured labels are a valid gold standard for concern type, resolution, and evidence.
    Section 4.1 uses GPT-5.5 for extraction and labeling; Appendix B.2 checks only agreement with another LLM, not with human annotation.
  • domain assumption The 20-category point-type taxonomy is complete and sufficient for characterizing scientific revision concerns.
    The taxonomy is author-defined (Appendix A, Table 5); no external validation of its coverage is provided.
  • domain assumption Accepted Nature Communications papers provide a representative sample of revision episodes.
    Section 6 acknowledges that the corpus overrepresents well-addressed concerns because only successful papers are included.
  • domain assumption MinerU PDF parsing faithfully reconstructs manuscript and review-record block structure.
    Section 4.1 relies on MinerU to produce the ordered block IDs that are the atomic units of grounding and evidence scoring; parsing errors propagate into all grounding metrics.

pith-pipeline@v1.3.0-daily-deepseek · 14544 in / 11368 out tokens · 93663 ms · 2026-08-01T00:20:47.436204+00:00 · methodology

0 comments
read the original abstract

Recent advances in large language models (LLMs) have enabled AI systems to assist scientific research and peer review. However, an essential capability for reliable AI-assisted scientific workflows remains underexplored: verifying whether reviewer feedback leads to meaningful and evidence-supported manuscript improvements. We introduce AutoSupervision, which evaluates whether scientific manuscript revisions genuinely address reviewer concerns through grounded evidence. AutoSupervision leverages transparent peer-review records as a natural source of supervision, where reviewer comments specify scientific concerns, author responses describe claimed resolutions, and revised manuscripts provide evidence of changes. Given reviewer comments, author responses, and revised manuscripts, models must characterize reviewer concerns, determine whether concerns have been addressed, and identify supporting manuscript evidence. We construct AutoSupervision from 56,000 Nature Communications articles and corresponding review records. Then we conducted experiments on LLMs, the ablation study, and the case study. Our results show that while LLMs perform well in characterizing reviewer concerns, with GPT-5.5 achieving a score of 0.754, evidence-based verification remains the primary bottleneck, with the best-performing model reaching only 0.501.

Figures

Figures reproduced from arXiv: 2607.27845 by Ben Fei, Eunseo Jung, Fenghua Ling, Feng Liu, Haobo Li, Jiong Wang, Kaiyi Xu, Lei Bai, Wenxiao Zhao, Zijie Guo, Zixin Chen.

Figure 1
Figure 1. Figure 1: AUTOSUPERVISION for scientific revision verification. Black loop: workflow without AUTOSUPERVISION. Orange loop: workflow with AUTOSUPERVISION: verify whether reviewer concerns are truly addressed. Revision verification poses challenges beyond lexical or semantic matching. Reviewer concerns may target experimental design, claim-evidence alignment, baseline coverage, figure clarity, sta￾tistical reporting, … view at source ↗
Figure 2
Figure 2. Figure 2: AUTOSUPERVISION construction and evaluation pipeline. First, we pair articles and peer reviews (A1), parse them into blocks with identifiers (A2), extract concerns (A3), and export revision episodes for concern characterization, resolution verification, and evidence grounding (A4). After building AUTOSUPERVISION, we benchmark existing LLMs on this task (B). These systems automate or assist the generation a… view at source ↗
Figure 3
Figure 3. Figure 3: Model performance across characterization, verification, and grounding. Gray segments [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 11 linked inside Pith

  1. [1]

    A Dataset of Peer Reviews ( P eer R ead): Collection, Insights and NLP Applications

    Kang, Dongyeop and Ammar, Waleed and Dalvi, Bhavana and van Zuylen, Madeleine and Kohlmeier, Sebastian and Hovy, Eduard and Schwartz, Roy. A Dataset of Peer Reviews ( P eer R ead): Collection, Insights and NLP Applications. Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Te...

    doi:10.18653/v1/n18-1149 2018

  2. [2]

    Annual Review of Information Science and Technology , volume =

    Bornmann, Lutz , title =. Annual Review of Information Science and Technology , volume =. doi:https://doi.org/10.1002/aris.2011.1440450112 , url =. https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/aris.2011.1440450112 , year =

    Pith/arXiv arXiv 2011

  3. [3]

    Research integrity and peer review , volume=

    The limitations to our understanding of peer review , author=. Research integrity and peer review , volume=. 2020 , publisher=

    2020

  4. [4]

    NLP eer: A Unified Resource for the Computational Study of Peer Review

    Dycke, Nils and Kuznetsov, Ilia and Gurevych, Iryna. NLP eer: A Unified Resource for the Computational Study of Peer Review. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.acl-long.277

    doi:10.18653/v1/2023.acl-long.277 2023

  5. [5]

    Neural Computing and Applications , volume=

    Moprd: A multidisciplinary open peer review dataset , author=. Neural Computing and Applications , volume=. 2023 , publisher=

    2023

  6. [6]

    MR e D : A Meta-Review Dataset for Structure-Controllable Text Generation

    Shen, Chenhui and Cheng, Liying and Zhou, Ran and Bing, Lidong and You, Yang and Si, Luo. MR e D : A Meta-Review Dataset for Structure-Controllable Text Generation. Findings of the Association for Computational Linguistics: ACL 2022. 2022. doi:10.18653/v1/2022.findings-acl.198

    doi:10.18653/v1/2022.findings-acl.198 2022

  7. [7]

    DISAPERE : A Dataset for Discourse Structure in Peer Review Discussions

    Kennard, Neha Nayak and O ' Gorman, Tim and Das, Rajarshi and Sharma, Akshay and Bagchi, Chhandak and Clinton, Matthew and Yelugam, Pranay Kumar and Zamani, Hamed and McCallum, Andrew. DISAPERE : A Dataset for Discourse Structure in Peer Review Discussions. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computation...

    doi:10.18653/v1/2022.naacl-main.89 2022

  8. [8]

    APE : Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning

    Cheng, Liying and Bing, Lidong and Yu, Qian and Lu, Wei and Si, Luo. APE : Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.569

    doi:10.18653/v1/2020.emnlp-main.569 2020

  9. [9]

    Plos one , volume=

    Peer review analyze: A novel benchmark resource for computational analysis of peer reviews , author=. Plos one , volume=. 2022 , publisher=

    2022

  10. [10]

    Automatic Analysis of Substantiation in Scientific Peer Reviews

    Guo, Yanzhu and Shang, Guokan and Rennard, Virgile and Vazirgiannis, Michalis and Clavel, Chlo \'e. Automatic Analysis of Substantiation in Scientific Peer Reviews. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023.findings-emnlp.684

    doi:10.18653/v1/2023.findings-emnlp.684 2023

  11. [11]

    arXiv preprint arXiv:2306.00622 , year=

    Reviewergpt? an exploratory study on using large language models for paper reviewing , author=. arXiv preprint arXiv:2306.00622 , year=

    Pith/arXiv arXiv

  12. [12]

    Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks

    Zhou, Ruiyang and Chen, Lu and Yu, Kai. Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024

    2024

  13. [13]

    NEJM AI , volume=

    Can large language models provide useful feedback on research papers? A large-scale empirical analysis , author=. NEJM AI , volume=. 2024 , publisher=

    2024

  14. [14]

    arXiv preprint arXiv:2405.02150 , year=

    The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates , author=. arXiv preprint arXiv:2405.02150 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2604.13940 , year=

    AI-assisted peer review at scale: The AAAI-26 AI review pilot , author=. arXiv preprint arXiv:2604.13940 , year=

    Pith/arXiv arXiv

  16. [16]

    Nature Communications , volume=

    Transparent peer review for all , author=. Nature Communications , volume=. 2022 , doi=

    2022

  17. [17]

    2026 , howpublished =

    Transparent Peer Review at. 2026 , howpublished =

    2026

  18. [18]

    arXiv preprint arXiv:2409.18839 , year=

    Mineru: An open-source solution for precise document content extraction , author=. arXiv preprint arXiv:2409.18839 , year=

    Pith/arXiv arXiv

  19. [19]

    Journal of the royal society of medicine , volume=

    Peer review: a flawed process at the heart of science and journals , author=. Journal of the royal society of medicine , volume=. 2006 , publisher=

    2006

  20. [20]

    Nature , volume=

    Scientific discovery in the age of artificial intelligence , author=. Nature , volume=. 2023 , publisher=

    2023

  21. [21]

    Nature , pages=

    A multi-agent system for automating scientific discovery , author=. Nature , pages=. 2026 , publisher=

    2026

  22. [22]

    arXiv preprint arXiv:2408.06292 , year=

    The ai scientist: Towards fully automated open-ended scientific discovery , author=. arXiv preprint arXiv:2408.06292 , year=

    Pith/arXiv arXiv

  23. [23]

    Agent Laboratory: Using LLM Agents as Research Assistants

    Schmidgall, Samuel and Su, Yusheng and Wang, Ze and Sun, Ximeng and Wu, Jialian and Yu, Xiaodong and Liu, Jiang and Moor, Michael and Liu, Zicheng and Barsoum, Emad. Agent Laboratory: Using LLM Agents as Research Assistants. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.320

    doi:10.18653/v1/2025.findings-emnlp.320 2025

  24. [24]

    2026 , howpublished =

    Using the. 2026 , howpublished =

    2026

  25. [25]

    2026 , month = apr, howpublished =

    2026

  26. [26]

    CSP aper Review: Fast, Rubric-Faithful Conference Feedback

    Cao, Lele and You, Lei and Team, R & D. CSP aper Review: Fast, Rubric-Faithful Conference Feedback. Proceedings of the 18th International Natural Language Generation Conference: System Demonstrations. 2025

    2025

  27. [27]

    2026 , howpublished =

    2026

  28. [28]

    2025 , howpublished =

    Jiang, Yixing and Ng, Andrew , title =. 2025 , howpublished =

    2025

  29. [29]

    Journal of the American Society for Information Science and Technology , volume=

    Peer review in a changing world: An international study measuring the attitudes of researchers , author=. Journal of the American Society for Information Science and Technology , volume=. 2013 , publisher=

    2013

  30. [30]

    arXiv preprint arXiv:2401.04259 , year=

    Marg: Multi-agent review generation for scientific papers , author=. arXiv preprint arXiv:2401.04259 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2510.08867 , year=

    ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review , author=. arXiv preprint arXiv:2510.08867 , year=

    arXiv

  32. [32]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Agentreview: Exploring peer review dynamics with llm agents , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  33. [33]

    arXiv preprint arXiv:2502.18864 , volume=

    Towards an AI co-scientist , author=. arXiv preprint arXiv:2502.18864 , volume=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2511.02824 , year=

    Kosmos: An ai scientist for autonomous discovery , author=. arXiv preprint arXiv:2511.02824 , year=

    Pith/arXiv arXiv

  35. [35]

    2026 , month = may, howpublished =

    Introducing. 2026 , month = may, howpublished =

    2026

  36. [36]

    2026 , month = jun, howpublished =

    2026

  37. [37]

    2026 , month = feb, howpublished =

    2026

  38. [38]

    2026 , month = may, howpublished =

    2026

  39. [39]

    arXiv preprint arXiv:2606.19348 , year=

    Deepseek-v4: Towards highly efficient million-token context intelligence , author=. arXiv preprint arXiv:2606.19348 , year=

    arXiv

  40. [40]

    2: Pushing the frontier of open large language models , author=

    Deepseek-v3. 2: Pushing the frontier of open large language models , author=. arXiv preprint arXiv:2512.02556 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2605.26494 , year =

    The. arXiv preprint arXiv:2605.26494 , year =

    Pith/arXiv arXiv

  42. [42]

    2024 , month = jul, howpublished =

    2024