Pith. sign in

REVIEW 8 cited by

What Can Natural Language Processing Do for Peer Review?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.06563 v1 pith:Q5ROMZW5 submitted 2024-05-10 cs.CL

Ilia Kuznetsov , Osama Mohammed Afzal , Koen Dercksen , Nils Dycke , Alexander Goldberg , Tom Hope , Dirk Hovy , Jonathan K. Kummerfeld
show 16 more authors
Anne Lauscher Kevin Leyton-Brown Sheng Lu Mausam Margot Mieskes Aurélie Névéol Danish Pruthi Lizhen Qu Roy Schwartz Noah A. Smith Thamar Solorio Jingyan Wang Xiaodan Zhu Anna Rogers Nihar B. Shah Iryna Gurevych
This is my paper · ORCID
classification cs.CL
keywords peerreviewhelpassistancecommunitylanguageprocessscientific
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review -- manuscripts, reviews, discussions -- are largely text-based, Natural Language Processing has great potential to improve reviewing. As the emergence of large language models (LLMs) has enabled NLP assistance for many new tasks, the discussion on machine-assisted peer review is picking up the pace. Yet, where exactly is help needed, where can NLP help, and where should it stand aside? The goal of our paper is to provide a foundation for the future efforts in NLP for peer-reviewing assistance. We discuss peer review as a general process, exemplified by reviewing at AI conferences. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance, illustrated by existing work. We then turn to the big challenges in NLP for peer review as a whole, including data acquisition and licensing, operationalization and experimentation, and ethical issues. To help consolidate community efforts, we create a companion repository that aggregates key datasets pertaining to peer review. Finally, we issue a detailed call for action for the scientific community, NLP and AI researchers, policymakers, and funding bodies to help bring the research in NLP for peer review forward. We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Gender and Discipline Shape Length, Content and Tone of Grant Peer Review Reports

    physics.soc-ph 2025-06 conditional novelty 7.0 of 10

    Grant peer review report length, content focus, and sentiment vary systematically by discipline and by the gender of reviewers and applicants.

  2. Causal Effect of Group Diversity on Redundancy and Coverage in Peer-Reviewing

    cs.DL 2024-11 conditional novelty 7.0 of 10

    Reviewer slates that are topically diverse, seniority-diverse, or from distinct publication networks produce broader or less redundant reviews, while geographic and organizational diversity show little or no effect.

  3. A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports

    econ.EM 2024-11 conditional novelty 6.0 of 10

    Fine-tuned transformer models classify Swiss National Science Foundation grant peer review sentences into twelve content categories with an average macro F1 of 0.85, enabling large-scale analysis of review reports.

  4. From peer review nuances to best practices

    cs.DL 2026-07 conditional novelty 5.0 of 10

    Paper version, score version, and input format vary across peer-review datasets and measurably affect LLM-based review experiments, so they should be reported explicitly.

  5. Are We Truly Innovating? A Qualitative and Quantitative Study of Originality in AI Research Papers

    cs.CL 2026-01 conditional novelty 5.0 of 10

    A fine-tuned LLM grounded in retrieved related papers predicts reviewer-style novelty scores and can flag paraphrased idea plagiarism, but the ground truth is itself LLM-generated.

  6. LLM4SR: A Survey on Large Language Models for Scientific Research

    cs.CL 2025-01 conditional novelty 5.0 of 10

    A systematic review of LLM-based systems for hypothesis discovery, experiment planning, scientific writing, and peer review, including benchmarks, evaluation methods, and open challenges.

  7. A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.

  8. Large language models for automated scholarly paper review: A survey

    cs.AI 2025-01 conditional novelty 3.0 of 10

    A survey of LLM-based automated scholarly paper review, cataloging models, datasets, methods, and publisher policies as of 2023-2024.

Pith tools