Pith. sign in

REVIEW 1 cited by

Automatic Analysis of Substantiation in Scientific Peer Reviews

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.11967 v1 pith:XSQTWNVJ submitted 2023-11-20 cs.CL

Yanzhu Guo , Guokan Shang , Virgile Rennard , Michalis Vazirgiannis , Chloé Clavel This is my paper · ORCID
classification cs.CL
keywords peerreviewsconferencesdatasetqualitysubstanreviewsubstantiationanalysis
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the increasing amount of problematic peer reviews in top AI conferences, the community is urgently in need of automatic quality control measures. In this paper, we restrict our attention to substantiation -- one popular quality aspect indicating whether the claims in a review are sufficiently supported by evidence -- and provide a solution automatizing this evaluation process. To achieve this goal, we first formulate the problem as claim-evidence pair extraction in scientific peer reviews, and collect SubstanReview, the first annotated dataset for this task. SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts. On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews. We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports

    econ.EM 2024-11 conditional novelty 6.0 of 10

    Fine-tuned transformer models classify Swiss National Science Foundation grant peer review sentences into twelve content categories with an average macro F1 of 0.85, enabling large-scale analysis of review reports.

Pith tools