A new benchmark of revised scientific paragraphs with human-written revision instructions shows that detailed instructions improve LLM revisions under reference-based similarity metrics compared with generic labels.
ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment -- especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4's ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
ParaRev: Building a dataset for Scientific Paragraph Revision annotated with revision instruction
A new benchmark of revised scientific paragraphs with human-written revision instructions shows that detailed instructions improve LLM revisions under reference-based similarity metrics compared with generic labels.