REVIEW 3 cited by
ReviewEval: An Evaluation Framework for AI-Generated Reviews
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
ReviewEval: An Evaluation Framework for AI-Generated Reviews
read the original abstract
The escalating volume of academic research, coupled with a shortage of qualified reviewers, necessitates innovative approaches to peer review. In this work, we propose: 1. ReviewEval, a comprehensive evaluation framework for AI-generated reviews that measures alignment with human assessments, verifies factual accuracy, assesses analytical depth, identifies degree of constructiveness and adherence to reviewer guidelines; and 2. ReviewAgent, an LLM-based review generation agent featuring a novel alignment mechanism to tailor feedback to target conferences and journals, along with a self-refinement loop that iteratively optimizes its intermediate outputs and an external improvement loop using ReviewEval to improve upon the final reviews. ReviewAgent improves actionable insights by 6.78% and 47.62% over existing AI baselines and expert reviews respectively. Further, it boosts analytical depth by 3.97% and 12.73%, enhances adherence to guidelines by 10.11% and 47.26% respectively. This paper establishes essential metrics for AIbased peer review and substantially enhances the reliability and impact of AI-generated reviews in academic research.
Forward citations
Cited by 3 Pith papers
-
GraphReview: Scientific Paper Evaluation via LLM-Based Graph Message Passing
GraphReview models paper evaluation as LLM-driven message passing on a semantic paper graph that links intrinsic quality, contemporaneous papers, and prior work, then applies Personalized PageRank for ranking and revi...
-
Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review
Official conference reviewer guidelines improve LLM review scores' agreement with human ratings, while LLM-distilled 'reviewer-imitating' guidelines and rubric-style scoring do worse.
-
Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI
Peer review reports in AI conferences have grown longer and more standardized after LLMs, with increased emphasis on surface-level clarity and summaries at the expense of deeper critiques on originality and replicability.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.