REVIEW 9 cited by
Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The peer review process in major artificial intelligence (AI) conferences faces unprecedented challenges with the surge of paper submissions (exceeding 10,000 submissions per venue), accompanied by growing concerns over review quality and reviewer responsibility. This position paper argues for the need to transform the traditional one-way review system into a bi-directional feedback loop where authors evaluate review quality and reviewers earn formal accreditation, creating an accountability framework that promotes a sustainable, high-quality peer review system. The current review system can be viewed as an interaction between three parties: the authors, reviewers, and system (i.e., conference), where we posit that all three parties share responsibility for the current problems. However, issues with authors can only be addressed through policy enforcement and detection tools, and ethical concerns can only be corrected through self-reflection. As such, this paper focuses on reforming reviewer accountability with systematic rewards through two key mechanisms: (1) a two-stage bi-directional review system that allows authors to evaluate reviews while minimizing retaliatory behavior, (2)a systematic reviewer reward system that incentivizes quality reviewing. We ask for the community's strong interest in these problems and the reforms that are needed to enhance the peer review process.
Forward citations
Cited by 9 Pith papers
-
No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions
Presentation-only revisions guided by AI feedback can boost AI reviewer scores by over 1 point on average with 75% success rate across tested systems.
-
ReVoicer: Conversational Voice Annotation for Human-Centered, LLM-Assisted Peer Review
ReVoicer is a prototype that turns spoken, in-the-moment reactions to a paper into cleaned, tagged annotations and a draft review aligned with the reviewer's own style.
-
PseudoBench: Measuring How Agentic Auto-Research Fuels Pseudoscience
PseudoBench shows current LLM agents produce persuasive pseudoscientific reports with near-zero refusal rates and at most 27.4% resistance.
-
MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility
MLReplicate benchmark evaluates six autonomous systems on 45 manuscripts from ICML 2025 papers, finding that automated reviews accept flawed outputs with fabricated claims while human review exposes methodological fai...
-
Position: Academic Conferences are Potentially Facing Denominator Gaming Caused by Fully Automated Scientific Agents
Malicious actors could use AI agents to submit large numbers of fake papers, inflating the submission count and thereby raising the acceptance odds for a small set of chosen legitimate papers under stable conference a...
-
Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review
An audit of 50,289 ICLR papers shows acceptance odds vary up to 8x across topics at equal reviewer scores, indicating scores are not comparable across research areas.
-
ReVoicer: Conversational Voice Annotation for Human-Centered, LLM-Assisted Peer Review
ReVoicer is a prototype annotation tool that cleans a reviewer's spoken/immediate reactions and drafts a rubric-aligned review using only the reviewer's own comments.
-
Toward an Engineering of Science: Rebalancing Generation and Verification in the Age of AI
AI lowers the cost of generating plausible scientific artifacts without lowering verification costs, so the paper proposes blueprints as typed graph components that decompose claims, evidence, and assumptions to enabl...
-
Analyzing the Effects of Two-Stage Peer Evaluation
Simulations indicate two-stage peer selection favors borderline agents in low-noise review settings and high-rank agents in high-noise settings, with strong sensitivity to parameters such as number selected and review...
Discussion (0). Sign in to comment.