REVIEW 5 cited by
Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment
read the original abstract
In this paper we revisit the 2014 NeurIPS experiment that examined inconsistency in conference peer review. We determine that 50\% of the variation in reviewer quality scores was subjective in origin. Further, with seven years passing since the experiment we find that for \emph{accepted} papers, there is no correlation between quality scores and impact of the paper as measured as a function of citation count. We trace the fate of rejected papers, recovering where these papers were eventually published. For these papers we find a correlation between quality scores and impact. We conclude that the reviewing process for the 2014 conference was good for identifying poor papers, but poor for identifying good papers. We give some suggestions for improving the reviewing process but also warn against removing the subjective element. Finally, we suggest that the real conclusion of the experiment is that the community should place less onus on the notion of `top-tier conference publications' when assessing the quality of individual researchers. For NeurIPS 2021, the PCs are repeating the experiment, as well as conducting new ones.
Forward citations
Cited by 5 Pith papers
-
AgentReview: Exploring Peer Review Dynamics with LLM Agents
AgentReview is the first LLM-based simulation framework for peer review that quantifies a 37.1% decision variation attributable to reviewer biases.
-
ReviewGuard: Aligning LLM-Assisted Peer Review with Long-Term Scientific Impact
ReviewGuard aligns LLM peer reviews with future citations via impact-aligned RL, achieving Spearman ρ=0.776 on rejected-then-published AI/ML papers versus 0.492 for human reviewers and flagging 5.6× more high-impact cases.
-
The State of Peer Review in Empirical Software Engineering: A Community Survey on Review Load, Quality, and GenAI Use
A 120-respondent survey maps ESE community perceptions of review load, quality problems, LLM use in reviewing, and proposed system improvements.
-
Towards Automating Scientific Review with Google's Paper Assistant Tool
Presents PAT, an agentic AI review tool using inference scaling that claims 34% better math error recall on SPOT benchmark and successful pilots at STOC and ICML conferences.
-
Can We Volunteer Out of the Peer Review Crisis?
A game-theoretic model demonstrates a Nash equilibrium in which authors voluntarily accept random pre-review rejection to reduce reviewer burden and raise evaluation quality.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.