AI reviews for all 22,977 AAAI-26 papers were preferred by authors and PC members over human reviews on accuracy and suggestions and outperformed baselines at spotting weaknesses.
hub
Can large language models provide useful feedback on research papers? a large-scale empirical analysis
12 Pith papers cite this work, alongside 166 external citations. Polarity classification is still indexing.
12
Pith papers citing it
166
external citations · OpenAlex
hub tools
citation-role summary
background 4
citation-polarity summary
years
2026 12roles
background 4polarities
background 4representative citing papers
A survey of 457 SE researchers finds widespread GenAI use concentrated in writing and ideation, with productivity gains but persistent concerns over accuracy, bias, and the need for clearer governance rules.
LLMs given only research questions from 1000 arXiv CS papers recommend a narrower set of methods than the original papers, with effective model-entity diversity dropping from 1232 to 59-96 and stronger agreement among LLMs than with papers.
FOXGLOVE dataset of 2340 comments shows LLMs and instructors align on feedback goals and positions but diverge on sentence selection, with LLMs using more complex language and fewer questions and higher quality ratings driven by comment length.
PRISM benchmark finds LLMs match or exceed humans on isolated review dimensions like novelty verification but none achieve the balanced performance of human reviewers across depth, flaw prioritization, and constructiveness.
ARA uses LLMs to build workflow graphs linking sources, methods, and outputs in papers, then scores reproducibility, reaching ~61% accuracy on 213 ReScience C articles and outperforming priors on ReproBench and GoldStandardDB.
AI peer review systems are vulnerable to prompt injections, prestige biases, assertion strength effects, and contextual poisoning, as demonstrated by a new attack taxonomy and causal experiments on real conference submissions.
Introduces the Agentic Publication Protocol (APP) as a repository-based standard for publishing papers together with reproducibility artifacts and agent instructions.
Peer review reports in AI conferences have grown longer and more standardized after LLMs, with increased emphasis on surface-level clarity and summaries at the expense of deeper critiques on originality and replicability.
A multi-agent writing tutor for Overleaf that uses 12 agents and an expert skill library to generate inline comments, with a 14-user study reporting 90.6% actionable and 67.5% valid comments that outperform a GPT-5.2 baseline.
A cross-disciplinary review of 151 studies concludes LLMs accelerate research workflows while introducing recurring technical and ethical risks, including ten it flags as underexplored.
MIRAI predicts 5-year PageRank and citation impact from paper title/abstract/date with Spearman's ρ 0.47/0.62, and generates ideas judged 4:3 more impactful by LLM.
citing papers explorer
-
AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot
AI reviews for all 22,977 AAAI-26 papers were preferred by authors and PC members over human reviews on accuracy and suggestions and outperformed baselines at spotting weaknesses.
-
Taking a Pulse on How Generative AI is Reshaping the Software Engineering Research Landscape
A survey of 457 SE researchers finds widespread GenAI use concentrated in writing and ideation, with productivity gains but persistent concerns over accuracy, bias, and the need for clearer governance rules.
-
Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods
LLMs given only research questions from 1000 arXiv CS papers recommend a narrower set of methods than the original papers, with effective model-entity diversity dropping from 1232 to 59-96 and stronger agreement among LLMs than with papers.
-
FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays
FOXGLOVE dataset of 2340 comments shows LLMs and instructors align on feedback goals and positions but diverge on sentence selection, with LLMs using more complex language and fewer questions and higher quality ratings driven by comment length.
-
PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers
PRISM benchmark finds LLMs match or exceed humans on isolated review dimensions like novelty verification but none achieve the balanced performance of human reviewers across depth, flaw prioritization, and constructiveness.
-
ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review
ARA uses LLMs to build workflow graphs linking sources, methods, and outputs in papers, then scores reproducibility, reaching ~61% accuracy on 213 ReScience C articles and outperforming priors on ReproBench and GoldStandardDB.
-
When AI reviews science: Can we trust the referee?
AI peer review systems are vulnerable to prompt injections, prestige biases, assertion strength effects, and contextual poisoning, as demonstrated by a new attack taxonomy and causal experiments on real conference submissions.
-
Agentic Publication Protocol: An Attempt to Modernize Scientific Publication
Introduces the Agentic Publication Protocol (APP) as a repository-based standard for publishing papers together with reproducibility artifacts and agent instructions.
-
Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI
Peer review reports in AI conferences have grown longer and more standardized after LLMs, with increased emphasis on surface-level clarity and summaries at the expense of deeper critiques on originality and replicability.
-
PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf
A multi-agent writing tutor for Overleaf that uses 12 agents and an expert skill library to generate inline comments, with a 14-user study reporting 90.6% actionable and 67.5% valid comments that outperform a GPT-5.2 baseline.
-
From Text to Discovery: How Large Language Models Are Reshaping Research Across Scientific and Humanistic Disciplines
A cross-disciplinary review of 151 studies concludes LLMs accelerate research workflows while introducing recurring technical and ethical risks, including ten it flags as underexplored.
-
MIRAI: Prediction and Generation of High-Impact Academic Research
MIRAI predicts 5-year PageRank and citation impact from paper title/abstract/date with Spearman's ρ 0.47/0.62, and generates ideas judged 4:3 more impactful by LLM.