Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Generative Adversarial Reviews: When LLMs Become the Critic

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GAR, an LLM-agent review system, outperforms human reviews in head-to-head preference tests and predicts conference accept/reject outcomes with an F1 of up to 0.69, against a human-consistency baseline of 0.49.

desk verdict Novel LLM-agent review architecture with a solid prediction study, but the headline claim of beating human reviewers rests on a same-family judge and underreported human data. read the letter →

arxiv 2412.10415 v1 pith:Q67UXFLD submitted 2024-12-09 cs.CL cs.AI

Nicolas Bougie , Narimasa Watanabe This is my paper · ORCID
classification cs.CLcs.AI
keywords generativeagentreviewersLLM-basedpeerreviewgraph-basedmanuscriptrepresentationmemory-augmentedreviewerpersonaacceptancepredictionmeta-reviewerhuman-likenessofAIreviews
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative Agent Reviewers (GAR) is a proposed multi-agent LLM framework for simulating peer review. The paper's central claim is that GAR produces reviews that expert readers and a GPT-4-based judge prefer over reviews written by human reviewers, and that its meta-reviewer can predict whether a conference paper is accepted about as well as, and in the reported F1 scores better than, real reviewers predict each other. The system turns a manuscript into a graph of ideas, claims, technical details, and evidence, partitions that graph into themed communities, and has persona-equipped reviewer agents evaluate those communities over several rounds while consulting retrieved human reviews of similar content. On three major machine-learning conference datasets, GAR's accept/reject F1 reaches 0.69 with a balanced accuracy of 0.70, versus a human consistency baseline of 0.49. The motivation is practical: on-demand, near-expert feedback that does not depend on a researcher's network.

What carries the argument

The load-bearing object is a graph-based manuscript representation $G(p) = (E, R)$, in which extracted entities (ideas, claims, technical details, evidence) are nodes and typed relationships such as 'supports' or 'proves' are edges. Conceptually duplicate nodes are merged, the Leiden algorithm partitions the graph into communities of related content, and the LLM writes a community descriptor for each cluster; these descriptors become the unit of review. Around that graph, the architecture adds a profile module (persona traits such as strictness and focus areas inferred by contrastive comparison across reviews), a novelty module that queries external literature, a memory module that retrieves human reviews of similar communities and papers via embedding similarity, and a multi-round review loop in which later rounds incorporate retrieved exemplars and the agent's own prior critique. A meta-reviewer equipped with its own memory then produces the final decision.

What would settle it

Run a fully reported blind preference study with many expert evaluators and a judge model from a different family than the reviewer model; if GAR's win rate over human reviews falls to chance or reverses, the preference claim is refuted. Separately, compute GAR's accept/reject F1 on papers published after the LLM's training cutoff; a drop toward the human baseline would show the reported advantage came from memorized training data.

Watch

Extended reading notes

Core claim

The discovery the paper reports is that a structured, memory-augmented agent pipeline can reproduce, and even exceed, the observed consistency of human peer review. GAR first compresses each manuscript into a community-level graph, then runs three to six reviewer agents, each carrying a persona inferred from historical reviews by contrastive comparison, through an initial review followed by refinement rounds in which similar previously reviewed communities are retrieved from memory. A meta-reviewer then aggregates the individual reviews into a final accept/reject decision. In the paper's evaluations, GAR ranks first in both GPT-4-based and human pairwise preferences (Bradley-Terry scores 0.684 versus 0.523 for humans in the LLM evaluation, and 0.143 versus 0.112 in the human evaluation), and the threshold-based variant GAR> reaches F1 0.69 on ICLR 2023 acceptance prediction, surpassing the human consistency baseline of 0.49 from the NeurIPS experiment. The authors attribute these results to the graph representation, memory retrieval, and persona modeling; ablations without memory lose roughly 0.07–0.15 in F1.

Load-bearing premise

The central claim depends on trusting GPT-4's pairwise preferences as a measure of review quality, even though the reviews being judged were produced by the same model family, and the human-preference study does not report enough details to independently confirm the ranking.

Editorial extensions

If this is right

  • Authors could get structured, on-demand pre-submission feedback similar to what a competent reviewer would provide, without waiting for an invitation or paying for professional editing services.
  • Conference organizers could deploy GAR as a consistency audit, flagging decisions that deviate sharply from the agent's prediction so that human chairs can re-check borderline cases.
  • Because reviewer personas can be set to novice or expert, the framework provides a controlled way to study how expertise shapes review detail and acceptance likelihood.
  • The graph-community representation is the component the ablations credit for the gain, so the approach should transfer to other long-document understanding tasks, not only peer review.
  • If GAR's F1 advantage over the human baseline generalizes, automated systems can serve as a quantitative benchmark for evaluating new review processes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline preference numbers should be read cautiously: a GPT-4-based judge evaluating reviews written by GPT-4-family agents may prefer the same stylistic family, so a cross-model judge or a fully specified human study is needed to confirm that the win is about quality rather than style.
  • The acceptance-prediction F1 could partly reflect training-data leakage, since many conference papers and their public reviews may have appeared in LLM pretraining; testing on papers released after the model's cutoff would separate memorization from genuine evaluation skill.
  • If the leakage check passes, the same architecture could be repurposed for other structured document decisions, such as grant triage, patent prior-art assessment, or clinical study screening, where a community graph and past-decision memory are available.
  • A testable extension the authors do not run: measure whether GAR's novelty module rewards papers that later become influential, such as high-citation outliers, more than papers that were merely accepted, which would indicate whether it captures research impact or conference conformity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces GAR, a multi-agent LLM framework for simulating peer review. Reviewer agents are equipped with personas extracted from historical reviews, a graph-based manuscript representation built by LLM extraction and community detection, a memory module that retrieves similar community descriptors and associated reviews, and a multi-round refinement process; a meta-reviewer aggregates individual reviews into an accept/reject decision. Experiments compare GAR with AI-Scientist, OpenReviewer, ReviewerGPT, and AI-Review on three dimensions: pairwise preference rankings judged by GPT-4o and by human evaluators, acceptance-prediction F1 on NeurIPS/ICLR datasets, and additional ablations over personas, paper representations, reviewer expertise, and foundation models. The paper claims that GAR-generated reviews are preferred over human reviews, and that the GAR> variant reaches F1 scores up to 0.69, surpassing a human consistency baseline of 0.49.

Significance. If the central comparative claims were established, this would be a meaningful contribution: the architecture is concrete and modular, the experimental surface is broad (three datasets, five foundation models, multiple ablations), and the acceptance-prediction tables report error bars and t-tests. The graph-based representation, contrastive persona extraction, and memory-augmented multi-round review are sensible innovations that the field would want to build on. However, the headline claims are comparative and currently rest on a same-family LLM judge, an underreported human preference study, and a human baseline that is not measured on the same task; the significance of the paper therefore depends on revisions that make those comparisons trustworthy.

major comments (5)
  1. [Sec. 5.1, Table 1] The claim that GAR reviews are preferred over human reviews is based on GPT-4o pairwise judgments, while GAR's backend is GPT-4o-mini (Sec. 5, Implementation). Because the judge and the reviewed outputs come from the same model family, the Bradley-Terry advantage (GAR 0.684 vs Human 0.523) may reflect same-family judge bias rather than review quality. The paper should add judge-family controls (e.g., Llama or Mistral judges), report inter-judge agreement, and provide confidence intervals or significance tests for the BT coefficients.
  2. [Sec. 5.2, Table 2] The human preference study is reported without the number of evaluators, the number of pairwise comparisons, evaluator recruitment or selection criteria, or inter-rater agreement. The opening sentence of Sec. 5.1 ('five expert evaluators were given 200 papers') creates ambiguity about whether any human reading underlies Table 2. As reported, Table 2 cannot validate the GPT-4o judge or support the claim that GAR is preferred over human reviewers.
  3. [Sec. 5.3, Table 3] The human baseline (F1=0.49, balanced accuracy=0.66) is a fixed scalar from the NeurIPS 2021 consistency experiment, applied identically to the NeurIPS, ICLR 22, and ICLR 23 test sets. It is not computed on the same papers or under the same decision rule, so the statement that GAR's F1 is 'significantly higher' than 0.49 (p<0.002) is not a valid comparison; a t-test against a constant requires the baseline's distribution on the same task. The authors should compute a human-reviewer baseline on the same test sets or explicitly reframe the comparison as a reference point rather than a statistical improvement.
  4. [Sec. 5.3, GAR>] GAR>'s decision rule thresholds the review score at 6, which the text says is 'aligned with ICLR's review standards'. No evidence is given that this threshold was fixed before evaluation or selected on a validation set; if it was selected based on the test data, the reported 0.69 F1 is a calibrated decision rule rather than an uncalibrated prediction. The paper should report a threshold-selection procedure (e.g., a validation split) and ideally show performance across thresholds, while keeping the meta-reviewer-based GAR as the primary reported result.
  5. [Secs. 5.1-5.2] Neither preference ranking includes confidence intervals or significance tests. Given the unknown number of pairwise comparisons and the small margins (e.g., 0.143 vs 0.112 in Table 2), the ranking differences may be within noise. Report the number of comparisons, standard errors, and a statistical test (e.g., bootstrap or a Bradley-Terry model with uncertainty estimates).
minor comments (6)
  1. [Sec. 3.2.1] 'height core attributes' should be 'eight core attributes'.
  2. [Sec. 5.5] The text refers to 'RAG's effectiveness' in the opening of Sec. 5.5, but the framework is GAR; RAG is not defined anywhere.
  3. [Sec. 5.3] The text cites the 'NeurIPS 2023 consistency study' but the cited reference (Beygelzimer et al., 2021) is the NeurIPS 2021 consistency experiment; the year should be corrected.
  4. [Eq. (5)] The union notation in Eq. (5), '⋃_{k=1}^{K}(Rr,k-1, ...)', is mathematically odd; this should be a tuple or list of turns, not a set union.
  5. [Sec. 3.2.3] The 'mxbai-embed-large' embedding model is cited as Li & Li (2023) (Angle-optimized text embeddings), but mxbai-embed-large is a different model; please provide the correct citation.
  6. [Fig. 5] The caption for Fig. 5 mentions 'Method A' and 'Method B', but these methods are never defined in the text or caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GAR's reviews and acceptance predictions are generated outputs, not reconstructions of the target labels; the GAR> threshold is a fixed semantic decision rule, and the same-family LLM-judge concern is an evaluation-validity issue rather than a derivation-level reduction.

full rationale

Walking the derivation chain: GAR produces reviews through a pipeline (graph-based paper representation, persona initialization from historical reviews, memory-augmented multi-round LLM review, and a meta-reviewer), and acceptance prediction is either the meta-reviewer's own generated decision (GAR) or a thresholded rule at score 6 (GAR>). The threshold is justified by ICLR's 'Weak Accept' category, not by fitting to the test labels, so the reported F1 values are empirical evaluations of a decision rule rather than a fitted parameter renamed as a prediction. The GPT-4o judge is a measurement instrument; the fact that GPT-4o judges outputs from GPT-4o-mini may bias preference results, but that is a validity threat, not a reduction of the claimed result to its inputs by construction. Section 6 explicitly asks whether target papers are in the LLM training corpus; this is an acknowledged contamination/leakage risk, but the paper does not show that any ground-truth decision was fed into the generation procedure, so it does not constitute demonstrated derivation-level circularity. Self-citations such as the anonymous SimUSER reference are contextual and not load-bearing. The underreported human-preference study and the same-family judge issue belong under correctness risk and evaluation rigor, not under circularity. I therefore find no circular step.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The reported performance rests on several unverified assumptions: the GPT-4o judge proxy, the sufficiency of the graph representation, the OpenReview ground truth, and the manually chosen decision threshold. No new physical or mathematical entities are introduced.

free parameters (3)
  • Decision threshold for GAR> = 6
    Set in Sec. 5.3 to align with ICLR's 'Weak Accept' standard; converts review scores into accept/reject decisions and is used to report the best-performing variant.
  • Community similarity threshold tau (Eq. 3)
    Scalar threshold in the structural similarity function simstruct; determines when two community descriptors are considered similar for paper-level retrieval. Value not specified in the text.
  • Retrieval sizes M and K2
    Numbers of similar communities and meta-reviews retrieved in the review and meta-review modules; values are not reported but they control the amount of memory context injected into the agents.
assumptions (3)
  • domain assumption GPT-4o evaluations are a valid proxy for human judgments of review quality.
    Invoked in Sec. 5.1: 'As LLM Evaluators achieve comparable performance with human evaluators, we use GPT-4o to evaluate the generated reviews.' This is load-bearing for all win-rate results.
  • domain assumption The graph-based representation preserves the information necessary for faithful review.
    The framework assumes community descriptors, built from extracted entities and relations, are sufficient for agents to evaluate a paper. Table 5 tests this against raw text, but not against a full-text reviewer with equal context.
  • domain assumption Official OpenReview decisions are the correct ground truth for acceptance prediction.
    Used throughout Sec. 5.3 to compute balanced accuracy and F1 for the prediction task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative Adversarial Reviews: When LLMs Become the Critic." pith.science (2026). https://pith.science/paper/Q67UXFLD

@misc{pith2026241210415,
  author       = {Pith},
  title        = {Pith review of: Generative Adversarial Reviews: When LLMs Become the Critic},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q67UXFLD}},
  note         = {Machine review of arXiv:2412.10415}
}
read the original abstract

The peer review process is fundamental to scientific progress, determining which papers meet the quality standards for publication. Yet, the rapid growth of scholarly production and increasing specialization in knowledge areas strain traditional scientific feedback mechanisms. In light of this, we introduce Generative Agent Reviewers (GAR), leveraging LLM-empowered agents to simulate faithful peer reviewers. To enable generative reviewers, we design an architecture that extends a large language model with memory capabilities and equips agents with reviewer personas derived from historical data. Central to this approach is a graph-based representation of manuscripts, condensing content and logically organizing information - linking ideas with evidence and technical details. GAR's review process leverages external knowledge to evaluate paper novelty, followed by detailed assessment using the graph representation and multi-round assessment. Finally, a meta-reviewer aggregates individual reviews to predict the acceptance decision. Our experiments demonstrate that GAR performs comparably to human reviewers in providing detailed feedback and predicting paper outcomes. Beyond mere performance comparison, we conduct insightful experiments, such as evaluating the impact of reviewer expertise and examining fairness in reviews. By offering early expert-level feedback, typically restricted to a limited group of researchers, GAR democratizes access to transparent and in-depth evaluation.

Figures

Figures reproduced from arXiv: 2412.10415 by the authors.

Figure 1
Figure 1. Illustration of GAR framework. Generative reviewers aim to provide a frame￾work for automated paper review. The system generates numerical scores (soundness, presen￾tation, contribution, overall, confidence), lists weaknesses and strengths, and predicts the ac￾ceptance outcome (accept or reject). At the center of this approach, we present a graph-based representation of manuscripts, de￾signed to: 1) condense the doc… view at source ↗
Figure 2
Figure 2. Win rates between six types of reviews (five LLM-generated and a human reviewer) based [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Violin plots showing the distribution of scores generated by several AI-generated models [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Alignment Between Human and LLM Reviewer Scores. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Relative frequency of feedback aspects by LLM, human reviewers, Method A, and Method [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation

    cs.AI 2025-06 conditional novelty 6.0 of 10

    A large-scale LLM-driven urban simulator with recursive planning, memory, and belief modules, claimed to reproduce real-world time use, travel, and crowd patterns better than prior agent frameworks.

  2. Large language models for automated scholarly paper review: A survey

    cs.AI 2025-01 conditional novelty 3.0 of 10

    A survey of LLM-based automated scholarly paper review, cataloging models, datasets, methods, and publisher policies as of 2023-2024.

Reference graph

Works this paper leans on

33 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    Llm- deliberation: Evaluating llms with interactive multi-agent negotiation games

    Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Sch ¨onherr, and Mario Fritz. Llm- deliberation: Evaluating llms with interactive multi-agent negotiation games. arXiv preprint arXiv:2309.17234,

    arXiv

  2. [3]

    Accessed: 2024-10-25

    URL https://www.aje.com/arc/ peer-review-process-15-million-hours-lost-time/ . Accessed: 2024-10-25. Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Se- bastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, ...

    work page 2024

  3. [5]

    under review

    URL https://openreview.net/forum?id=rv9c1BqY0L. under review. Anonymous et al. Towards automated meta-review generation via an nlp/ml approach. Springer Journal of Digital Libraries, 2023a. Anonymous et al. Llm-as-a-judge & reward model: What they can and cannot do. arXiv preprint arXiv:2409.11239, 2023b. Anonymous et al. Can large language models provide...

    arXiv

  4. [7]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150,

    arXiv 2004

  5. [10]

    Can large language models be an alternative to human evalu- ations? arXiv preprint arXiv:2305.01937,

    Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evalu- ations? arXiv preprint arXiv:2305.01937,

    arXiv

  6. [11]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146,

    arXiv

  7. [12]

    From local to global: A graph rag approach to query-focused summariza- tion

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summariza- tion. arXiv preprint arXiv:2404.16130,

    arXiv

  8. [13]

    doi: 10.1162/coli a 00524

    ISSN 0891-2017. doi: 10.1162/coli a 00524. Chen Gao, Xiaochong Lan, Zhihong Lu, Jinzhu Mao, Jinghua Piao, Huandong Wang, Depeng Jin, and Yong Li. S 3: Social-network simulation system with large language model-empowered agents. arXiv preprint arXiv:2307.14984,

    arXiv 2017

Show all 33 references
  1. [14]

    Samir Haffar, Fateh Bazerbachi, and M

    URL https://arxiv.org/abs/2407.21783. Samir Haffar, Fateh Bazerbachi, and M. Hassan Murad. Peer review bias: A critical review. Mayo Clinic Proceedings, 94(4):670–676,

    arXiv

  2. [15]

    Agentreview: Exploring academic peer review with llm agents

    Yiqiao Jin et al. Agentreview: Exploring academic peer review with llm agents. arXiv preprint arXiv:2406.12708,

    arXiv

  3. [16]

    Angle-optimized text embeddings

    Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871 ,

    arXiv

  4. [17]

    Training socially aligned language models in simulated human society.arXiv preprint arXiv:2305.16960, 2023a

    22 Preprint Version Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush V osoughi. Training socially aligned language models in simulated human society.arXiv preprint arXiv:2305.16960, 2023a. Ryan Liu and Nihar B Shah. Reviewergpt? a...

    arXiv

  5. [18]

    Multi-modal molecule structure–text model for text-based retrieval and editing

    Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Animashree Anandkumar. Multi-modal molecule structure–text model for text-based retrieval and editing. Nature Machine Intelligence, 5(12):1447–1457, 2023b. Chris Lu, Samue...

    arXiv

  6. [19]

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein

    URL https://arxiv.org/abs/2303.08774. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. InProceedings of the 36th annual acm symposium on user interface soft...

    arXiv

  7. [20]

    Gpt4 is slightly helpful for peer-review assistance: A pilot study.arXiv preprint arXiv:2307.05492,

    Zachary Robertson. Gpt4 is slightly helpful for peer-review assistance: A pilot study.arXiv preprint arXiv:2307.05492,

    arXiv

  8. [21]

    A decade of knowledge graphs in natural language processing: A survey

    Phillip Schneider, Tim Schopf, Juraj Vladika, Mikhail Galkin, Elena Simperl, and Florian Matthes. A decade of knowledge graphs in natural language processing: A survey. arXiv preprint arXiv:2210.00105,

    arXiv

  9. [22]

    Real ml: Recognizing, exploring, and articulating limitations of machine learning research

    Jessie J Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, and Jennifer Wortman Vaughan. Real ml: Recognizing, exploring, and articulating limitations of machine learning research. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pp...

    2022

  10. [23]

    Two heads are better than one: A multi-agent system has the potential to improve scientific idea generation

    Haoyang Su, Renqi Chen, Shixiang Tang, Xinzhe Zheng, Jingzhe Li, Zhenfei Yin, Wanli Ouyang, and Nanqing Dong. Two heads are better than one: A multi-agent system has the potential to improve scientific idea generation. arXiv preprint arXiv:2410.09403,

    arXiv

  11. [24]

    Peer review as a multi-turn and long-context dialogue with role-based interactions

    Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi Ma, Zicheng Liu, and Stan Z Li. Peer review as a multi-turn and long-context dialogue with role-based interactions. arXiv preprint arXiv:2406.05688,

    arXiv

  12. [27]

    Digress: Discrete denoising diffusion for graph generation

    Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, V olkan Cevher, and Pas- cal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734,

    arXiv

  13. [28]

    Scimon: Scientific inspiration machines optimized for novelty

    Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259, 2024a. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye...

    arXiv

  14. [30]

    The rise and potential of large language model based agents: A survey

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864,

    arXiv

  15. [31]

    Is your paper being re- viewed by an LLM? investigating AI text detectability in peer review

    Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal, and Phillip Howard. Is your paper being re- viewed by an LLM? investigating AI text detectability in peer review. InNeurips Safe Generative AI Workshop 2024,

    2024

  16. [32]

    Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan

    URL https://openreview.net/forum?id=f2G7C2fKxV. Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan. Large language models for scientific synthesis, inference and explanation. arXiv preprint arXiv:2310.07984,

    arXiv

  17. [33]

    Can large language models transform computational social science? arXiv preprint arXiv:2305.03514 ,

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? arXiv preprint arXiv:2305.03514 ,

    arXiv

  18. [2017]

    Simulating social me- dia using large language models to evaluate alternative news feed algorithms

    Petter T ¨ornberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. Simulating social me- dia using large language models to evaluate alternative news feed algorithms. arXiv preprint arXiv:2310.05984,

    arXiv

  19. [2018]

    Anonymous

    doi: 10.18653/v1/N18-3011. Anonymous. SimUSER: When language models pretend to be believable users in recommender systems. In Submitted to The Thirteenth International Conference on Learning Representations,

    doi

  20. [2019]

    Ai-driven review systems: Evaluating llms in scalable and bias-aware academic reviews

    Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, et al. Ai-driven review systems: Evaluating llms in scalable and bias-aware academic reviews. arXiv preprint arXiv:2408.10365,

    arXiv

  21. [2020]

    The neurips 2021 consistency experiment,

    19 Preprint Version Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The neurips 2021 consistency experiment,

    2021

  22. [2021]

    Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao

    URL https://blog.neurips.cc/2021/12/08/ the-neurips-2021-consistency-experiment . Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. The values encoded in machine learning research. In Proceedings of the 2022 ACM Conference on Fairness...

    2021

  23. [2022]

    Autogen: Enabling next-gen llm applications via multi- agent conversation framework

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi- agent conversation framework. arXiv preprint arXiv:2308.08155,

    arXiv

  24. [2023]

    Researchagent: Iterative research idea generation over scientific literature with large language models

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738,

    arXiv

  25. [2024]

    Playing repeated games with large language models

    Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprint arXiv:2305.16867,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.