{"id":"482f3882-a399-4fb6-a1c3-5390d0e4f7a0","arxiv_id":"2506.17311","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An LLM-based review system processing 290 real submissions was far faster and cheaper than human review, yet agreed with the conference's acceptance decisions only 38.6% of the time, indicating LLMs should assist rather than replace human reviewers.","lead":"The authors built an automated paper-review system using GPT-4o with retrieval-augmented generation, multi-agent discussion, and chain-of-thought prompting, and tested it on 290 real conference submissions. The system reduced review time and cost, but its accepted-paper list overlapped with the conference's actual decisions only 38.6% of the time, so the paper recommends keeping human reviewers in charge.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Undefined similarity metric and absent baseline make the 38.6% headline uninterpretable; the estimate is also unstable (26.3–50.9% across five runs).","rationale":"Reader identified rubric mismatch as the weakest assumption; I find a more fundamental problem. The paper's core evidence, the 38.6% similarity, is undefined and unbounded. Section VI-A presents Table I but does not state the formula for similarity, and the paper does not report WASA's acceptance count or any comparison baseline. Even if the prompt rubric matched WASA's rubric perfectly, the number could not be interpreted: for a binary accept/reject decision, the expected overlap of random selectors is a function of the acceptance rate, and human-human agreement may be far from 100%. The five runs (26.32%, 35.08%, 38.60%, 42.11%, 50.88%) show high variance, so the point estimate is fragile. This is a concrete, testable deficiency: the repository and code can be inspected to recover the metric and the raw outputs. It does not invalidate the system's engineering contribution, but it means the central claim is currently under-supported. Since the missing analysis can be added in revision, the verdict should remain conditional.","tokens_in":11368,"tokens_out":8961,"duration_ms":85354,"concrete_test":"Inspect the public repository to recover the raw acceptance sets and the exact similarity function used for Table I; recompute FinalSimilarity using Jaccard and overlap coefficient, and compare each against a random-selection baseline constructed with WASA's true acceptance count (e.g., 1,000 permutations). If the observed similarity falls within the baseline's 95% interval or the two metric definitions shift the value by more than 10 percentage points, the 38.6% headline is not a reliable basis for the paper's conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim depends entirely on the similarity between LLM-accepted and WASA-accepted papers, yet Section VI-A and Table I never define the similarity measure. It could be Jaccard, overlap coefficient, or accuracy, and the interpretation changes with the metric. More importantly, no baseline is reported: the paper does not give WASA's acceptance rate, the expected agreement of random selection, or human-human re-review agreement, despite acknowledging that even the same human reviewers may produce different outcomes. Without such a baseline, 38.6% cannot be judged low or taken as evidence of LLM unreliability; it might even be above chance. The instability of the estimate compounds the problem: the five FinalSimilarity runs range from 26.32% to 50.88%, producing a wide confidence interval around the point estimate. Thus the headline statistic does not support the conclusion that LLMs are not yet suitable substitutes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents an automated paper-review system that combines Retrieval Augmented Generation (RAG), the AutoGen multi-agent framework, and chain-of-thought prompting to emulate reviewer and chair roles. The system was run five times on all 290 WASA 2024 submissions using GPT-4o with temperature set to 0. The reported results are an average review time of 2.48 hours and an average cost of $104.28 per run, with an average similarity of 38.6% between the LLM final acceptance set and the human-accepted set. The authors interpret this low similarity as evidence that LLM-based reviewers are not yet reliable substitutes for human reviewers, and they present two supplementary experiments on content understanding and susceptibility to exaggerated language. The paper concludes that LLMs should serve as assistive tools, with final decisions left to domain experts.","tokens_in":11505,"tokens_out":3644,"duration_ms":35690,"significance":"If the evaluation were properly anchored, this would be a useful feasibility study: it uses real conference data, releases the implementation as open source, provides concrete cost and time measurements, and identifies concrete failure modes such as content confusion and retrieval preference. The repeated runs and the use of an external ground truth (WASA 2024 decisions) are also strengths. However, the central quantitative claim is currently uninterpretable because the similarity metric is never defined and no baseline—such as chance agreement, human-human re-review agreement, or the venue acceptance rate—is provided. The qualitative findings are suggestive but rest on very small samples without statistical testing. With the metric and baseline supplied, the study could support its conclusion; as written, the evidence does not yet establish that 38.6% is low.","major_comments":[{"comment":"The similarity measure used for FirstRoundSimilarity and FinalSimilarity is never defined. The numbers could be a Jaccard index, an overlap coefficient, or an accuracy, and the interpretation of the 38.6% figure changes accordingly. Please define the metric explicitly and report the WASA 2024 acceptance rate, the expected overlap under random selection, and a human-human re-review agreement baseline; without these anchors the headline statistic cannot be evaluated.","section":"Section VI-A, Table I"},{"comment":"Across the five runs, FinalSimilarity ranges from 26.32% to 50.88%. Reporting only the average hides this instability, and with n=5 and no confidence interval or significance test, the claim that 38.6% is 'low' is not statistically grounded. The manuscript should report the variance and, if possible, a confidence interval or a test against a chance baseline.","section":"Section VI-A, Table I"},{"comment":"The susceptibility experiment uses one paper and five evaluations per condition; the 2.6-point difference between the original (85.8) and modified (88.4) versions is not tested for statistical significance. This is too thin to support Finding (III) as a general claim about LLM susceptibility to exaggerated language, and the paper should either provide more papers or explicitly frame this as an illustrative case study.","section":"Section VI-B, Experiment 3, Table II"},{"comment":"The review criteria used in the prompts are summarized from generic IEEE, Elsevier, Springer, and ACM guidelines, but the baseline consists of actual WASA 2024 decisions. If WASA used different criteria or different weights across criteria, the low similarity could reflect a rubric mismatch rather than an inherent deficiency in LLM judgment. The manuscript should compare the prompt criteria against the venue's actual review criteria or at least discuss this confound as a limitation.","section":"Section VI-B, Step 2"},{"comment":"The system's format-checking step automatically rejects papers that fail format compliance, but the manuscript does not report how many papers were removed in each of the five runs. This number affects the denominator of the similarity calculation and the comparability of the five runs, so it should be disclosed alongside Table I.","section":"Section VI-A, Algorithm 2"}],"minor_comments":[{"comment":"The conference name 'International Conference on Learning Representatives' should be 'International Conference on Learning Representations'.","section":"Section I, reference [1]"},{"comment":"In line 5, the notation 'P P+ = P ′ i' is undefined; please use set-union notation and define P_P before first use.","section":"Algorithm 2"},{"comment":"The color-coding of semantically similar responses is difficult to read in grayscale, and the five experimental conditions are not clearly labeled in the figure; please annotate the conditions directly or provide the responses in a table.","section":"Figure 3"},{"comment":"The conference name appears as 'W ASA' with a spurious space in several places; the spacing should be corrected to 'WASA'.","section":"Throughout the manuscript"},{"comment":"The conclusion states that the overlap 'averages only 38.6%,' but does not restate the range across the five runs (26.32%–50.88%); including the range would give readers a more accurate picture of the result's stability.","section":"Section VIII"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of cs.CY and the underlying idea is worth publishing if the evaluation can be made interpretable. For the editor: the main concern is that the 38.6% headline number is being used rhetorically without a defined metric or baseline; a revision that adds these elements and tightens the statistical claims would make the paper acceptable. No concerns about citation or novelty disclosure beyond what is already stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a genuine system-building effort: the authors integrate RAG, AutoGen, and chain-of-thought prompting into a working pipeline, run it on 290 real WASA 2024 submissions with GPT-4o, and release the code as open source. That is real, reproducible evidence and should be counted in its favor. Second, the headline statistic - 38.6% similarity between LLM-accepted and human-accepted papers - is not as interpretable as the paper presents it. The similarity metric is never defined, and no baseline is reported: no random-selection agreement, no human-human re-review agreement, not even WASA's acceptance rate. Without that, a number in the 30-40% range cannot be judged low or high. The paper itself acknowledges that human review is subjective and that even the same reviewers may differ, yet it still treats 38.6% as clear evidence that LLMs are not suitable substitutes. That step does not hold up. What is genuinely new: an integrated system applied to a real conference dataset, plus two supplementary analyses that give concrete failure modes. The content-confusion example - where the model answered a question about one paper using another paper's abstract - is a real and useful observation. The finding that the model's responses did not improve when given more content, and that it leaned mainly on the abstract, is also worth knowing. These results support the qualitative conclusion that LLMs are not yet trustworthy reviewers, even if the headline agreement number does not. The soft spots are significant. Besides the missing baseline and metric, the five runs of the system give FinalSimilarity values from 26.32% to 50.88%, so the point estimate is fragile. The supplementary experiments use one paper and five evaluations, with no error bars or statistical tests. There is also a criteria-mismatch risk: the prompt embeds generic IEEE/Elsevier/Springer/ACM guidelines, not WASA's actual rubric, so some disagreement could reflect a rubric mismatch rather than intrinsic LLM deficiency. The paper largely sidesteps this by framing human decisions as a valuable reference, which is honest but insufficient. Who this is for: anyone building or evaluating LLM-assisted review tools. It is a solid feasibility data point, not a benchmark. I would send it to a serious referee: the system and the failure-mode analyses deserve engagement, and the evaluation problems are fixable. A revised version that defines the metric, reports baselines, and tightens the supplementary experiments could be a useful contribution.","headline":"A credible feasibility study with a real open-source system and useful failure-mode analyses, but the headline 38.6% agreement lacks the defined metric and baseline needed to support its conclusion.","tokens_in":775,"tokens_out":974,"would_cite":false,"duration_ms":28949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tests an LLM-based review pipeline against real conference decisions and finds that, despite large savings in time and cost, its accept list overlaps the human accept list by only 38.6 percent.","keywords":["LLM paper review","peer review automation","retrieval augmented generation","multi-agent review system","chain-of-thought prompting","acceptance overlap","hallucination","reviewer bias"],"falsifier":"Re-run the identical pipeline on the same 290 papers with a prompt whose criteria are taken from the actual WASA 2024 review form, or from a rubric elicited from that year's program committee, and check whether the accept-list overlap rises substantially. If overlap jumps above, say, 70 percent, the 38.6 percent result was an artifact of the generic rubric rather than evidence that LLMs cannot judge papers; conversely, a similarly low overlap under rubric-matched conditions would confirm the paper's conclusion.","tokens_in":11194,"feed_emoji":"🤖","tokens_out":11177,"duration_ms":100382,"temperature":0.7,"pith_summary":"This paper asks whether an LLM-based system can take over academic peer review. The authors build a pipeline that combines retrieval-augmented generation, multiple interacting agents, and a seven-step chain-of-thought prompt, and run it on all 290 real submissions to a 2024 wireless-computing conference. The system is far cheaper and faster than human review—about 2.48 hours and 104 dollars for a full pass—but the papers it accepts overlap the conference's actual accept list by only 38.6 percent on average. Two follow-up experiments trace the gap to specific failure modes: the model confuses different papers and invents content when information is thin, and its scores rise when a paper carries exaggerated, data-free claims. The paper's conclusion is that LLM review should assist human reviewers, not replace them.","feed_headline":"LLM reviewers cut time to 2.48 hours and match humans 38.6 percent","feed_subtitle":"Study says to use LLMs to screen papers, not to decide accept or reject.","key_machinery":"The load-bearing artifact is the review pipeline itself. Each paper is converted to Markdown, indexed in a vector store, and queried through retrieval-augmented generation by a large language model cast as a reviewer or chair; a prompt decomposes the task into seven fixed steps from layout check through comparative evaluation, scoring, and explanation. A multi-agent orchestration layer runs reviewer agents in parallel on batches and then has chair agents deliberate over the shortlist. The argument rides on two things the pipeline produces: the final accept list, whose overlap with the venue's real accept list is the headline number, and the two controlled probes that isolate why the lists diverge—a content-ablation study on one paper and a sentence-injection study measuring score shift. The 38.6 percent overlap is the number that carries the conclusion.","core_discovery":"On the paper's own terms, the central discovery is a head-to-head comparison between an LLM review pipeline and human decisions on the same real corpus. Using GPT-4o with temperature set to zero, a first stage of reviewer agents followed by chair agents, and an evaluation rubric summarized from major publisher guidelines, the system produced an accept list that matched the WASA 2024 human accept list only 38.6 percent of the time on average over five runs; the first-round screening match was 59.6 percent. The paper argues this divergence is not random noise but the signature of two reproducible biases: when given only a paper's title, the model borrowed text from the wrong paper in the same batch or fabricated an answer, and adding flattering but data-free sentences to a paper's abstract and conclusion raised its average score from 85.8 to 88.4. Taking the human accept list as the authoritative reference, the paper concludes that LLM-based review cannot yet be trusted to decide what gets published.","pith_inferences":["A missing comparison would sharpen the verdict: if a fresh set of human reviewers re-reviewed the same 290 papers and also overlapped the original accept list by roughly 40 percent, the 38.6 percent figure would reflect the inherent noise of peer review as much as any LLM-specific weakness.","The content-ablation result—responses stayed nearly identical from abstract-only to full-paper input—suggests a cheap fix worth testing: feed the entire paper as one context instead of retrieved snippets, since the bottleneck may be the retrieval step's focus on abstract keywords rather than the model's reading ability.","The score inflation from exaggerated language implies a gaming risk: authors who know an LLM is screening papers could inflate scores with grandiose, data-free claims, so any real deployment would need prompts designed to resist such gaming.","The authors' proposed domain fine-tuning can be tested immediately on the same venue by training on earlier years' accept/reject decisions and measuring overlap on the 2024 corpus, since both the pipeline and the historical data already exist."],"forward_implications":["A full LLM review pass over 290 submissions costs about 104 dollars and takes under three hours, making the pipeline viable as an inexpensive pre-screening or triage stage before human review.","If LLM accept lists are used as final decisions today, they will diverge from human judgment by roughly 60 percent of the accepted set, so accept/reject authority should remain with human editors and program committees.","The documented failures—cross-paper content confusion when input is thin, and score inflation from exaggerated but data-free language—give concrete engineering targets, such as document-isolated retrieval and prompts that require whole-paper synthesis rather than keyword snippets.","Because the failure modes are tied to retrieval and prompt design, the open-source pipeline can be re-run with different models, rubrics, or retrieval structures to measure whether the 38.6 percent gap closes.","Venues with very large submission volumes would gain the most from the time and cost savings, provided the system is framed as triage rather than evaluation."],"supporting_citations":[{"why":"Supplies the chain-of-thought prompting method that breaks the review task into the seven-step sequence used by every agent.","marker":"[5]"},{"why":"Supplies the retrieval-augmented generation technique by which agents query paper content from a vector store.","marker":"[11]"},{"why":"Supplies the multi-agent orchestration framework used to coordinate reviewer agents and chair agents.","marker":"[14]"},{"why":"Supplies the PDF-to-Markdown conversion tool that prepares submissions for indexing and review.","marker":"[15]"},{"why":"Supplies the real corpus of 290 submissions and the human accept list used as the reference baseline.","marker":"[17]"},{"why":"Supplies the specific paper used in the content-ablation experiment demonstrating cross-paper content confusion.","marker":"[18]"}],"fun_headline_variants":["LLM reviewers match humans only 38.6% — use as aids, not judges","LLM and human accept lists agree only 38.6% — best as assistive tools","LLM review: 2.48 hours and $104 per paper, but 38.6% match","GPT-4o accepts only 38.6% of human-chosen papers — don't let it decide","LLM reviewers hallucinate titles, inflate scores with flattery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the review criteria loaded into the system prompt—summarized from general publisher guidelines—are the same criteria the WASA 2024 human reviewers actually applied; if the venue weighted novelty, experimental soundness, or topical fit differently, the low 38.6 percent overlap could reflect a rubric mismatch rather than a limit of LLM judgment.","fun_headline_variants_meta":{"raw":{"variants":["LLM reviewers match humans only 38.6% — use as aids, not judges","LLM and human accept lists agree only 38.6% — best as assistive tools","LLM review: 2.48 hours and $104 per paper, but 38.6% match","GPT-4o accepts only 38.6% of human-chosen papers — don't let it decide","LLM reviewers hallucinate titles, inflate scores with flattery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000908,"raw_usage":{"total_tokens":3902,"prompt_tokens":943,"completion_tokens":2959,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2839}},"tokens_in":559,"tokens_out":2959,"duration_ms":22016,"temperature":1.0,"reasoning_tokens":2839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:35:33.626941+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the identical pipeline on the same 290 papers with a prompt whose criteria are taken from the actual WASA 2024 review form, or from a rubric elicited from that year's program committee, and check whether the accept-list overlap rises substantially. If overlap jumps above, say, 70 percent, the 38.6 percent result was an artifact of the generic rubric rather than evidence that LLMs cannot judge papers; conversely, a similarly low overlap under rubric-matched conditions would confirm the paper's conclusion.","supporting_citations":[{"cited_title":"Autogen,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-agent orchestration framework used to coordinate reviewer agents and chair agents."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PDF-to-Markdown conversion tool that prepares submissions for indexing and review."},{"cited_title":"Wasa2024,","cited_arxiv_id":null,"evidence_quote":"Supplies the real corpus of 290 submissions and the human accept list used as the reference baseline."},{"cited_title":"Anonymity on byzantine-resilient decentralized computing,","cited_arxiv_id":null,"evidence_quote":"Supplies the specific paper used in the content-ablation experiment demonstrating cross-paper content confusion."}],"review_version":2}