{"id":"6b142fd5-8640-4cb8-b32e-cc193f72415c","arxiv_id":"2607.21920","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-module multi-agent LLM system with human review screens trial registries and extracts endpoints, reproducing a published meta-analysis and flagging additional eligible trials.","lead":"This paper proposes two multi-agent AI systems to screen clinical trials and extract data for systematic reviews, with humans checking uncertain cases. A test on a published colorectal cancer meta-analysis recovered all original trials and identified extra eligible studies missed by the earlier manual review.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NMA update's 'updated clinical conclusions' rest on an unnamed NMA model and unverified HR inputs; the claim that MAS changes treatment rankings is not yet tied to a reproducible analysis.","rationale":"The reader's verdict is CONDITIONAL and correctly flags undisclosed human correction, missing single-LLM extraction baseline, and NMA internal inconsistencies. My independent pass identifies the most load-bearing concern as the NMA update's reproducibility: the 'updated clinical conclusions' in the abstract and Section 4 are produced by an unspecified statistical model applied to HRs that largely come from IPD reconstruction, not from the system being proposed. This is a correctness risk, not a disagreement with consensus: the methods section states the reconstruction tools are used without reporting validation accuracy on these particular curves, and the appendix itself concedes that pooling biomarker-selected trials may inflate P-scores. The screening experiments are the strongest part of the paper: within-model MAS vs single-agent comparisons on thousands of trials with 30 runs show consistent improvements, and the human-label caveat the reader identifies is real but moderate (Fleiss Kappa 0.70 with consensus resolution). The extraction results are plausible but the headline '97% accuracy after correction' depends on human review of flagged cells, which is disclosed. The concrete test I propose, rerunning the NMA from the released inputs with a standard model, would settle whether the headline result survives contact with an independent statistical analysis. Assuming the authors provide the model specification and inputs, the paper's central engineering claims remain conditionally acceptable; hence my recommendation is unchanged CONDITIONAL rather than REJECT or UNVERDICTED.","tokens_in":24470,"tokens_out":1615,"duration_ms":13956,"concrete_test":"Ask the authors to release the complete NMA input table (trial, arm, treatment, sample size, HR, 95% CI or reconstruction source) and the exact NMA specification (model, effect type, priors, software). Independently rerun the NMA on the 36-trial network with a standard frequentist random-effects consistency model and compare P-score ranks and HR estimates for Bev+FOLFOXIRI vs Bev+FOLFOX and Bev+FOLFIRI. Also rerun using only directly reported HRs; if rankings of the top arms change, the 'updated clinical conclusions' are not robust to the HR reconstruction step.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that the system 'recovers all trials... and identifies additional eligible trials missed by manual review, leading to updated clinical conclusions' depends on the NMA in Section 4/C.2. The paper never specifies the NMA model (frequentist vs Bayesian, fixed vs random effects, consistency model, prior specification), yet reports P-scores and HRs with credible intervals. Without the model, the updated conclusions cannot be reproduced or audited. The harm is concrete: the newly reported rankings are driven by 17 extra trials and 13 of 29 original trials whose OS HRs were not directly reported but estimated via SynthIPD/IPDfromKM from Kaplan-Meier curves (Appendix C.1). The extraction MAS itself is not used for these HRs; the paper states 'we estimated them by reconstructing individual patient data with methods IPDfromKM... or SynthIPD'. Thus the headline conclusion rests on secondary reconstruction methods whose validation against the MAS workflow is absent. Moreover, Section C.2 notes P-scores may be inflated because RAS wild-type restricted trials are pooled with unselected populations, acknowledging a possible statistical artifact. The screening and extraction evaluations may be sound, but the 'updated clinical conclusions' claim is a network analysis performed outside the MAS, with unspecified methods.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two multi-agentic LLM systems (MAS) for systematic literature review: a screening MAS that combines heterogeneous personas, multi-round cross-review, and human-in-the-loop adjudication, and an extraction MAS that standardizes target fields, iteratively corrects extractions, assigns confidence labels, and uses retrieval-based context control. The screening system is evaluated on a 2,245-trial NSCLC benchmark with 30 replicates, comparing each MAS against a same-base-model single-agent baseline under two adjudication rules. The extraction system is evaluated on 804 entries from TNBC, NSCLC, and gastric cancer trials. As a real-world application, the authors reproduce a published network meta-analysis (NMA) of first-line metastatic colorectal cancer, claim to recover all 29 original trials, identify 17 additional eligible trials, and report updated clinical conclusions.","tokens_in":24676,"tokens_out":6662,"duration_ms":58155,"significance":"If the screening and extraction results hold, the paper makes a useful contribution to semi-automated systematic reviewing. The screening experiments are well designed in an important respect: the MAS is compared with a single-LLM baseline using the same base model, so gains are attributable to the multi-agent workflow rather than to a stronger foundation model. The 30-run replication design, the reporting of majority-vote (Rule A) results without human correction, and the explicit prompt library in the appendix all strengthen the credibility of the screening comparison. The extraction confidence-label scheme is also a practical contribution, and the claim that human-review flags concentrate errors is valuable for workload estimation. However, the headline claim of 'updated clinical conclusions' from the reproduced NMA is not currently reproducible or auditable, and the 'uniform improvement' wording is stronger than Table 3 supports. These issues are load-bearing for the abstract and conclusion but appear fixable within the manuscript's scope.","major_comments":[{"comment":"The conclusion that 'MAS uniformly improves accuracy' is not supported by the metrics reported in Table 3. For GemF, Rule A gives sensitivity 0.988 versus 0.991 for the single agent and WSS 0.865 versus 0.866. For Hope, Rule A ties sensitivity at 0.941 and slightly lowers WSS (0.824 versus 0.825). For DS, Rule A lowers specificity from 1.000 to 0.995 and PPV from 0.997 to 0.941. If 'uniform improvement' refers only to the accuracy column, the text should say so; otherwise the claim should be revised or the trade-offs explicitly discussed.","section":"Section 5 and Table 3"},{"comment":"The abstract's claim that the system leads to 'updated clinical conclusions' is not reproducible from the manuscript. Section 4.2 reports P-scores and hazard ratios without specifying the NMA model, estimation method, prior distributions, consistency assumptions, or software. Appendix C.1 further states that many overall-survival hazard ratios were estimated by IPDfromKM or SynthIPD from Kaplan-Meier curves rather than by the extraction MAS, and Appendix C.2 explicitly acknowledges that P-scores for biomarker-selected regimens 'may be inflated' because such trials are pooled with unselected populations. The authors should provide the full NMA specification, the reconstructed hazard-ratio inputs, and sensitivity analyses excluding or stratifying reconstructed and biomarker-restricted data; otherwise the 'updated clinical conclusions' claim should be removed or substantially weakened.","section":"Section 4.2 and Appendix C.2"},{"comment":"The statement that 'our extraction MAS with the same LLM model recovers all of Table 2 correctly' is not demonstrated. The one-shot baseline in Table 2 uses Gemini 3.1 Pro on the IPSOS trial, while the MAS extraction evaluation in Section 3.2 uses GemF, Hope, and DS on different trials and endpoints. No controlled MAS-versus-SAS extraction comparison on the same inputs is reported. The authors should either add a like-for-like comparison (same model, same trials, same query) or explicitly label the Table 2 contrast as an illustrative example rather than an experimental result.","section":"Section 2.3 versus Section 3.2"}],"minor_comments":[{"comment":"The text states that 2,245 trials were screened, but the Table 3 caption reports n = 2,236; these numbers should be reconciled.","section":"Section 3.1 and Table 3"},{"comment":"Fleiss' Kappa of 0.70 is described as 'moderate'; standard conventions usually classify values in 0.61-0.80 as substantial, so the wording should be adjusted or justified.","section":"Section 3.1"},{"comment":"Table 5 contains 19 rows while the text says 17 newly identified studies; the table should make explicit that some trials contribute multiple treatment comparisons and should report a unique-study count.","section":"Table 5"},{"comment":"The sentence '13 (resp. 14) included overall survival results and included in our analysis' is grammatically unclear and should state the counts for original and newly identified trials separately.","section":"Appendix C.1"},{"comment":"The data availability statement names only public data sources, while the CRC answer sheet in Appendix A.3.1 is 'available upon request'; the authors should state where the MAS workflow code, prompts, and analysis scripts will be deposited.","section":"Data availability statement"}],"recommendation":"major_revision","confidential_remarks":"The screening and extraction experiments are the strongest part of the paper and are within the scope of stat.AP. My main concern is that the NMA update functions as a headline result in the abstract but lacks the methodological detail needed for audit. The authors should either fully specify the NMA and its reconstructed inputs or reposition it as an illustrative application with clearly scoped claims. The 'uniform improvement' language also needs to be made metric-specific. These are fixable, so I am not recommending rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on Ren et al.\n\nThe screening module is the genuine contribution. Same-base-model SAS-vs-MAS comparisons, 30 replications, and the F1/sensitivity gains are consistent—DS sensitivity jumps from 0.583 to 0.926 with little PPV loss under Rule B. That's a real, useful result. The extraction confidence-labeling scheme is also sensible: 90.4% of high-confidence cells matched the source exactly, and the flagged error patterns (low-res figures, cut-off ambiguity) are the ones you'd want human eyes on. The reproduction effort—screening 1,784 registry records, recovering all 29 original NMA trials, finding 17 additional candidates—shows real work.\n\nBut the claims run ahead of the evidence in three places.\n\nFirst, 'uniformly improves accuracy' is not what Table 3 shows. DS Agentic-A's PPV drops from 0.997 to 0.941, and GemF Agentic-A has a slight WSS decrease. The improvements are consistent for F1 and MCC, but not uniform across metrics. Easy fix: soften the claim.\n\nSecond, the 97% extraction accuracy after correction is a human-in-the-loop number, not an autonomous one. The paper doesn't report how many cells actually required review, nor the fully autonomous pre-correction accuracy. Without that, you can't fairly compare against a single-LLM baseline on the same data.\n\nThird, the NMA reproduction—the headline 'updated clinical conclusions' claim—is not auditable. The statistical model is never specified: frequentist or Bayesian, fixed or random effects, consistency assumptions, priors, software, nothing. And the HRs driving those conclusions were not extracted by the MAS; 13 of 29 original trials and most of the 17 new ones had OS HRs estimated via IPDfromKM/SynthIPD from Kaplan-Meier curves, external methods with no validation in this workflow. The paper itself admits in Appendix C.2 that P-scores may be inflated because RAS wild-type restricted trials are pooled with unselected populations. As written, the NMA results cannot be reproduced or checked.\n\nAlso, no code or full benchmark data are released, despite a data-availability statement that just points to public sources. For a systems paper, that's a real gap.\n\nThe benchmark labels rest on initial Fleiss Kappa 0.70 before consensus, but that's a standard limitation and doesn't sink the within-model screening comparison.\n\nMy bottom line: this deserves a serious referee, but the NMA section needs to be reframed as a scaled screening/extraction demonstration, not source of updated clinical conclusions, and the model must be specified. I'd send it to review with that as a required revision.","headline":"The screening MAS evidence is solid and worth a look, but the network-meta-analysis heading is not reproducible as reported and the 'uniform improvement' claim overstates Table 3.","tokens_in":25252,"tokens_out":3477,"would_cite":true,"duration_ms":31670,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62P10","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Two multi-agent systems with human-in-the-loop can reproduce a published network meta-analysis and add 17 eligible trials that change treatment rankings.","keywords":["multi-agent systems","systematic literature review","clinical trials screening","data extraction","human-in-the-loop","network meta-analysis","large language models","evidence synthesis"],"falsifier":"Re-run the NSCLC screening set with an independent panel of reviewers who do not see the original consensus labels, then compare the panel's final decisions with the MAS decisions; if the panel labels a substantial fraction of the MAS's high-confidence inclusions or exclusions as errors, or if the sensitivity gain from 0.58 to 0.93 reverses under the new labels, the uniform-improvement claim fails. A cheaper check is to apply the full pipeline to a completed Cochrane or similar review with published decisions and extracted data and count exact agreement on both screening and numeric extraction.","tokens_in":24242,"feed_emoji":"🩺","tokens_out":7297,"duration_ms":61977,"temperature":0.7,"pith_summary":"This paper tries to establish that a systematic literature review of clinical trials can be largely automated without losing the accuracy a regulator would demand, by splitting the work into two purpose-built multi-agent systems with a human reviewing only flagged cases. The screening system claims to beat a single-language-model baseline uniformly across three base models, converting a conservative model's sensitivity from 0.58 to 0.93 while keeping precision high. The extraction system claims near-exact capture of survival and response outcomes across 804 cells, with a confidence label that routes the genuinely uncertain cases to humans. The decisive test is a published network meta-analysis (a statistical synthesis of many treatments) in metastatic colorectal cancer: the system reproduces all 29 original trials and adds 17 eligible trials that the manual review missed, changing the estimated treatment rankings. The point, if true, is that study-selection bias in evidence synthesis can be reduced by a reproducible, audit-friendly machine workflow.","feed_headline":"Agentic review finds 17 trials that manual review missed","feed_subtitle":"Two multi-agent systems with a human reviewer reproduced a published meta-analysis and updated its conclusions.","key_machinery":"The load-bearing mechanism is the separation of screening and extraction into two fit-to-purpose multi-agent systems. Screening uses N agents with heterogeneous personas (strict regulator, permissive clinician, statistician, clinical pharmacologist) plus an inspector that generates targeted follow-up questions and a result agent; consensus finalizes, while disagreement and 'maybe' go to a human. Extraction uses specialized treatment, subgroup, and endpoint agents to build a standardized mock table, an extraction agent that proposes values and re-reads the source document for T rounds, assigns High, Medium, or Low confidence from cross-round consistency, and retrieves only matching evidence clips to control token cost. These mechanisms convert individual-model failure modes, such as over-conservative screening, hallucinated numbers, and missed subgroups, into measurable disagreement signals that a human can audit.","core_discovery":"The central claim is that a multi-agent architecture, rather than any one model, is what makes LLM-based systematic review trustworthy enough for clinical use. Screening is performed by several agents with deliberately different personas who vote and then answer targeted follow-up questions from an inspector until consensus or a turn limit; the decision rule sends disagreement, 'maybe,' and externally discovered trials to a human. In 30 Monte Carlo runs on 2,236 non-small-cell lung cancer trials, this raises F1 from 0.736 to 0.955 for the most conservative base model and improves F1 for all three tested models over their single-agent baselines. Extraction is similarly decomposed into standardization, iterative correction with confidence labels, and retrieval-based context control; on 804 endpoint cells across three cancers, 92.2% are high-confidence, 90.4% of those exactly match the source, and correcting flagged cells raises overall accuracy to about 97%. Reproducing a published metastatic colorectal cancer network meta-analysis, the system recovered all 29 originally included trials (seven of them outside the ClinicalTrials.gov knowledge base), surfaced 17 additional eligible studies, and the updated network changes the relative standing of bevacizumab-based combinations.","pith_inferences":["Editorial extension: because the evaluation is oncology-only and tied to registry-based NCT identifiers, the strongest untested check is a cross-specialty gold-standard benchmark, such as a completed Cochrane review in cardiology or infectious disease, where inclusion criteria are structured differently.","Editorial extension: the paper recovers pre-2007 and non-NCT trials through external web-search agents rather than through the knowledge base itself; a natural next test is whether the screening MAS can ingest those registries natively and sustain the same recall gain.","Editorial extension: the confidence-label mechanism could be validated prospectively as a triage instrument by recording, in new disease areas, whether human correction rates rise monotonically from High to Medium to Low confidence cells.","Editorial extension: the 17 newly identified trials in the colorectal cancer reproduction come from a single published review; re-running the same pipeline on several independent network meta-analyses would show whether the missed-trial rate is typical of manual reviews or specific to this benchmark."],"forward_implications":["Screening time can decouple from the number of studies because agents process the knowledge base in parallel, approaching the latency of a single inference cycle rather than a linear per-paper cost.","The architecture is model-agnostic, so stronger underlying language models, retrieval engines, or image-recognition tools should improve both modules without redesign.","The confidence labels plus source-risk flags concentrate human audit on roughly 10% of extracted data points while catching most errors, making the workload of human oversight explicit and learnable.","If the reproduction result holds, published network meta-analyses can be revisited cheaply as new trials appear, and evidence syntheses used in regulatory and clinical decisions can be kept current.","The uniform screening improvement over single-agent baselines suggests that agent disagreement itself is a signal worth routing to human review, not just a voting byproduct."],"supporting_citations":[{"why":"Supplies the published network meta-analysis that the real-world application reproduces and extends, including the original 29 trials and the study-selection criteria used as query input.","marker":"Xu et al. [2021]"},{"why":"Provides the IPSOS trial whose full publication is used to demonstrate one-shot single-LLM extraction failure in Table 2.","marker":"Lee et al. [2023]"},{"why":"Documents the time and worker cost of systematic reviews, motivating the parallelization and automation claims.","marker":"Borah et al. [2017]"},{"why":"Supports the premise that manual screening is prone to missed studies and inconsistent decisions.","marker":"Bannach-Brown et al. [2019]"},{"why":"Supports the premise that manual data extraction is subject to human error and inconsistency.","marker":"Jonnalagadda et al. [2015]"},{"why":"Provides the PRISMA 2020 reporting guideline that shapes the workflow's PRISMA flow diagram and reporting of the reproduced review.","marker":"Page et al. [2021]"},{"why":"Supplies the self-consistency idea that underlies the extraction MAS's cross-round confidence labels.","marker":"Wang et al. [2022]"},{"why":"Motivates the heterogeneous-persona design for screening agents as a mitigation of LLM mode collapse.","marker":"Zhang et al. [2025b]"},{"why":"One of the methods used to reconstruct individual patient data from Kaplan-Meier plots when hazard ratios are not reported.","marker":"Liu et al. [2021]"},{"why":"The SynthIPD method used for reconstructing hazard-ratio estimates from vector-graphics survival plots in the reproduced network meta-analysis.","marker":"Zhao et al. [2025]"}],"fun_headline_variants":["Agentic review catches 17 trials manual screening missed","Multi-agent system updates clinical trial evidence base","AI agents with human oversight find 17 omitted studies","Screening agents beat single-LLM baseline in systematic review","Multi-agent architecture recovers missed trials, shifts conclusions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the final human-consensus labels as ground truth, and in the NSCLC screening study the three reviewers reached only moderate initial agreement (Fleiss' Kappa = 0.70) before reconciling, so if that consensus is biased the measured accuracy gains are measured against a flawed yardstick.","fun_headline_variants_meta":{"raw":{"variants":["Agentic review catches 17 trials manual screening missed","Multi-agent system updates clinical trial evidence base","AI agents with human oversight find 17 omitted studies","Screening agents beat single-LLM baseline in systematic review","Multi-agent architecture recovers missed trials, shifts conclusions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1732,"prompt_tokens":975,"completion_tokens":757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":681}},"tokens_in":591,"tokens_out":757,"duration_ms":7094,"temperature":1.0,"reasoning_tokens":681,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:30:09.275887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the NSCLC screening set with an independent panel of reviewers who do not see the original consensus labels, then compare the panel's final decisions with the MAS decisions; if the panel labels a substantial fraction of the MAS's high-confidence inclusions or exclusions as errors, or if the sensitivity gain from 0.58 to 0.93 reverses under the new labels, the uniform-improvement claim fails. A cheaper check is to apply the full pipeline to a completed Cochrane or similar review with published decisions and extracted data and count exact agreement on both screening and numeric extraction.","supporting_citations":[],"review_version":1}