{"id":"3a9c2f00-2b38-4132-b46f-982ad9ac4e16","arxiv_id":"2507.15152","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMs used for meta-analysis data extraction show high precision but low recall, with domain-specific prompts providing the largest recall gains and supporting a three-tier human-oversight framework.","lead":"Three commercial LLMs were tested on extracting data from 58 full-text clinical trials for meta-analysis, across four prompting strategies and three medical fields. The models were precise but missed many fields, and prompts tailored to each medical topic recovered up to 15 percentage points of recall.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Precision numbers may be inflated: extractions with no ground-truth counterpart are never penalized, so the claimed 'precision stable' result is unverified.","rationale":"The reader's weakest assumption focuses on Gemini-as-judge bias, which is plausible but partly mitigated by the 96.1% human-validation agreement. The more load-bearing issue is structural: the evaluation prompt in Appendix B.4 defines the task as matching ground-truth fields and explicitly forbids penalizing extracted fields that do not appear in the ground truth. That choice directly contradicts the paper's own element-level precision definition and can only inflate precision. It is especially relevant to the central claim because the paper's headline includes 'high precision' and 'precision roughly stable' as part of the finding. A customised prompt that guides the model to produce additional domain-specific extractions could generate more off-schema outputs, and those would be invisible to the reported precision numbers. The recall improvement and the three-tier recommendation are more robust to this issue, which is why the verdict remains conditional rather than being rejected. The reader's judge-bias concern and this precision-denominator concern are both about evaluation validity, so there is partial agreement, but the specific load-bearing weak point identified here is different and more directly evidenced by the supplied prompts.","tokens_in":31130,"tokens_out":6660,"duration_ms":85086,"concrete_test":"Recompute precision for all four extraction methods (EXT, EXT+Self-reflection, Combined EXT, Customised EXT) across all three models by treating every leaf-level element in each EXT JSON that has no semantically matching ground-truth field as a false positive, and recompute precision as TP/(TP+FP). Use a second judge (human or a different LLM) for a stratified subsample to verify the classification. If Customised EXT's recomputed precision falls by more than roughly 3-5 percentage points relative to baseline, the 'precision stable' assertion fails; if it does not, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported precision values in Table 2 depend on an evaluation protocol that systematically ignores a class of false positives. The evaluation prompts in Appendix B.4 instruct Gemini to iterate over ground-truth fields, mark a field 'Missing' if no extraction exists, and then state explicitly: 'If GT does not include the field: DO NOT evaluate or penalize this EXT field.' This means any extracted element that has no matching ground-truth field is excluded from the denominator of precision, even though the paper defines precision as the proportion of extracted elements that are correct. Under the standard definition TP/(TP+FP), those off-schema extractions are false positives and belong in the denominator. Their exclusion inflates every precision figure in Table 2 and makes the central claim that Customised EXT keeps precision 'roughly stable' (only -0.8% on average) unsupported: if Customised EXT, prompted to attend to domain-specific fields, produces more off-schema outputs than the generic baseline, its true precision loss could be materially larger. The recall improvement of 14.8 percentage points is less affected because recall depends only on ground-truth fields being recovered, so the main recommendation about customised prompts is not directly invalidated; however, the 'high precision' and 'precision stable' components of the headline claim are not backed by the described evaluation. The 900-field human validation reports 96.1% agreement with LLM-assigned labels, but it does not resolve this issue unless the sampled fields also included extractions that had no ground-truth counterpart, which the paper does not state.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks three LLMs (Gemini-2.0-flash, Grok-3, GPT-4o-mini) on structured data extraction from 58 full-text RCTs drawn from six published meta-analyses across hypertension, diabetes, and orthopaedics. Four prompting strategies are compared: a generic baseline (EXT), self-reflection (EXT+Self-reflection), an ensemble merge performed by Gemini (Combined EXT), and domain-customised prompts (Customised EXT). Performance is reported as field-level precision and recall for three data categories (statistical results, quality assessment, study information). The central findings are that all models achieve high precision but low recall, and that Customised EXT raises recall by an average of 14.8 percentage points with only a 0.8-point precision drop. Based on these results, the authors propose a three-tier automation guideline mapping information types to appropriate levels of human oversight.","tokens_in":31386,"tokens_out":5025,"duration_ms":57808,"significance":"If the reported results are correct, the paper would provide a practically useful benchmark and a low-cost recommendation (prompt specialisation) for improving LLM-based meta-analysis data extraction. The construction of an independent ground-truth dataset from 58 full-text RCTs, the inclusion of multiple medical domains, and the blinded human validation of 900 fields are notable strengths. The paper also attempts to address evaluation bias through a role-separated design, which is commendable. However, two load-bearing issues materially weaken the quantitative claims: the evaluation protocol appears to exclude a class of false positives from the precision denominator, and the customised prompts leak the gold-standard outcome list into the extraction step. These issues do not necessarily invalidate the main qualitative conclusion that current LLMs have poor recall and that prompt specificity helps, but they do mean the precision figures and the size of the customised-prompt benefit are not reliably established.","major_comments":[{"comment":"The evaluation protocol in all three evaluation prompts (B.4.1, B.4.2, B.4.3) explicitly states: \"If GT does not include the field: DO NOT evaluate or penalize this EXT field.\" Combined with the instruction to iterate over ground-truth fields, this means that any extracted element that has no matching ground-truth field is never scored, and therefore never enters the denominator of precision. The paper defines precision in §2.3.2 as \"what proportion of extracted elements were correct,\" which corresponds to TP/(TP+FP). By ignoring unpaired extractions, the reported precision values in Table 2 are inflated and the claim that Customised EXT causes only a -0.8% precision drop (Figure 2) is unsupported. This is load-bearing because the paper's headline \"high precision\" claim and its guidance in Table 5 (e.g., precision thresholds of 90-95%) rely on these numbers. The recall improvements are less affected because recall depends only on ground-truth fields, but the precision component of the central claim needs to be re-evaluated, either by counting unpaired EXT fields as false positives or by reporting the rate of such extractions per method and showing they are balanced across methods.","section":"Appendix B.4, Section 2.3.2, Table 2"},{"comment":"The Customised EXT prompt in Appendix B.3 lists the exact outcome variables used in the meta-analysis (e.g., \"Bone Mineral Density (femoral neck, total hip, lumbar spine) and Bone Turnover Markers (CTX, P1NP, BONE ALP, Osteocalcin)\"). Section 2.1 states that the ground truth was constructed by extracting \"the specific measurements and outcomes that were actually used in the quantitative analyses of each meta-analysis.\" This creates a leakage: the customised prompt reveals the gold-standard outcome list to the extractor, whereas a real user would not know the final synthesis outcome list a priori. The reported 14.8% recall gain for Customised EXT may therefore partly reflect access to the answer key rather than the general benefit of domain-specialised prompting. To support the paper's central recommendation, the comparison should include a control prompt that specifies a plausible but non-gold outcome list, or the finding should be reframed as \"prompts that specify the target outcome list improve recall\" rather than \"domain-specialised prompts improve recall.\"","section":"Section 2.2 (Step 2d), Section 2.1, Appendix B.3"},{"comment":"The paper claims in Section 2.3.1 that the evaluation is unbiased because the judge \"did not have access to the original prompt, full document, or information about which model had produced the extraction,\" and Section 4.3 states that the design \"prevents feedback loops.\" However, Gemini is used as the judge for all outputs, including Gemini's own EXT extractions and Gemini's merged Combined EXT outputs. Blinding to model identity is not sufficient to rule out self-preference, since the judge may recognise stylistic patterns of its own outputs. The human validation of 900 fields reports aggregate agreement of 96.1% and a Cohen's kappa of 0.987 between human reviewers, but it does not report agreement stratified by model or by extraction method. If Gemini's self-evaluation is systematically more lenient for its own outputs, the model comparisons in Section 3.1.2 and the precision/recall differences among models could be distorted. The authors should report human-judge agreement separately for each model and method, and ideally include a second non-Gemini judge or a subset judged by both to test for evaluator bias.","section":"Section 2.3.1, Section 4.3"},{"comment":"The statistical analysis is reported only for recall. The Friedman test (chi-square(3)=9.81, p=0.0203) and the Nemenyi post-hoc test support the claim that Customised EXT differs from baseline EXT in recall. However, the \"precision stable\" part of the central claim is not statistically tested; the 0.8% average precision drop is reported without confidence intervals or per-dataset variability. Given that the precision measurement itself is affected by the protocol issue in the first major comment, the authors should either provide a corrected precision analysis with confidence intervals or explicitly state that precision differences are not statistically evaluable from the current protocol.","section":"Section 3.1.1, Figure 3"}],"minor_comments":[{"comment":"The ground truth construction reports that two trained research staff \"independently reviewed and checked the JSON outputs\" but does not report inter-rater reliability for the ground truth itself. Adding a kappa or percentage-agreement statistic would strengthen confidence in the gold standard.","section":"Section 2.1"},{"comment":"The three evaluation prompts contain common typos: \"INSTRCTIONS\" in B.4.1, \"word-leading\" in B.2, and inconsistent formatting of the status labels (\"Hallucinate\" vs \"Hallucinated\"). These should be corrected for reproducibility.","section":"Appendix B.4.1, B.4.2, B.4.3"},{"comment":"The error distribution analysis is descriptive and does not include confidence intervals or tests. Since the error types are derived from the same Gemini-judged labels, the reported percentages inherit the potential evaluator bias noted above. Presenting them with per-model human-validation agreement would help.","section":"Section 3.3.1, Table 6 and Table 7"},{"comment":"The phrase \"All models demonstrate high precision\" is too strong given that precision values in Table 2 range from 0.745 to 0.986 and the evaluation protocol does not count unpaired extractions. Consider softening to \"moderate-to-high precision\" or qualifying the claim.","section":"Abstract and Section 4.1"},{"comment":"The Combined EXT prompt in Appendix B.2 instructs the merging LLM to use majority voting but does not specify how to handle cases where two models provide semantically equivalent but textually different values that might not be recognized as agreement. This could bias the merge toward the model whose phrasing is closest to the judge's expectations. A brief clarification of the matching criterion would be helpful.","section":"Section 2.2 (Step 2c)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal and addresses an important practical problem. The two main issues—precision denominator exclusion and gold-standard leakage in customised prompts—are fixable but require substantive re-analysis or reframing. I would encourage the editor to send the manuscript back for major revision rather than reject, because the recall-focused findings and the proposed automation guidelines are still potentially valuable. The lack of per-model human validation for the Gemini judge is a correctable gap that should be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a worthwhile benchmark for LLM-based data extraction in meta-analysis, but the headline \"high precision\" numbers are not as solid as they look. The recall results, and the finding that customised prompts help, hold up better.\n\nWhat's new: they benchmark three current LLMs (Gemini, Grok, GPT-4o-mini) across four prompting strategies and three clinical domains on 58 full-text RCTs, with a blinded role-separated evaluation and a 900-field human validation. That combination is genuinely new, and the design is more careful than most prior single-model/single-domain studies. The three-tier automation framework is a reasonable synthesis of the findings, though it is not prospectively validated.\n\nWhere it's solid: the ground truth was manually extracted from the original RCTs, not copied from the meta-analyses. The independence of the judge from the original prompts and document is good. The 96.1% agreement on human validation is reassuring for the recall comparisons. And the recall gains from customised prompts (14.8 points on average) are consistent with the reported numbers.\n\nSoft spots: the precision metric is miscalculated. The evaluation prompt in Appendix B.4 tells the judge that if a ground-truth field does not exist, it should not evaluate or penalize the extracted field. That means any extraction that has no ground-truth counterpart is excluded from the denominator. Under the standard definition of precision, those are false positives. This inflates every precision value in Table 2 and undermines the claim that customised prompts keep precision roughly stable (-0.8%). It is possible that the precision drop for customised EXT is materially larger. The recall improvement is less affected because recall only depends on ground-truth fields being found. So the main recommendation about prompt specialisation is not invalidated, but the \"high precision\" half of the conclusion is not supported by the evaluation as described.\n\nOther concerns, mostly minor in comparison: no confidence intervals (they do a Friedman/Nemenyi test but only on recall), no code or data released, and Gemini serves as both extractor (in Combined EXT) and judge, which leaves room for self-preference. The 900-field human validation helps, but per-model agreement is not reported, and it is unclear whether off-schema extractions were included in the sample.\n\nBottom line: this deserves a serious referee. The evaluation design is mostly careful, and the recall finding is useful for practitioners. The precision issue needs to be fixed, either by counting off-schema extractions as false positives or by clearly reporting both raw and matched precision. If the authors address that and release the data, this becomes a solid contribution.\n\nRecommendation: send it to peer review, but ask for a re-run of the precision analysis and confidence intervals.","headline":"Useful benchmark for LLM-based meta-analysis extraction, but the reported precision figures are inflated by an evaluation protocol that ignores off-schema extractions; the recall findings are more trustworthy.","tokens_in":31912,"tokens_out":2190,"would_cite":true,"duration_ms":23427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs extract trial data precisely but incompletely; domain-customised prompts recover 14.8 recall points on average, yet statistical extraction still needs a human in the loop.","keywords":["large language models","meta-analysis data extraction","prompt engineering","precision and recall","risk of bias assessment","randomised controlled trials","automated evidence synthesis","model ensemble"],"falsifier":"Re-score the same 900-field stratified sample with the judge's labels computed separately for each model and prompting method, and test whether human–judge agreement is uniform across models; a significantly higher agreement for Gemini's own extractions (or for any single model) would show that the reported recall gaps are partly an artefact of the judge rather than a property of the extractors. A cheaper second check is to hand-label the statistical-results fields of a single dataset such as MA6 and ask whether Gemini's claimed recall of 0.760 under customised prompting survives human labelling.","tokens_in":2069,"feed_emoji":"🤖","tokens_out":2851,"duration_ms":104445,"temperature":0.7,"pith_summary":"This paper asks how much of the data-extraction work behind meta-analysis can be safely handed to large language models, and which prompting strategy recovers the most missing information. Working from 58 full-text randomised trials that underlie six published meta-analyses in hypertension, diabetes, and orthopaedics, the authors benchmark three models — GPT-4o-mini, Gemini-2.0-flash, and Grok-3 — against a manually built ground truth. Their central finding is a consistent precision–recall split: everything the models return is usually correct, but they omit a large share of target fields, and 87.8% of all recorded errors are missing values, concentrated in statistical outcomes. Domain-customised prompts are the strongest lever, raising recall by an average 14.8 percentage points over a generic expert prompt at a cost of only 0.8 precision points, with the largest gains on datasets full of specialised terminology. From this the authors conclude that end-to-end automation of statistical extraction is not yet safe, and they propose a three-tier routing that automates study-information fields now, keeps human review on quality assessments, and mandates human verification for pooled effect estimates.","feed_headline":"Custom prompts lift LLM trial-data recall by 15%","feed_subtitle":"Across 58 full-text trials, every model proved precise but incomplete; domain-specific prompts recovered the missing data.","key_machinery":"The load-bearing instrument is the evaluation pipeline: a role-separated judge (Gemini) compares each extracted field with the human ground truth and labels it Correct, Missing, or Hallucinated, without seeing the original PDF or the identity of the extracting model, so that no model scores its own output. A blinded manual review of 900 sampled fields found 96.09% agreement between the judge's labels and two independent human reviewers, which is the evidence that the recall and precision numbers rest on. Around that judging core sit the four prompting strategies that constitute the experimental treatment: a structured generic extraction prompt (EXT), a self-reflection revision pass (EXT+Self-reflection) built on a reflection loop, an ensemble merge of all three models' baseline outputs using majority vote with confidence-based tie-breaking (Combined EXT), and domain-customised prompts (Customised EXT) that prime the model with the exact outcomes and variables of the target meta-analysis. Customised EXT is the mechanism that carries the paper's main result: priming the model with the specific variables of interest is what converts high precision into usable recall.","core_discovery":"The paper's central claim is that current commercial LLMs are reliable extractors only in one direction: precision stays high (0.77–0.95 across settings) while recall trails badly, especially for the statistical results a meta-analysis actually pools, where baseline recall falls as low as 0.21 for GPT-4o-mini. The decisive comparison is between prompting strategies: a customised prompt that names the target outcomes of a specific meta-analysis (for example, bone mineral density and turnover markers for an orthopaedics synthesis) beats the generic expert prompt, the self-reflection loop, and the three-model ensemble on recall in every data category, with a statistically significant overall difference (Friedman $\\chi^2(3) = 9.81$, $p = 0.0203$). Model identity also shapes the result: Gemini-2.0-flash leads statistical extraction (recall 0.760 under customised prompts), Grok-3 leads overall by mean recall rank (1.3), and GPT-4o-mini trails in all three categories. The authors conclude that the cheapest and most effective route to better extraction is prompt specialisation, and that automated meta-analysis should be built modularly — automating the fields that are already \"good enough\" while reserving human judgment for the statistics that drive effect estimates.","pith_inferences":["The 87.8% missing-field error rate suggests the binding constraint is detection rather than accuracy — information buried in tables and figures is systematically skipped — which points toward retrieval-augmented and vision-enabled pipelines as the next testable step, beyond what this paper runs.","The paper does not test it, but its own design implies an easy external-validity check: swap the judge for a different model or for rule-based matching on a subset and re-measure agreement, to see whether the 96.1% human-agreement figure is judge-specific.","The tier framework suggests a cost-geometry reading the authors do not spell out: in Tier-3 statistical fields a recall error silently drops a study from the pooled estimate, whereas a precision error poisons it, so future evaluation metrics for extraction tools should weight recall far more heavily than precision in exactly the fields this paper assigns to human oversight.","If the observed pattern that newer models capture more fields continues, the three-tier map is a moving target: tier boundaries should be re-benchmarked with each model generation rather than treated as fixed guidance."],"forward_implications":["Systematic-review teams can lift extraction completeness immediately by writing prompts that name the exact outcomes and variables of their synthesis, without retraining or changing models.","Statistical results stay a human-in-the-loop task: even at its best, the customised-prompt configuration reaches roughly 76% recall in that category, below the 90% the paper sets as the automation target.","Ensemble merging (Combined EXT) is the stable fallback: it gains 5.9% recall and 2.0% precision on average, and helps the weakest model most (GPT gains 14.3% recall through merging).","Self-reflection adds only 1–2 percentage points of recall and should not be the primary mechanism for recovering omitted data.","Model choice should follow the data type: Gemini-class models for statistical fields, Grok-class models for risk-of-bias and study-characteristic fields, under the current generation."],"supporting_citations":[{"why":"The six published meta-analyses (MA1–MA6) whose 58 underlying full-text RCTs and analysed outcomes define the ground-truth extraction targets.","marker":"[25]–[30]"},{"why":"Supplier documentation for GPT-4o-mini, one of the three models under test.","marker":"[31]"},{"why":"Supplier documentation for Gemini-2.0-flash, a model under test and the model selected as the blinded judge and ensemble merger.","marker":"[32]"},{"why":"Supplier documentation for Grok-3, one of the three models under test.","marker":"[33]"},{"why":"The Reflexion self-correction loop that the EXT+Self-reflection strategy is built on.","marker":"[39]"},{"why":"The ensemble-learning theory that motivates combining multiple models' outputs into the merged Combined EXT extraction.","marker":"[42]"},{"why":"The RoB 2 risk-of-bias tool, the semi-structured framework that the Tier-2 automation guidance recommends pairing with LLM pre-filling.","marker":"[50]"},{"why":"Prior work on automatically extracting numerical results from randomised trials with LLMs, the closest baseline this benchmark extends.","marker":"[23]"}],"fun_headline_variants":["LLM recall lags precision in trial extraction","Custom prompts boost LLM recall up to 15%","Which LLM extracts trial data best? None fully","Precision high, recall low: LLM extraction gap","Prompts matter most for LLM data extraction"],"cache_read_input_tokens":34048,"weakest_assumption_plain":"The entire comparison of precision and recall rests on one automated judge, Gemini, deciding whether an extracted value \"means the same\" as the ground truth; the authors validate that judge on only 900 of roughly 22,000 fields (96.1% agreement) and do not report agreement separately per model, so a systematic favouritism in Gemini's semantic judgements toward any one model family would distort the reported gaps between methods.","fun_headline_variants_meta":{"raw":{"variants":["LLM recall lags precision in trial extraction","Custom prompts boost LLM recall up to 15%","Which LLM extracts trial data best? None fully","Precision high, recall low: LLM extraction gap","Prompts matter most for LLM data extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":2939,"prompt_tokens":1012,"completion_tokens":1927,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":1851}},"tokens_in":628,"tokens_out":1927,"duration_ms":13698,"temperature":1.0,"reasoning_tokens":1851,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:39:25.851735+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the same 900-field stratified sample with the judge's labels computed separately for each model and prompting method, and test whether human–judge agreement is uniform across models; a significantly higher agreement for Gemini's own extractions (or for any single model) would show that the reported recall gaps are partly an artefact of the judge rather than a property of the extractors. A cheaper second check is to hand-label the statistical-results fields of a single dataset such as MA6 and ask whether Gemini's claimed recall of 0.760 under customised prompting survives human labelling.","supporting_citations":[{"cited_title":"Reflexion: language agents with verbal reinforcement learning","cited_arxiv_id":null,"evidence_quote":"The Reflexion self-correction loop that the EXT+Self-reflection strategy is built on."},{"cited_title":"Dietterich","cited_arxiv_id":null,"evidence_quote":"The ensemble-learning theory that motivates combining multiple models' outputs into the merged Combined EXT extraction."}],"review_version":1}