Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs extract trial data precisely but incompletely; domain-customised prompts recover 14.8 recall points on average, yet statistical extraction still needs a human in the loop.

desk verdict Useful benchmark for LLM-based meta-analysis extraction, but the reported precision figures are inflated by an evaluation protocol that ignores off-schema extractions; the recall findings are more trustworthy. read the letter →

arxiv 2507.15152 v1 pith:RVLXFYW7 submitted 2025-07-20 cs.CL cs.AIcs.LG

Lingbo Li , Anuradha Mathrani , Teo Susnjak This is my paper
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelsmeta-analysisdataextractionpromptengineeringprecisionandrecallriskofbiasassessmentrandomisedcontrolledtrialsautomatedevidencesynthesismodelensemble
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how much of the data-extraction work behind meta-analysis can be safely handed to large language models, and which prompting strategy recovers the most missing information. Working from 58 full-text randomised trials that underlie six published meta-analyses in hypertension, diabetes, and orthopaedics, the authors benchmark three models — GPT-4o-mini, Gemini-2.0-flash, and Grok-3 — against a manually built ground truth. Their central finding is a consistent precision–recall split: everything the models return is usually correct, but they omit a large share of target fields, and 87.8% of all recorded errors are missing values, concentrated in statistical outcomes. Domain-customised prompts are the strongest lever, raising recall by an average 14.8 percentage points over a generic expert prompt at a cost of only 0.8 precision points, with the largest gains on datasets full of specialised terminology. From this the authors conclude that end-to-end automation of statistical extraction is not yet safe, and they propose a three-tier routing that automates study-information fields now, keeps human review on quality assessments, and mandates human verification for pooled effect estimates.

What carries the argument

The load-bearing instrument is the evaluation pipeline: a role-separated judge (Gemini) compares each extracted field with the human ground truth and labels it Correct, Missing, or Hallucinated, without seeing the original PDF or the identity of the extracting model, so that no model scores its own output. A blinded manual review of 900 sampled fields found 96.09% agreement between the judge's labels and two independent human reviewers, which is the evidence that the recall and precision numbers rest on. Around that judging core sit the four prompting strategies that constitute the experimental treatment: a structured generic extraction prompt (EXT), a self-reflection revision pass (EXT+Self-reflection) built on a reflection loop, an ensemble merge of all three models' baseline outputs using majority vote with confidence-based tie-breaking (Combined EXT), and domain-customised prompts (Customised EXT) that prime the model with the exact outcomes and variables of the target meta-analysis. Customised EXT is the mechanism that carries the paper's main result: priming the model with the specific variables of interest is what converts high precision into usable recall.

What would settle it

Re-score the same 900-field stratified sample with the judge's labels computed separately for each model and prompting method, and test whether human–judge agreement is uniform across models; a significantly higher agreement for Gemini's own extractions (or for any single model) would show that the reported recall gaps are partly an artefact of the judge rather than a property of the extractors. A cheaper second check is to hand-label the statistical-results fields of a single dataset such as MA6 and ask whether Gemini's claimed recall of 0.760 under customised prompting survives human labelling.

Watch

Extended reading notes

Core claim

The paper's central claim is that current commercial LLMs are reliable extractors only in one direction: precision stays high (0.77–0.95 across settings) while recall trails badly, especially for the statistical results a meta-analysis actually pools, where baseline recall falls as low as 0.21 for GPT-4o-mini. The decisive comparison is between prompting strategies: a customised prompt that names the target outcomes of a specific meta-analysis (for example, bone mineral density and turnover markers for an orthopaedics synthesis) beats the generic expert prompt, the self-reflection loop, and the three-model ensemble on recall in every data category, with a statistically significant overall difference (Friedman $\chi^2(3) = 9.81$, $p = 0.0203$). Model identity also shapes the result: Gemini-2.0-flash leads statistical extraction (recall 0.760 under customised prompts), Grok-3 leads overall by mean recall rank (1.3), and GPT-4o-mini trails in all three categories. The authors conclude that the cheapest and most effective route to better extraction is prompt specialisation, and that automated meta-analysis should be built modularly — automating the fields that are already "good enough" while reserving human judgment for the statistics that drive effect estimates.

Load-bearing premise

The entire comparison of precision and recall rests on one automated judge, Gemini, deciding whether an extracted value "means the same" as the ground truth; the authors validate that judge on only 900 of roughly 22,000 fields (96.1% agreement) and do not report agreement separately per model, so a systematic favouritism in Gemini's semantic judgements toward any one model family would distort the reported gaps between methods.

Editorial extensions

If this is right

  • Systematic-review teams can lift extraction completeness immediately by writing prompts that name the exact outcomes and variables of their synthesis, without retraining or changing models.
  • Statistical results stay a human-in-the-loop task: even at its best, the customised-prompt configuration reaches roughly 76% recall in that category, below the 90% the paper sets as the automation target.
  • Ensemble merging (Combined EXT) is the stable fallback: it gains 5.9% recall and 2.0% precision on average, and helps the weakest model most (GPT gains 14.3% recall through merging).
  • Self-reflection adds only 1–2 percentage points of recall and should not be the primary mechanism for recovering omitted data.
  • Model choice should follow the data type: Gemini-class models for statistical fields, Grok-class models for risk-of-bias and study-characteristic fields, under the current generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 87.8% missing-field error rate suggests the binding constraint is detection rather than accuracy — information buried in tables and figures is systematically skipped — which points toward retrieval-augmented and vision-enabled pipelines as the next testable step, beyond what this paper runs.
  • The paper does not test it, but its own design implies an easy external-validity check: swap the judge for a different model or for rule-based matching on a subset and re-measure agreement, to see whether the 96.1% human-agreement figure is judge-specific.
  • The tier framework suggests a cost-geometry reading the authors do not spell out: in Tier-3 statistical fields a recall error silently drops a study from the pooled estimate, whereas a precision error poisons it, so future evaluation metrics for extraction tools should weight recall far more heavily than precision in exactly the fields this paper assigns to human oversight.
  • If the observed pattern that newer models capture more fields continues, the three-tier map is a moving target: tier boundaries should be re-benchmarked with each model generation rather than treated as fixed guidance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper benchmarks three LLMs (Gemini-2.0-flash, Grok-3, GPT-4o-mini) on structured data extraction from 58 full-text RCTs drawn from six published meta-analyses across hypertension, diabetes, and orthopaedics. Four prompting strategies are compared: a generic baseline (EXT), self-reflection (EXT+Self-reflection), an ensemble merge performed by Gemini (Combined EXT), and domain-customised prompts (Customised EXT). Performance is reported as field-level precision and recall for three data categories (statistical results, quality assessment, study information). The central findings are that all models achieve high precision but low recall, and that Customised EXT raises recall by an average of 14.8 percentage points with only a 0.8-point precision drop. Based on these results, the authors propose a three-tier automation guideline mapping information types to appropriate levels of human oversight.

Significance. If the reported results are correct, the paper would provide a practically useful benchmark and a low-cost recommendation (prompt specialisation) for improving LLM-based meta-analysis data extraction. The construction of an independent ground-truth dataset from 58 full-text RCTs, the inclusion of multiple medical domains, and the blinded human validation of 900 fields are notable strengths. The paper also attempts to address evaluation bias through a role-separated design, which is commendable. However, two load-bearing issues materially weaken the quantitative claims: the evaluation protocol appears to exclude a class of false positives from the precision denominator, and the customised prompts leak the gold-standard outcome list into the extraction step. These issues do not necessarily invalidate the main qualitative conclusion that current LLMs have poor recall and that prompt specificity helps, but they do mean the precision figures and the size of the customised-prompt benefit are not reliably established.

major comments (4)
  1. [Appendix B.4, Section 2.3.2, Table 2] The evaluation protocol in all three evaluation prompts (B.4.1, B.4.2, B.4.3) explicitly states: "If GT does not include the field: DO NOT evaluate or penalize this EXT field." Combined with the instruction to iterate over ground-truth fields, this means that any extracted element that has no matching ground-truth field is never scored, and therefore never enters the denominator of precision. The paper defines precision in §2.3.2 as "what proportion of extracted elements were correct," which corresponds to TP/(TP+FP). By ignoring unpaired extractions, the reported precision values in Table 2 are inflated and the claim that Customised EXT causes only a -0.8% precision drop (Figure 2) is unsupported. This is load-bearing because the paper's headline "high precision" claim and its guidance in Table 5 (e.g., precision thresholds of 90-95%) rely on these numbers. The recall improvements are less affected because recall depends only on ground-truth fields, but the precision component of the central claim needs to be re-evaluated, either by counting unpaired EXT fields as false positives or by reporting the rate of such extractions per method and showing they are balanced across methods.
  2. [Section 2.2 (Step 2d), Section 2.1, Appendix B.3] The Customised EXT prompt in Appendix B.3 lists the exact outcome variables used in the meta-analysis (e.g., "Bone Mineral Density (femoral neck, total hip, lumbar spine) and Bone Turnover Markers (CTX, P1NP, BONE ALP, Osteocalcin)"). Section 2.1 states that the ground truth was constructed by extracting "the specific measurements and outcomes that were actually used in the quantitative analyses of each meta-analysis." This creates a leakage: the customised prompt reveals the gold-standard outcome list to the extractor, whereas a real user would not know the final synthesis outcome list a priori. The reported 14.8% recall gain for Customised EXT may therefore partly reflect access to the answer key rather than the general benefit of domain-specialised prompting. To support the paper's central recommendation, the comparison should include a control prompt that specifies a plausible but non-gold outcome list, or the finding should be reframed as "prompts that specify the target outcome list improve recall" rather than "domain-specialised prompts improve recall."
  3. [Section 2.3.1, Section 4.3] The paper claims in Section 2.3.1 that the evaluation is unbiased because the judge "did not have access to the original prompt, full document, or information about which model had produced the extraction," and Section 4.3 states that the design "prevents feedback loops." However, Gemini is used as the judge for all outputs, including Gemini's own EXT extractions and Gemini's merged Combined EXT outputs. Blinding to model identity is not sufficient to rule out self-preference, since the judge may recognise stylistic patterns of its own outputs. The human validation of 900 fields reports aggregate agreement of 96.1% and a Cohen's kappa of 0.987 between human reviewers, but it does not report agreement stratified by model or by extraction method. If Gemini's self-evaluation is systematically more lenient for its own outputs, the model comparisons in Section 3.1.2 and the precision/recall differences among models could be distorted. The authors should report human-judge agreement separately for each model and method, and ideally include a second non-Gemini judge or a subset judged by both to test for evaluator bias.
  4. [Section 3.1.1, Figure 3] The statistical analysis is reported only for recall. The Friedman test (chi-square(3)=9.81, p=0.0203) and the Nemenyi post-hoc test support the claim that Customised EXT differs from baseline EXT in recall. However, the "precision stable" part of the central claim is not statistically tested; the 0.8% average precision drop is reported without confidence intervals or per-dataset variability. Given that the precision measurement itself is affected by the protocol issue in the first major comment, the authors should either provide a corrected precision analysis with confidence intervals or explicitly state that precision differences are not statistically evaluable from the current protocol.
minor comments (5)
  1. [Section 2.1] The ground truth construction reports that two trained research staff "independently reviewed and checked the JSON outputs" but does not report inter-rater reliability for the ground truth itself. Adding a kappa or percentage-agreement statistic would strengthen confidence in the gold standard.
  2. [Appendix B.4.1, B.4.2, B.4.3] The three evaluation prompts contain common typos: "INSTRCTIONS" in B.4.1, "word-leading" in B.2, and inconsistent formatting of the status labels ("Hallucinate" vs "Hallucinated"). These should be corrected for reproducibility.
  3. [Section 3.3.1, Table 6 and Table 7] The error distribution analysis is descriptive and does not include confidence intervals or tests. Since the error types are derived from the same Gemini-judged labels, the reported percentages inherit the potential evaluator bias noted above. Presenting them with per-model human-validation agreement would help.
  4. [Abstract and Section 4.1] The phrase "All models demonstrate high precision" is too strong given that precision values in Table 2 range from 0.745 to 0.986 and the evaluation protocol does not count unpaired extractions. Consider softening to "moderate-to-high precision" or qualifying the claim.
  5. [Section 2.2 (Step 2c)] The Combined EXT prompt in Appendix B.2 instructs the merging LLM to use majority voting but does not specify how to handle cases where two models provide semantically equivalent but textually different values that might not be recognized as agreement. This could bias the merge toward the model whose phrasing is closest to the judge's expectations. A brief clarification of the matching criterion would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark with an independently constructed ground truth, a role-separated evaluation protocol, and no central result that reduces to its own inputs by construction.

full rationale

This is an empirical benchmark rather than a derivation chain, so circularity tests apply to the evaluation design. The ground truth was constructed manually from the 58 RCT full texts, checked by two trained staff plus medical experts (Section 2.1), and was not copied from the meta-analyses being reproduced, so extraction scores are not fitted from the reference values. The nearest self-involvement is Gemini acting as both an extractor and the LLM-based evaluator/merger; however, the protocol blinds the evaluator to model identity and original documents (Section 2.3.1), and a blinded 900-field human review achieved 96.09% agreement with LLM-assigned labels, providing independent grounding. The single self-citation [52] supports background discussion of automated meta-analysis and is not load-bearing. Two non-circular validity caveats remain: the evaluation prompt instructs that fields not present in the ground truth are not to be evaluated or penalized (Appendix B.4), so precision is computed only over GT-matched extractions and may be inflated; and the customised prompts list the target outcome names that also define the ground-truth schema, which limits generalisation but does not force the recall gain, because models still must locate and extract the underlying values. Neither caveat makes any headline result equivalent to the paper's inputs by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical claim depends on the accuracy of the human ground truth, the validity of Gemini-as-judge, and the representativeness of the six selected meta-analyses; only the judge validity is explicitly tested, and only on a 900-field sample. The customised prompts encode the exact target outcomes, which is a design choice that affects interpretation of the recall gain.

free parameters (1)
  • Numerical correctness tolerance = ±1% relative difference
    Appendix B.4.1 defines a numerical extraction as correct if it falls within 1% of the ground truth. This threshold is chosen by hand and could influence precision and recall, though the qualitative finding is robust to moderate changes.
assumptions (4)
  • domain assumption The human-annotated ground truth is accurate and complete.
    Section 2.1 states two trained staff and medical experts checked the JSON, but no inter-rater reliability metric is reported for the ground truth construction.
  • domain assumption Gemini's evaluation labels approximate human judgment.
    Section 2.3.1 reports 96.09% agreement with human labels on 900 sampled fields, but this is a single overall figure without per-model breakdown.
  • domain assumption The six selected meta-analyses are representative of hypertension, diabetes, and orthopaedics.
    Section 2.1 selects two meta-analyses per domain from Q1/Q2 journals, but representativeness is not empirically established.
  • ad hoc to paper Customised prompts leak the gold-standard outcome list into the extraction step.
    Appendix B.3 shows custom prompts for orthopaedics specify exact outcomes (e.g., CTX, P1NP, BONE ALP, Osteocalcin), which are the same outcomes used to build the ground truth, so the recall gain may overestimate real-world use where users may not name every outcome in advance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction." pith.science (2026). https://pith.science/paper/RVLXFYW7

@misc{pith2026250715152,
  author       = {Pith},
  title        = {Pith review of: What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RVLXFYW7}},
  note         = {Machine review of arXiv:2507.15152}
}
read the original abstract

Automating data extraction from full-text randomised controlled trials (RCTs) for meta-analysis remains a significant challenge. This study evaluates the practical performance of three LLMs (Gemini-2.0-flash, Grok-3, GPT-4o-mini) across tasks involving statistical results, risk-of-bias assessments, and study-level characteristics in three medical domains: hypertension, diabetes, and orthopaedics. We tested four distinct prompting strategies (basic prompting, self-reflective prompting, model ensemble, and customised prompts) to determine how to improve extraction quality. All models demonstrate high precision but consistently suffer from poor recall by omitting key information. We found that customised prompts were the most effective, boosting recall by up to 15\%. Based on this analysis, we propose a three-tiered set of guidelines for using LLMs in data extraction, matching data types to appropriate levels of automation based on task complexity and risk. Our study offers practical advice for automating data extraction in real-world meta-analyses, balancing LLM efficiency with expert oversight through targeted, task-specific automation.

Figures

Figures reproduced from arXiv: 2507.15152 by the authors.

Figure 1
Figure 1. Overview of the whole workflow. Full-text RCTs were collected from published meta-analyses and annotated [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Average performance change (∆ precision and ∆ recall) of three extraction strategies relative to the EXT baseline [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Friedman–Nemenyi critical difference (CD) graph based on mean rank in recall. If the two horizontal line [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Methods Comparison Across Meta-analyses the error distribution with a focus on general trends. Additional analyses, including field-specific differences and the interaction between models and methods, are presented in the appendix for further reference. 3.3.1 Overall A…
Figure 5
Figure 5. Figure 5: Models Comparison Across Meta-analyses 4 Discussion This study evaluated how well LLMs perform structured data extraction for AMA, aiming to identify their current capabilities and key weaknesses in practical, cross-domain situations. We summarise the main findings her…
Figure 6
Figure 6. Figure 6: Three-tier automation guideline for structured data extraction in meta-analysis, based on task difficulty, error [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis

    cs.CL 2025-08 conditional novelty 4.0 of 10

    Declarative prompt optimization is adapted into a four-step, code-backed framework (goal, standard, compile, package) for reproducible LLM-assisted systematic review screening.

Reference graph

Works this paper leans on

99 extracted references · 66 canonical work pages · cited by 1 Pith paper

  1. [1]

    Research Synthesis and Meta-Analysis: A Step-by-Step Approach

    Harris Cooper. Research Synthesis and Meta-Analysis: A Step-by-Step Approach . SAGE Publications, Inc, Thousand Oaks, California, 5 edition, 2017. doi:10.4135/9781071878644. URL https://methods.sagepub. com/book/mono/research-synthesis-and-meta-analysis-5e/toc

    doi:10.4135/9781071878644 2017

  2. [2]

    Analysing data and undertaking meta-analyses, chapter 10, pages 241–284

    Jonathan J Deeks, Julian PT Higgins, Douglas G Altman, and on behalf of the Cochrane Statistical Methods Group. Analysing data and undertaking meta-analyses, chapter 10, pages 241–284. John Wiley & Sons, Ltd, 2019. ISBN 9781119536604. doi:https://doi.org/10.1002/9781119536604.ch10. URL https://onlinelibrary.wiley. com/doi/abs/10.1002/9781119536604.ch10

    doi:10.1002/9781119536604.ch10 2019

  3. [3]

    Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the prospero registry.BMJ Open, 7 (2), 2017

    Rohit Borah, Andrew W Brown, Patrice L Capers, and Kathryn A Kaiser. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the prospero registry.BMJ Open, 7 (2), 2017. ISSN 2044-6055. doi:10.1136/bmjopen-2016-012545. URL https://bmjopen.bmj.com/content/ 7/2/e012545

    doi:10.1136/bmjopen-2016-012545 2017

  4. [4]

    Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J

    Julian P.T. Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and Vivian A. Welch. Cochrane handbook for systematic reviews of interventions. wiley, January 2019. ISBN 9781119536628. doi:10.1002/9781119536604. Publisher Copyright: © 2019 The Cochrane Collaboration

    work page doi:10.1002/9781119536604 2019

  5. [5]

    Validity of data extraction in evidence synthesis practice of adverse events: repro- ducibility study

    Chang Xu, Tianqi Yu, Luis Furuya-Kanamori, Lifeng Lin, Liliane Zorzela, Xiaoqin Zhou, Hanming Dai, Yoon Loke, and Sunita V ohra. Validity of data extraction in evidence synthesis practice of adverse events: repro- ducibility study. BMJ, 377, 2022. doi:10.1136/bmj-2021-069155. URL https://www.bmj.com/content/ 377/bmj-2021-069155

    work page doi:10.1136/bmj-2021-069155 2022

  6. [6]

    Toward systematic review automation: a practical guide to using machine learning tools in research synthesis

    Iain J Marshall and Byron C Wallace. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic reviews, 8:1–10, 2019. URL https://doi.org/10.1186/ s13643-019-1074-9

    2019

  7. [7]

    Exact: automatic extraction of clinical trial characteristics from journal publications

    Svetlana Kiritchenko, Berry De Bruijn, Simona Carini, Joel Martin, and Ida Sim. Exact: automatic extraction of clinical trial characteristics from journal publications. BMC medical informatics and decision making, 10:1–17,

  8. [8]

    Summerscales, Shlomo Argamon, Shangda Bai, Jordan Hupert, and Alan Schwartz

    Rodney L. Summerscales, Shlomo Argamon, Shangda Bai, Jordan Hupert, and Alan Schwartz. Automatic summarization of results from clinical trials. In 2011 IEEE International Conference on Bioinformatics and Biomedicine, pages 372–377, 2011. doi:10.1109/BIBM.2011.72

    work page doi:10.1109/bibm.2011.72 2011

Show all 99 references
  1. [9]

    A. A. Boyko, A. M. Kaidina, Y . C. Kim, A. Yu. Lupatov, A. I. Panov, R. E. Suvorov, and A. V . Shvets. A framework for automated meta-analysis: Dendritic cell therapy case study. In 2016 IEEE 8th International Conference on Intelligent Systems (IS), pages 160–166, 2016. doi:10...

    2016

  2. [10]

    M. W. Lorenz, N. A. Abdi, F. Scheckenbach, A. Pflug, A. Bülbül, A. L. Catapano, S. Agewall, M. Ezhov, M. L. Bots, S. Kiechl, A. Orth, G. D. Norata, J. P. Empana, H. J. Lin, S. McLachlan, L. Bokemark, K. Ronkainen, M. Amato, U. Schminke, S. R. Srinivasan, L. Lind, A. Kato, C. D...

    2017

  3. [11]

    Automating meta-analyses of randomized clinical trials: a first look

    Matthew Michelson. Automating meta-analyses of randomized clinical trials: a first look. In 2014 AAAI Fall Symposium Series, 2014

    2014

  4. [12]

    Katz-Rogozhnikov, Kush R

    Lu Cheng, Dmitriy A. Katz-Rogozhnikov, Kush R. Varshney, and Ioana Baldini. Automated meta-analysis in medical research: A causal learning perspective. In In ACM Conference on Health, Inference, and Learning, April 2021

    2021

  5. [13]

    F. W. Mutinda, K. Liew, S. Yada, S. Wakamiya, and E. Aramaki. Automatic data extraction to support meta- analysis statistical analysis: a case study on breast cancer. BMC Med Inform Decis Mak, 22(1):158, June 2022. ISSN 1472-6947. doi:10.1186/s12911-022-01897-4

    2022 doi

  6. [14]

    Data extraction methods for systematic review (semi)automation: Update of a living systematic review [version 2; peer review: 3 approved]

    L Schmidt, AN Finnerty Mutlu, R Elmore, BK Olorisade, J Thomas, and JPT Higgins. Data extraction methods for systematic review (semi)automation: Update of a living systematic review [version 2; peer review: 3 approved]. F1000Research, 10(401), 2023. doi:10.12688/f1000research....

    2023 doi

  7. [15]

    Iain Marshall, Joël Kuiper, Edward Banner, and Byron C. Wallace. Automating biomedical evidence synthesis: RobotReviewer. In Mohit Bansal and Heng Ji, editors, Proceedings of ACL 2017, System Demonstrations, pages 7–12, Vancouver, Canada, July 2017. Association for Computation...

    2017

  8. [16]

    The data is in: Deciding when to automate screening in your slr, November 2023

    Nested Knowledge. The data is in: Deciding when to automate screening in your slr, November 2023. URL https://about.nested-knowledge.com/2023/11/10/ the-data-is-in-deciding-when-to-automate-screening-in-your-slr/ . Accessed: 2025-06- 09

    2023

  9. [17]

    Toward automated data extraction according to tabular data structure: Cross-sectional pilot survey of the comparative clinical literature

    Karl Holub, Nicole Hardy, and Kevin Kallmes. Toward automated data extraction according to tabular data structure: Cross-sectional pilot survey of the comparative clinical literature. JMIR Form Res, 5(11):e33124, Nov

  10. [18]

    MetaMate: Large Language Model to the Rescue of Automated Data Extraction for Educational Systematic Reviews and Meta-analyses, 2024

    Xue Wang and Gaoxiang Luo. MetaMate: Large Language Model to the Rescue of Automated Data Extraction for Educational Systematic Reviews and Meta-analyses, 2024

    2024

  11. [19]

    Chatgpt: Large language model (mar 14 version)

    OpenAI. Chatgpt: Large language model (mar 14 version). https://chat.openai.com/, 2023. Accessed May 15, 2025

    2023

  12. [20]

    Claude 2 model announcement

    Anthropic. Claude 2 model announcement. https://www.anthropic.com/news/claude-2, 2023. Accessed: 2025-05-28

    2023

  13. [21]

    Zero-shot infor- mation extraction for clinical meta-analysis using large language models

    David Kartchner, Selvi Ramalingam, Irfan Al-Hussaini, Olivia Kronick, and Cassie Mitchell. Zero-shot infor- mation extraction for clinical meta-analysis using large language models. In Dina Demner-fushman, Sophia Ananiadou, and Kevin Cohen, editors, The 22nd Workshop on Biomed...

    2023 doi

  14. [22]

    Performance of two large language models for data extraction in evidence synthesis

    Amanda Konet, Ian Thomas, Gerald Gartlehner, Leila Kahwati, Rainer Hilscher, Shannon Kugley, Karen Crotty, Meera Viswanathan, and Robert Chew. Performance of two large language models for data extraction in evidence synthesis. Research synthesis methods, 15(5):818–824, 2024

    2024

  15. [23]

    Automatically extracting numerical results from randomized controlled trials with large language models

    Hye Sun Yun, David Pogrebitskiy, Iain James Marshall, and Byron C Wallace. Automatically extracting numerical results from randomized controlled trials with large language models. In Kaivalya Deshpande, Madalina Fiterau, Shalmali Joshi, Zachary Lipton, Rajesh Ranganath, and Iñ...

    2024

  16. [24]

    Exploring the use of a large language model for data extraction in systematic reviews: a rapid feasibility study

    Lena Schmidt, Kaitlyn Hair, Sergio Graziozi, Fiona Campbell, Claudia Kapp, Alireza Khanteymoori, Dawn Craig, Mark Engelbert, and James Thomas. Exploring the use of a large language model for data extraction in systematic reviews: a rapid feasibility study. In Proceedings of th...

    2024

  17. [25]

    Lee, Shigeki Yamada, and Tomohiro Mizuno

    Noriaki Matsumoto, Tsuyoshi Nakai, Mikio Sakakibara, Yukinori Aimiya, Shinya Sugiura, Jeannie K. Lee, Shigeki Yamada, and Tomohiro Mizuno. Remote follow-up by pharmacists for blood pressure control in patients with hypertension: a systematic review and a meta-analysis of rando...

    2024 doi

  18. [26]

    Effects of the Modified DASH Diet on Adults With Elevated Blood Pressure or Hypertension: A Systematic Review and Meta-Analysis

    Ru Guo, Nian Li, Rong Yang, Xiao-Yang Liao, Yu Zhang, Ben-Fu Zhu, Qian Zhao, Lingmin Chen, Yong- Gang Zhang, and Yi Lei. Effects of the Modified DASH Diet on Adults With Elevated Blood Pressure or Hypertension: A Systematic Review and Meta-Analysis. Frontiers in nutrition, 8:7...

    2021

  19. [27]

    Abdelrahim, Nivine Hanach, Refat AlKurd, Moien Khan, Lana Mahrous, Hadia Radwan, Farah Naja, Mohamed Madkour, Khaled Obaideen, Husam Khraiwesh, and MoezAlIslam Faris

    Narmin Khalid, Dana N. Abdelrahim, Nivine Hanach, Refat AlKurd, Moien Khan, Lana Mahrous, Hadia Radwan, Farah Naja, Mohamed Madkour, Khaled Obaideen, Husam Khraiwesh, and MoezAlIslam Faris. Effect of camel milk on lipid profile among patients with diabetes: a systematic review...

    2023 doi

  20. [28]

    Effect of dietary glycemic index on insulin resistance in adults without diabetes mellitus: a systematic review and meta-analysis

    Yu-Ting Yu, Yu-Hsiang Fu, Yi-Hsien Chen, Yu-Wei Fang, and Ming-Hsien Tsai. Effect of dietary glycemic index on insulin resistance in adults without diabetes mellitus: a systematic review and meta-analysis. Frontiers in Nutrition, 12:1458353, February 2025. ISSN 2296-861X. doi:...

    2025

  21. [29]

    Choi, Min-Sun Gu, Seo-Yeong Ko, Jae-Hee Kwon, Ja-Young Han, Jae Hyun Kim, and Myeong Gyu Kim

    Hee-Ju Kim, Seo-A. Choi, Min-Sun Gu, Seo-Yeong Ko, Jae-Hee Kwon, Ja-Young Han, Jae Hyun Kim, and Myeong Gyu Kim. Effects of Glucagon-Like Peptide-1 Receptor Agonist on Bone Mineral Density and Bone Turnover Markers: A Meta-Analysis. Diabetes/metabolism research and reviews, 40...

    2024 doi

  22. [30]

    V olar locking plate vs cast immobilization for distal radius fractures: a systematic review and meta- analysis

    Lorenzo Massimo Oldrini, Pietro Feltri, Jacopo Albanese, Stefano Lucchina, Giuseppe Filardo, and Christian Candrian. V olar locking plate vs cast immobilization for distal radius fractures: a systematic review and meta- analysis. EFORT Open Reviews, 7(9):644–652, September 202...

    2022 doi

  23. [31]

    Gpt-4o mini: Advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/ , 2024

    2024

  24. [32]

    Gemini 2.0 flash

    Google DeepMind. Gemini 2.0 flash. https://deepmind.google/technologies/gemini/flash/, 2024

    2024

  25. [33]

    Grok-3 language model

    xAI. Grok-3 language model. https://x.ai/, 2024

    2024

  26. [34]

    The impact of temperature on extracting information from clinical trial publications using large language models

    Paul Windisch, Fabio Dennstädt, Carole Koechli, Christina Schröder, Daniel M Aebersold, Robert Förster, Daniel R Zwahlen, and Paul Y Windisch. The impact of temperature on extracting information from clinical trial publications using large language models. Cureus, 16(12), 2024

    2024

  27. [35]

    Schroeder, Chris Davis Jaldi, and Shan Zhang

    Noah L. Schroeder, Chris Davis Jaldi, and Shan Zhang. Large language models with human-in-the-loop validation for systematic review data extraction, 2025. URL https://arxiv.org/abs/2501.11840

    2025 arXiv

  28. [36]

    Use gemini 2.0 to speed up data processing

    Google Cloud. Use gemini 2.0 to speed up data processing. https://cloud.google.com/blog/products/ ai-machine-learning/use-gemini-2-0-to-speed-up-data-processing , 2025. Blog post

    2025

  29. [37]

    Harnessing ai for integrative medicine: Exploring grok 3’s role in researching qigong, tai chi, yoga, and mindfulness for college students’ mental health

    Hao Zhang, Lin Wang, and Shiyu Liu. Harnessing ai for integrative medicine: Exploring grok 3’s role in researching qigong, tai chi, yoga, and mindfulness for college students’ mental health. American Journal of Biomedical Science and Research , 26(3), 2025. doi:10.34297/AJBSR....

    2025 doi

  30. [38]

    Microsoft adds elon musk’s grok-3 to azure, citing health- care and science use cases

    Microsoft. Microsoft adds elon musk’s grok-3 to azure, citing health- care and science use cases. https://www.mobihealthnews.com/news/ microsoft-adds-elon-musks-grok-3-azure-citing-healthcare-and-science-use-cases , 2025. News release

    2025

  31. [39]

    Reflexion: language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems ...

    2023

  32. [40]

    Towards mitigating LLM halluci- nation via self reflection

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating LLM halluci- nation via self reflection. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827–1843,...

    2023 doi

  33. [41]

    When hindsight is not 20/20: Testing limits on reflective thinking in large language models

    Yanhong Li, Chenghao Yang, and Allyson Ettinger. When hindsight is not 20/20: Testing limits on reflective thinking in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024 , pages ...

    2024

  34. [42]

    Dietterich

    Thomas G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems, pages 1–15, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg. ISBN 978-3-540-45014-6

    2000

  35. [43]

    A survey on ensemble learning

    Xueying Dong, Zhiwen Yu, Wen Cao, Yanchao Shi, and Qiang Ma. A survey on ensemble learning. Frontiers of Computer Science, 14(2):241–258, 2020. doi:10.1007/s11704-019-8208-z

    2020 doi

  36. [44]

    Ensemble pretrained language models to extract biomedical knowledge from litera- ture

    Zhao Li, Qiang Wei, Liang-Chin Huang, Jianfu Li, Yan Hu, Yao-Shun Chuang, Jianping He, Avisha Das, Vipina Kuttichi Keloth, Yuntao Yang, Chiamaka S Diala, Kirk E Roberts, Cui Tao, Xiaoqian Jiang, W Jim Zheng, and Hua Xu. Ensemble pretrained language models to extract biomedical...

    1904 doi

  37. [45]

    Zhang and A.L.P

    Z. Zhang and A.L.P. Chen. Biomedical named entity recognition with the combined feature attention and fully-shared multi-task learning. BMC Bioinformatics, 23(1):458, 2022. doi:10.1186/s12859-022-04994-3. 18 ... A PREPRINT

    2022 doi

  38. [46]

    Comprehensive testing of large language models for extraction of structured data in pathology

    Bernd Grothey, Jonas Odenkirchen, Ana Brkic, et al. Comprehensive testing of large language models for extraction of structured data in pathology. Communications Medicine, 5:96, 2025. doi:10.1038/s43856-025- 00808-8. URL https://doi.org/10.1038/s43856-025-00808-8

    2025 doi

  39. [47]

    Innocence discovery lab - harnessing large language models to surface data buried in wrongful conviction case documents

    Ayyub Ibrahim, Huy Dao, and Tarak Shah. Innocence discovery lab - harnessing large language models to surface data buried in wrongful conviction case documents. The Wrongful Conviction Law Review, 5(1):103–126, 2024. doi:https://doi.org/10.29173/wclawr112

    2024 doi

  40. [48]

    Match, compare, or select? an investigation of large language models for entity matching

    Tianshu Wang, Xiaoyang Chen, Hongyu Lin, Xuanang Chen, Xianpei Han, Le Sun, Hao Wang, and Zhenyu Zeng. Match, compare, or select? an investigation of large language models for entity matching. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio...

    2025

  41. [49]

    María Fernández-Pichel, José Carlos Pichel, and David E. Losada. Evaluating search engines and large language models for answering health questions. npj Digital Medicine, 8:153, 2025. doi:10.1038/s41746-025-01546-w. URL https://doi.org/10.1038/s41746-025-01546-w

    2025 doi

  42. [50]

    Jonathan A. C. Sterne, Jelena Savovi´c, Matthew J. Page, Roy G. Elbers, Natalie S. Blencowe, Isabelle Boutron, Christopher J. Cates, Hung-Yuan Cheng, Mark S. Corbett, Sandra M. Eldridge, Jonathan R. Emberson, Miguel A. Hernán, Sally Hopewell, Asbjørn Hróbjartsson, Daniela R. J...

    2019

  43. [51]

    Guyatt, Andrew D

    Gordon H. Guyatt, Andrew D. Oxman, Elie A. Akl, Regina Kunz, Gunn E. Vist, Jan Brozek, Yaolong Chen, and Holger J. Schünemann. Grade guidelines: 1. introduction—grade evidence profiles and summary of findings tables. Journal of Clinical Epidemiology, 64(4):383–394, 2011. doi:1...

    2011 doi

  44. [52]

    Transforming evidence synthesis: A systematic review of the evolution of automated meta-analysis in the age of ai, 2025

    Lingbo Li, Anuradha Mathrani, and Teo Susnjak. Transforming evidence synthesis: A systematic review of the evolution of automated meta-analysis in the age of ai, 2025. URL https://arxiv.org/abs/2504.20113

    2025 arXiv

  45. [53]

    Agentic reasoning: Reasoning llms with tools for the deep research,

    Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning llms with tools for the deep research,

  46. [54]

    TART: An open-source tool- augmented framework for explainable table-based reasoning

    Xinyuan Lu, Liangming Pan, Yubo Ma, Preslav Nakov, and Min-Yen Kan. TART: An open-source tool- augmented framework for explainable table-based reasoning. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025,...

    2025

  47. [55]

    Medical hallucination in foundation models and their impact on healthcare

    Yubin Kim, Hyewon Jeong, Shen Chen, Shuyue Stella Li, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minseok Jung, Rodrigo R Gameiro, Lizhou Fan, Eugene Park, Tristan Lin, Joonsik Yoon, Wonjin Yoon, Maarten Sap, Yulia Tsvetkov, Paul Pu Liang, Xuhai Xu, Xin Liu, Daniel M...

    2025 doi

  48. [56]

    Yeo Wei Jie, Ranjan Satapathy, Rick Goh, and Erik Cambria. How interpretable are reasoning explanations from prompting large language models? In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024 , pages ...

    2024 doi

  49. [57]

    Potential roles of large language models in the production of systematic reviews and meta-analyses

    Xufei Luo, Fengxian Chen, Di Zhu, Ling Wang, Zijun Wang, Hui Liu, Meng Lyu, Ye Wang, Qi Wang, and Yaolong Chen. Potential roles of large language models in the production of systematic reviews and meta-analyses. J Med Internet Res, 26:e56780, 6 2024. ISSN 1438-8871. doi:10.219...

    2024 doi

  50. [62]

    justification

    Data Sources & Priority - Extract from all available sections: abstract, methods, results, tables, figures, appendices(if appaliable). - IMPORTANT: Under no circumstances should the LLM attempt to calculate any statistical values. - *Source Priority:* - *Tables* are the prefer...

  51. [63]

    other_time_points

    Outcome Measures (OM) Standardization - Extract outcome descriptions, time points, and numerical values for both intervention and control groups. - If multiple time points exist, prioritize the *final* follow-up but list all time points in `"other_time_points"`. - Report exact...

  52. [64]

    needs_transformation

    Unit Standardization - Ensure all extracted data maintains its original unit. Always report the original unit, 20 ... A PREPRINT even if the source paper is consistent about the unit used. - ONLY When unit conversions are needed *or if multiple units are reported or implied*, ...

  53. [65]

    null"`, NOT `

    Handling Missing Data - **Numerical Data:** If *numerical* data (e.g., means, standard deviations, sample sizes, p_value) is absent from the PDF, ONLY return `"null"`, NOT `"Not reported" ` or `"NA " `. If the missing numerical data can be calculated from other reported data (...

  54. [66]

    more common in the intervention group

    Adverse Events & Dropouts - Extract total adverse events, serious events, and dropout numbers. - If adverse events are reported for <3 types, extract exact values for each type. For example, extract cardiovascular, gastrointestinal, neurological, psychiatric adverse events, et...

  55. [67]

    data_conflicts

    Data Conflict Handling - If conflicting values exist, store them in `"data_conflicts"`, specifying the sources. - Justify which value is used in `"justification"`. Example: ```json { "data_conflicts": { "sample_size": { "table": 120, "results_section": 115 } }, "justification"...

  56. [68]

    pdf_status

    PDF Processing Status If the PDF is unreadable, return: ```json { "pdf_status": "Unreadable" } ``` Otherwise, return "pdf_status": "Processed". Output Format: Structured JSON with the following keys. Adjust keys as needed to fit specific data to ensure you can capture all data...

    2023

  57. [69]

    pdf_status

    Iterate Through JSON: You MUST systematically iterate through every key-value pair in the initial JSON output, only record issues for fields with errors or inconsistencies in value, source, or confidence, except `"pdf_status"` and `"notes"`

  58. [70]

    - Identify any *structural inconsistencies* (e.g., missing key study characteristics, incomplete sample size reporting)

    Validation Against Source Structure: - Ensure that extracted values match the expected format based on study sections (e.g., outcome measures should be from tables or results, not introduction). - Identify any *structural inconsistencies* (e.g., missing key study characteristi...

  59. [71]

    - *Unit consistency*: Verify all measurements use the correct units (e.g., blood pressure should be in mmHg)

    Internal Consistency Checks: - *Mathematical consistency*: Ensure numerical values are logically consistent (e.g., total sample size = sum of intervention + control). - *Unit consistency*: Verify all measurements use the correct units (e.g., blood pressure should be in mmHg). ...

  60. [72]

    - Identify discrepancies and data conflicts between different sections of the paper (e.g., abstract vs

    Direct PDF Comparison: For *each key-value pair*: - Review the extracted data alongside the original PDF content. - Identify discrepancies and data conflicts between different sections of the paper (e.g., abstract vs. results section vs. tables)

  61. [73]

    source" the most appropriate location in the paper for this data? If not, provide a more accurate source. - Confidence Justification: Is the assigned

    Critical Assessment: For each key-value pair, consider these questions: - Accuracy: Does it exactly match the article’s reported data? - Data Conflicts: Do different sections of the PDF report inconsistent values? - Relevance: Is this data critical for meta-analysis? - Justifi...

  62. [74]

    needs_transformation

    Outcome Measures Verification: - Confirm that all *time points* are correctly extracted and *final follow-up* is prioritised. - If the extracted value includes *median & IQR* or *range* and requires transformation to calculate mean and standard deviation, ensure `"needs_transf...

  63. [75]

    Adverse Events and Dropouts: - Verify if *dropout data* or *adverse events* is extracted (if reported)

  64. [76]

    null"` or `

    Completeness Check: - For fields marked as `"null"` or `"Not reported" ` in the initial JSON, verify that this status accurately reflects the information presented (or lack thereof) in the PDF. Do not attempt to fill in missing values, maintain original status

  65. [77]

    revised_value

    Adjust or Skip: - If the initial extracted value is INCORRECT, include the following keys in the output: - corrected and improved `"revised_value"`. Ensure that only the keys are changed that there is a mismatch to the original extract. DO NOT include unchanged key-value pairs...

  66. [78]

    pdf_status

    Include PDF Status: - Always include `"pdf_status"`: `"Processed"` or `"Unreadable"` in the output, even when corrections are needed, to confirm the PDF was analysed. Definition of "Error": An "error" means the originally extracted value is factually incorrect according to the...

    2023

  67. [79]

    confidence

    For each field: - If two models agree on a value and the third differs, use the majority (2-vs-1 voting). - If all three values are the same, keep it as is. - If all three values are different: - If a "confidence" field is present, choose the one with the highest confidence (p...

  68. [80]

    For nested fields (such as `outcome_bmd`, `participant_characteristics`, etc.), apply the same rules recursively

  69. [81]

    Maintain the original JSON structure in the final result

  70. [82]

    Just return the final merged JSON object

    Do NOT include any explanations or commentary. Just return the final merged JSON object. Please return the final merged JSON. --- ### Model A Output: <PASTE FULL JSON OF MODEL A HERE> ### Model B Output: <PASTE FULL JSON OF MODEL B HERE> ### Model C Output: <PASTE FULL JSON OF...

  71. [84]

    LGL_group

    *Semantic Similarity Match:* If an exact match is *not* found, use your expert knowledge of statistical terminology to identify fields with similar *meaning*. Consider variations in naming conventions. For example: - `"LGL_group"` is likely semantically equivalent to `"interve...

  72. [85]

    **Note:** EXT fields may be nested

    *Missing Field:* If a field exists in GT but is not found in EXT (after exact and semantic matching), you must mark it as `"Missing"` and count it as a False Negative. **Note:** EXT fields may be nested. You must traverse the full structure to find possible semantic matches. *...

  73. [86]

    kg/m²"` and `

    **Numerical Values:** - A numerical value in EXT is considered "Correct" if it falls within ±1% of the corresponding numerical value in GT. Calculate percentage difference as: `abs(EXT_value - GT_value) / GT_value `. - **Units:** The extracted value must be expressed in *equiv...

  74. [87]

    low glycemic load diet

    **String Values:** - A string in EXT is considered **"Correct"** if its **meaning** is semantically equivalent to the GT value. Do **not** require exact character matches. You should: - Ignore case, formatting, hyphens, extra whitespace - Accept rewordings if meaning is unchan...

  75. [88]

    null" or

    **Special Case: "null" or "Not reported" Values in EXT** EXT fields may contain: `"null"` for missing numerical values or `"Not reported" ` or equivalent words for missing non-numerical values. This reflects that the extraction system could not find these values in the origina...

  76. [89]

    *Exact Match:* Use this when both field names are identical

  77. [90]

    randomised controlled trial

    *Semantic Match:* Match fields with equivalent *meaning*, even if the names differ. For example: - `"randomised controlled trial" ` equivalent to `"randomised"` - `"blinding of outcome assessors" ` equivalent to `"outcome assessor blinded" ` - `"ethics approved" ` equivalent t...

  78. [91]

    You may refer to EXT field meta-information (e.g., `source`, `notes`, `confidence`) to aid in field matching, especially when EXT uses vague or ambiguous labels

    *Missing Field:* If no semantically equivalent field exists in EXT, mark the GT field as "Missing". You may refer to EXT field meta-information (e.g., `source`, `notes`, `confidence`) to aid in field matching, especially when EXT uses vague or ambiguous labels. **Step 2: Value...

  79. [92]

    not reported

    **Correct:** if - The EXT value expresses the *same meaning* as GT (even if phrasing differs) - Examples: - `"not reported" ` equivalent to `"not mentioned" ` - `"randomly assigned" ` equivalent to `"randomised"` - `"IRB approved" ` equivalent to `"ethics approval obtained" `

  80. [93]

    **Incorrect Values:** - The EXT value has a *different meaning* than the GT - The value misrepresents study design or mislabels methods

  81. [94]

    null" or

    **Special Case: "null" or "Not reported" Values in EXT** EXT fields may contain: `"Not reported" ` or equivalent words for missing values. This reflects that the extraction system could not find these values in the original PDF (as per extraction prompt instructions). - If GT ...

  82. [95]

    This is the preferred method

    *Exact Name Match:* If a field with precisely the same name exists in both GT and EXT, consider it a direct match. This is the preferred method

  83. [96]

    study_characteristics.PC

    *Semantic Similarity Match:* If an exact match is *not* found, use your expert knowledge of study information terminology to identify fields with similar *meaning*. Consider variations in naming conventions. For example: - `"study_characteristics.PC"` is likely semantically eq...

  84. [97]

    **Note:** GT and EXT fields may be nested

    *Missing Field:* If a field exists in GT but is not found in EXT (after exact and semantic matching), you must mark it as `"Missing"` and count it as a False Negative. **Note:** GT and EXT fields may be nested. You must traverse the full structure to find possible semantic mat...

  85. [98]

    A field is *Correct* if: - The EXT value expresses the *same meaning* as GT (even if phrasing differs) - Ignore case, formatting, hyphens, extra whitespace - Accept rewordings if meaning is unchanged - Evaluate synonym phrases as equivalent

  86. [99]

    Hallucinated

    A field is *Hallucinated* if: - There is a meaningful content mismatch. If meaning is different, mark as `"Hallucinated"` or `"Overgeneralized"` depending on content loss

  87. [100]

    Not reported

    **Special Case: "Not reported" Values in EXT** EXT fields may contain: `"Not reported" ` or equivalent words for missing values. This reflects that the extraction system could not find these values in the original PDF (as per extraction prompt instructions). - If GT expects a ...

    1976

  88. [2010]

    doi:10.1186/1472-6947-10-56

    doi

  89. [2021]

    doi:10.2196/33124

    ISSN 2561-326X. doi:10.2196/33124

    doi

  90. [2024]

    doi:10.18653/v1/2024.findings-naacl.237

    Association for Computational Linguistics. doi:10.18653/v1/2024.findings-naacl.237. URL https: //aclanthology.org/2024.findings-naacl.237/

    2024 doi

  91. [2025]

    URL https://arxiv.org/abs/2502.04644

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.