Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Constructing Executable Analytical Knowledge Representations for Meta-Analysis Synthesis Using an Agentic Harness

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims automated meta-analysis should build an explicit, contract-validated record of analytical decisions before statistical execution, and shows that doing so matches published syntheses far more often than direct LLM generation

desk verdict A genuinely useful explicit intermediate representation for meta-analysis synthesis, but the headline structural-superiority claim is driven by one case and should be reframed as a single-case effect. read the letter →

arxiv 2608.01711 v1 pith:PDUOJT7M submitted 2026-08-03 cs.AI

Lingbo Li , Anuradha Mathrani , Teo Susnjak This is my paper
classification cs.AI
keywords ExecutableAnalyticalKnowledgeRepresentationmeta-analysisagenticAIlargelanguagemodelsevidencesynthesisscientificworkflowautomatedsystematicreview
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Structured study data alone does not tell a computer what to pool: before any meta-analysis can run, someone must decide which measurements measure the same thing at comparable times, which arm is the intervention, which effect-size formula applies, and whether the numbers are in the right form. The paper argues that these decisions should be stored in an explicit, machine-actionable object—the Executable Analytical Knowledge Representation (EAKR)—rather than left implicit inside prompts, generated code, or agent messages. It introduces MetaSynDec, a harness where language models propose updates but deterministic schemas, contracts, and validation rules decide what enters the representation and when statistics may run. Across 58 synthesis units from six published meta-analyses, the harness built all 58 representations, executed 57 analyses, matched the reference evidence set exactly in 75% of units, and reproduced 54 of 55 published confidence intervals. The decisive comparison: with identical inputs and the same underlying model, the EAKR workflow agreed with the published synthesis structure in 57/58 units versus 23/58 for direct generation, and on analytical formulation 23/23 versus 1/23 among jointly completed units.

What carries the argument

The central object is the Executable Analytical Knowledge Representation (EAKR): a structured state, written as ⟨C, E, O, U, A⟩, that records review context, structured evidence, outcome knowledge, executable analysis objects, and provenance/control information. The load-bearing mechanism is the readiness predicate Ready(u), an execution contract requiring valid outcome, valid evidence mapping, valid contrast, valid numerical inputs, complete provenance, and no unresolved blocking issue before any analysis object may pass to deterministic statistical execution. The harness enforces this by stage-bounded transitions: each construction stage may modify only its authorized EAKR fields, and LLM-

What would settle it

Build a gold-standard benchmark where every measurement used by the published meta-analysis—including supplementary-only values—is inventoried in the input, then run MetaSynDec: if exact evidence-set agreement stays at or below 75% and the remaining disagreements no longer trace to knowledge-coverage gaps, the paper's account of its own error budget fails and the representation's role in the fidelity gains is less central than claimed.

Watch

Extended reading notes

Core claim

The central claim is that automated meta-analysis fails not at statistical computation—which existing software already handles—but at constructing an explicit, verifiable representation of the analytical knowledge that connects evidence to computation. The paper defines the EAKR as a structured synthesis state holding review context, evidence, outcome knowledge, planned analysis objects, and provenance, and instantiates it in MetaSynDec, an agentic harness in which LLMs propose bounded updates and deterministic services enforce schema compliance, methodological constraints, and input contracts before execution. The empirical demonstration: all 58 predefined synthesis units produced schema- a

Load-bearing premise

The comparison assumes the structured study records fed to the system contain all the measurements the published meta-analyses actually used, including outcomes reported only in supplementary materials.

Editorial extensions

If this is right

  • If the EAKR-centered workflow is as reliable as reported, automated meta-analysis can be built so every synthesis decision is inspectable and traceable before statistics run, making the analytical layer independently verifiable and reusable.
  • The 37/37 outcome-knowledge fidelity implies that the semantic layer—outcome classification, type, and measurement-scale harmonization—can be externalized and validated, not left to implicit model reasoning.
  • The 54/55 confidence-interval overlap indicates the representation preserves enough numerical fidelity for generated pooled estimates to be close to published results relative to reference uncertainty.
  • The ablation result (57/58 vs 23/58; 23/23 vs 1/23) implies that direct LLM generation systematically loses measurement-time structure and analytical-formulation constraints; the loss is architectural, not a prompt-tuning artifact.
  • The disagreement taxonomy implies that the largest remaining error source is upstream evidence coverage—especially supplementary-material outcomes—rather than the analytical reasoning the paper contributes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer the same representation-centered pattern would transfer to other evidence-grading tasks, such as health technology assessment or guideline development, because those workflows also sit on the gap between structured evidence and executable decisions; this transfer is plausible but untested.
  • A concrete testable extension follows from the paper's own error taxonomy: if supplementary-material retrieval were added to the intake stage, the knowledge-coverage gap should shrink and exact evidence-set agreement should rise above 75%; the paper does not run this experiment.
  • I conjecture that swapping the underlying language model while keeping the schemas and contracts fixed would preserve most of the fidelity gains, because the representational scaffolding rather than model capacity appears to drive agreement—this is my inference, not a paper claim.
  • The EAKR could plausibly serve as an interchange format between evidence-extraction tools and statistical packages, making the analytical layer a durable artifact independent of the LLM that helped construct it; the paper argues for separability but does not demonstrate cross-tool portability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Executable Analytical Knowledge Representation (EAKR), a structured intermediate representation designed to make explicit the analytical decisions—evidence assignment, contrasts, outcome/time-point alignment, effect-size formulation, and methodological admissibility—that separate raw study evidence from executable meta-analysis. The authors operationalize EAKR in MetaSynDec, an agentic harness in which LLMs propose bounded updates and deterministic services validate schema compliance, readiness contracts, and statistical execution. The evaluation covers six meta-analyses comprising 58 synthesis units and 932 structured evidence records, reporting high construction success (58/58 EAKRs, 57/58 executed), high analytical fidelity (e.g., 100% outcome-knowledge fidelity, 67.9% complete analysis-object fidelity), high numerical overlap with published confidence intervals (98.2%), and a system-level ablation claiming that MetaSynDec outperforms direct LLM generation on synthesis-structure agreement (57/58 vs 23/58, p<0.001) and formulation agreement (23/23 vs 1/23, p<0.001).

Significance. If the empirical claims hold, EAKR is a genuinely useful contribution: it converts implicit synthesis reasoning into an inspectable, provenance-rich object that can be validated before computation, and the separation of probabilistic interpretation from deterministic execution is a sound architectural idea. The paper is also unusually transparent in reporting frozen configurations, repeated runs, leave-one-case-out sensitivity, and a qualitative disagreement taxonomy. However, the headline comparative result is much weaker than the abstract suggests: the structural superiority is driven entirely by one meta-analysis case (MA6), the evaluation includes the same case used for development/debugging (MA1), and reported evidence-set fidelity is contaminated by known input-coverage gaps. The current evidence supports a feasibility claim and a single-case demonstration, not a general claim of superiority over direct LLM generation.

major comments (4)
  1. [§5.2, Table 8; Abstract] The headline structural-superiority claim rests entirely on MA6. With MA6 excluded, both conditions achieve structural agreement on all 23 remaining units (23/23 vs 23/23); within MA6 the comparison is roughly 34/35 vs 0/35. The 58 units are nested in six reviews, so McNemar's test on units is invalid pseudo-replication: every discordant pair comes from one review. At the review level there are five ties and one discordant case, so a sign test (excluding ties) gives p=0.5. The abstract's 'outperformed ... p<0.001' is therefore not a general result. Reanalyze with review as the unit of analysis, report a cluster-robust test, or explicitly frame the structural advantage as an MA6-specific finding.
  2. [§4.3, §5] The system was developed and debugged on MA1, and MA1 is included in every reported aggregate (58 units, 37 outcomes, 57/58 execution). Even if no manual records were used, debugging on the exact LLM-extracted records that later appear in the test set is a form of leakage; prompts, rules, or thresholds could have been adjusted until MA1 behaved well. This is particularly problematic because MA1 is one of only six cases. Please either exclude MA1 from the reported evaluation, or show that the main conclusions are unchanged when each development-influenced case is removed.
  3. [§5.3, Table 7] Evidence-set fidelity is partly an artifact of input coverage. The paper itself reports that 8 of 17 disagreements are 'knowledge-coverage gaps' where supplementary-material outcomes were absent from the supplied evidence, and 2 of 58 units were excluded because the reference evidence set could not be reconstructed. The 42/56 (75.0%) exact evidence-set agreement therefore measures the difference between two evidence states, not the analytical quality of EAKR construction. Report evidence-set fidelity separately for units whose input evidence is known to be complete relative to the reference, and distinguish 'input-coverage fidelity' from 'analytical-decision fidelity' in the abstract and §5.1.
  4. [§4.2 EQ4, Table 8, Appendix] The direct-generation baseline may not be task-equivalent. MetaSynDec is evaluated against 58 predefined synthesis tasks (Table 5), whereas the baseline generated 37 outputs for 58 reference units and often combined measurement-time-specific targets. If the direct-generation prompt did not instruct the model to produce one plan per reference unit, the structural-comparison result conflates prompt-specified granularity with the value of the EAKR workflow. The prompt is deferred to an appendix; the main text should state exactly what unit-level information (if any) was given to each condition and how 'matched' units were defined.
minor comments (5)
  1. [§6, first paragraph] Typo: 'and and' appears in the opening sentence of the Discussion.
  2. [§5.1 vs Table 7] The text reports statistical model-policy agreement as 89.5% (51/57), while Table 7 reports 52/57 (91.2%). Please reconcile.
  3. [§5.2 sensitivity paragraph] The phrase 'all 23 remaining reference synthesis units were structurally comparable under both conditions' is ambiguous. Use explicit numbers, e.g., 'structural agreement was 23/23 for both conditions after excluding MA6'.
  4. [§4.3, general] No data or code availability statement is provided. Given the emphasis on traceability and reproducibility, please include a statement about whether the EAKR artifacts, prompts, and evaluation scripts will be released.
  5. [Table 4, §3.3] The outcome-typing weights are fixed but appear ad hoc, and no sensitivity analysis is reported. Since deterministic typing achieves 100% agreement, please state whether plausible variations of these weights would change the result, or justify the weights as part of the frozen configuration.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity; one minor development/evaluation overlap (MA1) does not drive the central claims.

  1. other [Section 4.3 (Implementation Details); Section 4.1 / Table 5 (Evaluation dataset)]
    "The LLM-extracted structured study records from MA1 from our study [52] were used solely as the implementation development case for constructing, integrating, and debugging the end-to-end workflow."

    MA1 was used to build, integrate, and debug the end-to-end workflow, and the same MA1 case is then part of the '58 synthesis units' evaluation set (Table 5). Reported MA1 contributions to construction/fidelity metrics are therefore not independent out-of-sample predictions: the system was iterated against that case before the reported run. This is a real but mild self-use circularity. It is not load-bearing: the headline structural-superiority difference is concentrated in MA6 (the paper's own sensitivity analysis shows 23/23 vs 23/23 outside MA6), and the formulation-agreement claim (23/23 vs 1/23) is computed over the non-MA6 units; removing MA1 would leave 22/22 vs 1/22, so the central conclusions do not reduce to this overlap.

full rationale

The central comparative claims are grounded in external published meta-analyses as references and a direct-generation baseline receiving identical structured inputs and the same underlying model. The derivation chain is therefore not equivalent to its inputs by construction: MetaSynDec's outputs are scored against independent published syntheses, not against quantities used to define the system. The only circular-adjacent element is the use of MA1 for development and debugging before including it in the evaluation; this is concrete but minor and non-load-bearing, as argued above. The remaining concerns noted in the reader's take, such as the MA6-driven pseudo-replication in the structural p-value and the hand-selected outcome-typing weights, are statistical-validity and reporting-transparency issues rather than circularity: they do not make any reported 'prediction' identical to the system's inputs by construction. The paper's explicit configuration freeze, use of temperature=0, and external reference standards support the conclusion that the central feasibility and comparative results have independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or formal entities; EAKR and analysis objects are structured representations, not postulated entities. The main 'extra' elements are hand-chosen weights and iteration bounds, plus a set of domain assumptions about the completeness of input evidence and the reconstructibility of published reference syntheses.

free parameters (4)
  • Outcome typing evidence-strength weights (Table 4) = decisive=3, strong=2, weak=1
    Hand-selected weights for the deterministic outcome-typing service in Section 3.3. They directly determine whether records are typed continuous, binary, or time-to-event, and are fixed without a formal derivation or tuning procedure.
  • Maximum planning attempts K_plan = 3
    Bounded agent iterations in Algorithm 1; this arbitrary limit affects completion rates (52/58 within automated bound, 5/58 needing human action).
  • Maximum stage-local update attempts K_i = 3
    Bound on within-stage reasoning/tool iterations for each agent-assisted stage (Section 3.2); influences whether units require escalation and terminal status.
  • Decoding temperature and top_p = temperature=0.0, top_p=1.0
    Sampling parameters frozen before evaluation; reported run-to-run stability depends on these choices, which suppress stochastic variation and may mask instability that higher temperatures would reveal.
assumptions (4)
  • domain assumption Published meta-analyses are reliable ground truth and are unambiguously reconstructible from the source publications
    EQ2 fidelity compares against reference syntheses reconstructed from publications. The paper itself excludes 2 of 58 units because the reference evidence set could not be unambiguously reconstructed and documents 4 'reference-synthesis reproducibility discrepancies' where the published result requires reinterpreting reported standard errors as standard deviations.
  • domain assumption The structured evidence records supplied to MetaSynDec accurately and completely represent the source studies' reported results
    The evaluation uses curated structured inputs. Section 6.5 states that upstream extraction errors propagate into the EAKR, and Section 5.3 shows that missing supplementary-material outcomes caused 8 of 17 disagreements, so the evidence-completeness assumption is load-bearing.
  • domain assumption Established meta-analytic principles (Cochrane Handbook, Hedges and Olkin) correctly define what constitutes a valid synthesis
    These principles inform the EAKR schema, readiness predicates, and validation contracts (Section 3.1). If the methodological standards are inappropriate for a given review, the system inherits that error, as the paper acknowledges for aggregate pairwise designs.
  • domain assumption LLM reasoning within bounded stages is a reliable source of semantic interpretation
    The agentic design depends on the Mistral-Small model correctly performing outcome grounding, contrast assignment, and time-point selection. The paper provides no lower bound on LLM reliability; 5 of 17 disagreements were attributed to semantic-inference failures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Constructing Executable Analytical Knowledge Representations for Meta-Analysis Synthesis Using an Agentic Harness." pith.science (2026). https://pith.science/paper/PDUOJT7M

@misc{pith2026260801711,
  author       = {Pith},
  title        = {Pith review of: Constructing Executable Analytical Knowledge Representations for Meta-Analysis Synthesis Using an Agentic Harness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDUOJT7M}},
  note         = {Machine review of arXiv:2608.01711}
}
read the original abstract

Meta-analysis synthesis highlights a fundamental challenge in knowledge-based scientific analysis: structured evidence does not by itself represent the analytical knowledge required for executable computation. Decisions about evidence assignment, analytical contrasts, outcome and time-point alignment, effect-size formulation, and methodological admissibility must be explicit before statistical execution. Existing automated approaches often embed these decisions in model outputs, generated code, or workflow traces rather than representing them as independently verifiable knowledge. We introduce the Executable Analytical Knowledge Representation (EAKR), a machine-actionable representation of the knowledge required to transform structured evidence into executable meta-analysis. An EAKR represents evidence, relations, numerical inputs, constraints, provenance, and unresolved issues. We operationalise EAKR in MetaSynDec, an agentic harness in which large language models propose structured updates and deterministic services govern schema- and contract-based validation and execution. Across 58 synthesis units, MetaSynDec constructed all EAKRs, with 57 proceeding to statistical execution. Of 56 units with sufficient information to define a reference analysis object, 38 (67.9%) achieved complete object fidelity and 42 (75.0%) exact evidence-set agreement, with a mean Jaccard similarity of 0.909. Generated and published confidence intervals overlapped in 54 of 55 units (98.2%). MetaSynDec outperformed direct LLM generation in reference synthesis-structure agreement (57/58 versus 23/58; p<0.001) and among 23 jointly completed units, exact reference-formulation agreement (23/23 versus 1/23; p<0.001). These findings provide feasibility evidence that EAKR supports formal validation, traceability, statistical execution, and improved methodological agreement relative to direct LLM generation.

Figures

Figures reproduced from arXiv: 2608.01711 by the authors.

Figure 1
Figure 1. Meta-analysis synthesis as evidence routing before statistical combination. Clinical studies are first trans [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Conceptual shift from generation-centred reasoning to representation-centred reasoning. Existing approaches [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. MetaSynDec agentic harness for constructing Executable Analytical Knowledge Representations. The harness [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of knowledge and numerical fidelity across synthesis units. (A) Evidence-set Jaccard similarity [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: Diagnosed sources of synthesis disagreement. Among the 17 disagreement units examined, eight reflected [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 42 canonical work pages

  1. [1]

    Nikil Ravi, Pranshu Chaturvedi, E. A. Huerta, Zhengchun Liu, Ryan Chard, Aristana Scourtas, Ben Blaiszik, and Ian Foster. FAIR principles for AI models with a practical application for accelerated high energy diffraction 26 ...A PREPRINT microscopy.Scientific Data, 9(1):657, 2022. doi:10.1038/s41597-022-01712-9. URL https://doi.org/10. 1038/s41597-022-01712-9

    work page doi:10.1038/s41597-022-01712-9 2022

  2. [2]

    E. A. Huerta, Ben Blaiszik, L. Catherine Brinson, Kristofer E. Bouchard, Daniel Diaz, Caterina Doglioni, Javier M. Duarte, Murali Emani, Ian Foster, Geoffrey Fox, Philip Harris, Lukas Heinrich, Shantenu Jha, Daniel S. Katz, V olodymyr Kindratenko, Christine R. Kirkpatrick, Kati Lassila-Perini, Ravi K. Madduri, Mark S. Neubauer, Fotis E. Psomopoulos, Avik ...

    2023

  3. [3]

    Alvaro A. A. Fernandes, Martin Koehler, Nikolaos Konstantinou, Pavel Pankin, Norman W. Paton, and Rizos Sakellariou. Data preparation: A technological perspective and review.SN Computer Science, 4(4):425, 2023. doi:10.1007/s42979-023-01828-8. URLhttps://doi.org/10.1007/s42979-023-01828-8

    doi:10.1007/s42979-023-01828-8 2023

  4. [4]

    SAGE Publications, Inc, Thousand Oaks, California, 5 edition, 2017

    Harris Cooper.Research Synthesis and Meta-Analysis: A Step-by-Step Approach. SAGE Publications, Inc, Thousand Oaks, California, 5 edition, 2017. doi:10.4135/9781071878644. URL https://methods.sagepub. com/book/mono/research-synthesis-and-meta-analysis-5e/toc

    work page doi:10.4135/9781071878644 2017

  5. [5]

    Analysing data and undertaking meta-analyses, chapter 10, pages 241–284

    Jonathan J Deeks, Julian PT Higgins, Douglas G Altman, and on behalf of the Cochrane Statistical Methods Group. Analysing data and undertaking meta-analyses, chapter 10, pages 241–284. John Wiley & Sons, Ltd, 2019. ISBN 9781119536604. URLhttps://onlinelibrary.wiley.com/doi/abs/10.1002/9781119536604.ch10

    work page doi:10.1002/9781119536604.ch10 2019

  6. [6]

    John Wiley & Sons, 2008

    Matthias Egger, George Davey Smith, and Douglas Altman.Systematic reviews in health care: meta-analysis in context. John Wiley & Sons, 2008

    work page 2008

  7. [7]

    Academic press, 2014

    Larry V Hedges and Ingram Olkin.Statistical methods for meta-analysis. Academic press, 2014

    work page 2014

  8. [8]

    Marshall and Byron C

    Iain J. Marshall and Byron C. Wallace. Toward systematic review automation: A practical guide to using machine learning tools in research synthesis.Systematic Reviews, 8(1):163, 2019. doi:10.1186/s13643-019-1074-9

    doi:10.1186/s13643-019-1074-9 2019

Show all 56 references
  1. [9]

    F. W. Mutinda, K. Liew, S. Yada, S. Wakamiya, and E. Aramaki. Automatic data extraction to support meta- analysis statistical analysis: a case study on breast cancer.BMC Med Inform Decis Mak, 22(1):158, June 2022. ISSN 1472-6947. doi:10.1186/s12911-022-01897-4

    2022 doi

  2. [10]

    F. W. Mutinda, S. Yada, S. Wakamiya, and E. Aramaki. AUTOMETA: Automatic Meta-Analysis System Employing Natural Language Processing.Studies in health technology and informatics, 290:612–616, June 2022. ISSN 0926-9630. doi:10.3233/shti220150

    2022 doi

  3. [11]

    Finnerty Mutlu, Rob Elmore, Babatunde K

    Lena Schmidt, Amy N. Finnerty Mutlu, Rob Elmore, Babatunde K. Olorisade, James Thomas, and Julian P. T. Higgins. Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Research, 10:401, 2025. doi:10.12688/f1000research.51117.3

    2025 doi

  4. [12]

    A Latent Variable Approach for Meta-Analysis of Gene Expression Data from Multiple Microarray Experiments.BMC Bioinformatics, 8(1): 364, December 2007

    Hyungwon Choi, Ronglai Shen, Arul M Chinnaiyan, and Debashis Ghosh. A Latent Variable Approach for Meta-Analysis of Gene Expression Data from Multiple Microarray Experiments.BMC Bioinformatics, 8(1): 364, December 2007. ISSN 1471-2105. doi:10.1186/1471-2105-8-364. URL https://...

    2007 doi

  5. [13]

    Moderated effect size andP -value combinations for microarray meta-analyses.Bioinformatics, 25(20):2692–2699, October 2009

    Guillemette Marot, Jean-Louis Foulley, Claus-Dieter Mayer, and Florence Jaffrézic. Moderated effect size andP -value combinations for microarray meta-analyses.Bioinformatics, 25(20):2692–2699, October 2009. ISSN 1367- 4811, 1367-4803. doi:10.1093/bioinformatics/btp444. URL htt...

    2009 doi

  6. [14]

    Conducting Meta-Analyses inRwith themetaforPackage.Journal of Statistical Software, 36(3), 2010

    Wolfgang Viechtbauer. Conducting Meta-Analyses inRwith themetaforPackage.Journal of Statistical Software, 36(3), 2010. ISSN 1548-7660. doi:10.18637/jss.v036.i03. URLhttp://www.jstatsoft.org/v36/i03/

    2010 doi

  7. [15]

    Riley, Kym Snell, Johannes B

    Thomas Pa Debray, Johanna Aag Damen, Richard D. Riley, Kym Snell, Johannes B. Reitsma, Lotty Hooft, Gary S. Collins, and Karel Gm Moons. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes.Statistical Methods in Medical Research, 28...

    2019 doi

  8. [16]

    Cumpston, Joanne E

    Miranda S. Cumpston, Joanne E. McKenzie, James Thomas, and Sue E. Brennan. The use of PICO for synthesis and methods for synthesis without meta-analysis: Protocol for a survey of current practice in systematic reviews of health interventions.F1000Research, 9:678, 2021. doi:10....

    2021 doi

  9. [17]

    McKenzie, Sue E

    Joanne E. McKenzie, Sue E. Brennan, Rebecca E. Ryan, Hilary J. Thomson, and Rachel V . Johnston. Sum- marizing study characteristics and preparing for synthesis. In Julian P. T. Higgins, James Thomas, Jacque- line Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and V...

    2019 doi

  10. [18]

    Potential roles of large language models in the production of systematic reviews and meta-analyses

    Xiao Luo et al. Potential roles of large language models in the production of systematic reviews and meta-analyses. Journal of Medical Internet Research, 26:e56780, 2024. doi:10.2196/56780

    2024 doi

  11. [19]

    Dmitry Scherbakov, Nina Hubig, Vinita Jansari, Alexander Bakumenko, and Leslie A. Lenert. The emergence of large language models as tools in literature reviews: An LLM-automated systematic review.Journal of the American Medical Informatics Association, 32(6):1071–1086, 2025. d...

    2025 doi

  12. [20]

    Tim Reason, Emma Benbow, Julia Langham, Andy Gimblett, Sven L Klijn, and Bill Malcolm. Artificial intelligence to automate network meta-analyses: four case studies to evaluate the potential application of large language models.PharmacoEconomics-Open, 8(2):205–220, 2024

    2024

  13. [21]

    Manalyzer: End-to-end automated meta-analysis with multi-agent system.arXiv preprint arXiv:2505.20310, 2025

    Wanghan Xu, Wenlong Zhang, Fenghua Ling, Ben Fei, Yusong Hu, Fangxuan Ren, Jintai Lin, Wanli Ouyang, and Lei Bai. Manalyzer: End-to-end automated meta-analysis with multi-agent system.arXiv preprint arXiv:2505.20310, 2025

    2025

  14. [22]

    Metamind: A multi-agent transformer-driven framework for automated network meta-analyses.Plos one, 21(2):e0342895, 2026

    Achilleas Livieratos, Maria Kudela, Yuxi Zhao, All-shine Chen, Xin Luo, Junjing Lin, Di Zhang, Sai Dharmarajan, Sotirios Tsiodras, Vivek Rudrapatna, et al. Metamind: A multi-agent transformer-driven framework for automated network meta-analyses.Plos one, 21(2):e0342895, 2026

    2026

  15. [23]

    Cochrane Training. Revman. https://training.cochrane.org/online-learning/core-software# RevMan, 2026. Accessed: 2026-06-17

    2026

  16. [24]

    Comprehensive meta-analysis

    Biostat, Inc. Comprehensive meta-analysis. https://www.meta-analysis.com/, 2026. Accessed: 2026-06- 17

    2026

  17. [25]

    Stata.https://www.stata.com/, 2026

    StataCorp. Stata.https://www.stata.com/, 2026. Accessed: 2026-06-17

    2026

  18. [26]

    Spss statistics.https://www.ibm.com/spss, 2026

    IBM. Spss statistics.https://www.ibm.com/spss, 2026. Accessed: 2026-06-17

    2026

  19. [27]

    Automating meta-analyses of randomized clinical trials: a first look

    Matthew Michelson. Automating meta-analyses of randomized clinical trials: a first look. In2014 AAAI Fall Symposium Series, 2014

    2014

  20. [28]

    Exploration of meta analysis automation

    Xu Yang, Haijing Tang, Xiaonan Dongye, and Guo Chen. Exploration of meta analysis automation. In2018 International Conference on Network Infrastructure and Digital Content (IC-NIDC), pages 218–222, 2018. doi:10.1109/ICNIDC.2018.8525710

    2018

  21. [29]

    Revmanhal: towards automatic text generation in systematic reviews

    Mercedes Torres Torres and Clive E Adams. Revmanhal: towards automatic text generation in systematic reviews. Systematic reviews, 6(1):27, 2017

    2017

  22. [30]

    Hughey, Sandra Andorf, Hao-Chih Lee, Pier Federico Gherar- dini, Matthew H

    Zicheng Hu, Chethan Jujjavarapu, Jacob J. Hughey, Sandra Andorf, Hao-Chih Lee, Pier Federico Gherar- dini, Matthew H. Spitzer, Cristel G. Thomas, John Campbell, Patrick Dunn, Jeff Wiser, Brian A. Kidd, Joel T. Dudley, Garry P. Nolan, Sanchita Bhattacharya, and Atul J. Butte. M...

    2018 doi

  23. [31]

    S. Y . Ryu and G. A. Wendt. MetaMSD: meta analysis for mass spectrometry data.PeerJ, 2019. URL https: //peerj.com/articles/6699/

    2019

  24. [32]

    M. Lam, S. Awasthi, H. J. Watson, J. Goldstein, G. Panagiotaropoulou, V . Trubetskoy, R. Karlsson, O. Frei, C. C. Fan, W. De Witte, N. R. Mota, N. Mullins, K. Brügger, S. H. Lee, N. R. Wray, N. Skarabis, H. Huang, B. Neale, M. J. Daly, M. Mattheisen, R. Walters, and S. Ripke. ...

    2020 doi

  25. [33]

    Sabates, S

    J. Sabates, S. Belleville, M. Castellani, T. Dwolatzky, B. M. Hampstead, A. Lampit, S. Simon, K. Anstey, B. Goodenough, S. Mancuso, D. Marques, R. Sinnott, and A. Bahar-Fuchs. CogTale: an online platform for the evaluation, synthesis, and dissemination of evidence from cogniti...

    2021 doi

  26. [34]

    Burgard, M

    T. Burgard, M. Bosnjak, and R. Studtrucker. PsychOpen CAMA: Publication of community-augmented meta- analyses in psychology.Res Synth Methods, 13(1):134–143, January 2022. ISSN 1759-2879 (Print) 1759-2879. doi:10.1002/jrsm.1536

    2022 doi

  27. [35]

    Tools to support the automation of systematic reviews: a scoping review.Journal of clinical epidemiology, 144:22–42, 2022

    Hanan Khalil, Daniel Ameen, and Armita Zarnegar. Tools to support the automation of systematic reviews: a scoping review.Journal of clinical epidemiology, 144:22–42, 2022

    2022

  28. [36]

    Kristen L Scotti, Sarah Young, Melanie A Gainey, and Haoyong Lan. Artificial intelligence and automation in evidence synthesis: An investigation of methods employed in cochrane, campbell collaboration, and environmental evidence reviews.Cochrane Evidence Synthesis and Methods,...

    2025

  29. [37]

    Transforming evidence synthesis: A systematic review of the evolution of automated meta-analysis in the age of AI.Research Synthesis Methods, 17(3):403–450, 2026

    Lingbo Li, Anuradha Mathrani, and Teo Susnjak. Transforming evidence synthesis: A systematic review of the evolution of automated meta-analysis in the age of AI.Research Synthesis Methods, 17(3):403–450, 2026. doi:10.1017/rsm.2025.10065

    2026

  30. [38]

    Lattereview: a multi-agent framework for systematic review automation using large language models.arXiv preprint arXiv:2501.05468, 2025

    Pouria Rouzrokh, Bardia Khosravi, Parsa Rouzrokh, and Moein Shariatnia. Lattereview: a multi-agent framework for systematic review automation using large language models.arXiv preprint arXiv:2501.05468, 2025

    2025

  31. [39]

    Bing Hu, Emmalie Tomini, Tricia Corrin, Kusala Pussegoda, Elias Sandner, Andre Henriques, Alice Simniceanu, Luca Fontana, Andreas Wagner, Stephanie Brazeau, et al. Enhancing evidence synthesis efficiency: Leveraging large language models and agentic workflows for optimized lit...

    2025

  32. [40]

    Automation of systematic reviews with large language models

    Christian Cao, Rohit Arora, Paul Cento, Adil Budak, Katherine Manta, Elina Farahani, Matthew Cecere, Anabel Selemon, Jason Sang, Ling Xi Gong, et al. Automation of systematic reviews with large language models. medRxiv, pages 2025–06, 2025

    2025

  33. [41]

    Empowering meta-analysis: Leveraging large language models for scientific synthesis

    Jawad Ibn Ahad, Rafeed Mohammad Sultan, Abraham Kaikobad, Fuad Rahman, Mohammad Ruhul Amin, Nabeel Mohammed, and Shafin Rahman. Empowering meta-analysis: Leveraging large language models for scientific synthesis. In2024 IEEE International Conference on Big Data (BigData), page...

    2024

  34. [42]

    Autogen: Enabling next-gen llm applications via multi-agent conversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. InFirst conference on language modeling, 2024

    2024

  35. [43]

    Langgraph.https://github.com/langchain-ai/langgraph, 2024

    LangChain Inc. Langgraph.https://github.com/langchain-ai/langgraph, 2024

    2024

  36. [44]

    What is a knowledge representation?AI Magazine, 14(1): 17–33, 1993

    Randall Davis, Howard Shrobe, and Peter Szolovits. What is a knowledge representation?AI Magazine, 14(1): 17–33, 1993. doi:10.1609/aimag.v14i1.1029

    1993 doi

  37. [45]

    MLIR: Scaling compiler infrastructure for domain specific computation

    Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Oleksandr Zinenko. MLIR: Scaling compiler infrastructure for domain specific computation. In2021 IEEE/ACM International Symposium on ...

    2021

  38. [46]

    xDSL: Sidekick compilation for SSA-based compilers

    Mathieu Fehr, Michel Weber, Christian Ulmann, Alexandre Lopoukhine, Martin Lücke, Théo Degioanni, Michel Steuwer, and Tobias Grosser. xDSL: Sidekick compilation for SSA-based compilers. InProceedings of the 23rd ACM/IEEE International Symposium on Code Generation and Optimizat...

    2025

  39. [47]

    Crusoe, Laura Rodríguez-Navas, Raül Sirvent, Alexander Kanitz, Paul De Geest, Rudolf Wittner, Luca Pireddu, Daniel Garijo, José M

    Simone Leo, Michael R. Crusoe, Laura Rodríguez-Navas, Raül Sirvent, Alexander Kanitz, Paul De Geest, Rudolf Wittner, Luca Pireddu, Daniel Garijo, José M. Fernández, et al. Recording provenance of workflow runs with RO-Crate.PLOS ONE, 19(9):e0309210, 2024. doi:10.1371/journal.p...

    2024 doi

  40. [49]

    Discovering operational decisions from data—a framework supporting decision discovery from data.DECISION, 51:417–436, 2024

    Sam Leewis, Koen Smit, and Johan Versendaal. Discovering operational decisions from data—a framework supporting decision discovery from data.DECISION, 51:417–436, 2024. doi:10.1007/s40622-024-00402-2

    2024 doi

  41. [50]

    Model-based decision support for knowledge-intensive processes.Journal of Intelligent Information Systems, 61:143–165, 2023

    Anjo Seidel, Stephan Haarmann, and Mathias Weske. Model-based decision support for knowledge-intensive processes.Journal of Intelligent Information Systems, 61:143–165, 2023. doi:10.1007/s10844-022-00770-0

    2023 doi

  42. [51]

    Julian P. T. Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and Vivian A. Welch, editors.Cochrane Handbook for Systematic Reviews of Interventions. Cochrane, version 6.5 edition, 2024. URLhttps://www.cochrane.org/handbook. Updated A...

    2024

  43. [52]

    good enough

    Lingbo Li, Anuradha Mathrani, and Teo Susnjak. What level of automation is “good enough”? a benchmark of large language models for meta-analysis data extraction.Research Synthesis Methods, 17(4):671–692, 2026. doi:10.1017/rsm.2025.10066

    2026

  44. [53]

    Mistral-Small-3.2-24B-Instruct-2506

    Mistral AI. Mistral-Small-3.2-24B-Instruct-2506. https://huggingface.co/mistralai/ Mistral-Small-3.2-24B-Instruct-2506, 2025. Hugging Face model card. Accessed: 2026-06-19

    2025

  45. [54]

    The role of metadata in reproducible computational research.Patterns, 2(9):100322, 2021

    Jeremy Leipzig, Daniel Nüst, Charles Tapley Hoyt, Stian Soiland-Reyes, Karthik Ram, and Jane Green- berg. The role of metadata in reproducible computational research.Patterns, 2(9):100322, 2021. doi:10.1016/j.patter.2021.100322

    2021

  46. [55]

    Introducing swirl: an intermediate representation language for scientific workflows

    Iacopo Colonnelli, Doriana Medi ´c, Alberto Mulone, Viviana Bono, Luca Padovani, and Marco Aldinucci. Introducing swirl: an intermediate representation language for scientific workflows. InInternational Symposium on Formal Methods, pages 226–244. Springer, 2024. 29 ...A PREPRINT

    2024

  47. [56]

    text to plan

    Sudhir Agarwal and Anu Sreepathy. Tic: Translate-infer-compile for accurate “text to plan” using llms and logical representations. InInternational Conference on Neural-Symbolic Learning and Reasoning, pages 222–244. Springer, 2024

    2024

  48. [57]

    Pritchard and Andreas Wicenec

    Nicholas J. Pritchard and Andreas Wicenec. Formal definition and implementation of reproducibility tenets for computational workflows.Future Gener. Comput. Syst., 166(C), May 2025. ISSN 0167-739X. doi:10.1016/j.future.2024.107684. URLhttps://doi.org/10.1016/j.future.2024.107684. 30

    2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.