Pith. sign in

REVIEW 6 major objections 6 minor 72 references

An agentic AI system can run a complete meta-analysis end to end, producing pooled effect estimates within 0.12 Hedges' g of expert syntheses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:43 UTC pith:OYXJ5XTN

load-bearing objection Sensible agent architecture for automated meta-analysis, but the validation is too entangled with the benchmark to support the headline claim of close agreement with expert meta-analyses. the 6 major comments →

arxiv 2607.15247 v1 pith:OYXJ5XTN submitted 2026-07-16 cs.AI

AutoSynthesis: An agentic system for automated meta-analysis

Moein Taherinezhad , Sebastian Maier , Gerardo Vitagliano , Francesco Pierri , Stefan Feuerriegel This is my paper
classification cs.AI
keywords automated meta-analysismulti-agent LLM systemevidence synthesisHedges' grandom-effects meta-analysisPRISMAliterature retrievalstatistical extraction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AutoSynthesis is a multi-agent system that attempts to automate the entire meta-analysis pipeline: given a research question in plain language, it plans a search, retrieves papers, screens and extracts quantitative results, computes standardized effect sizes, and runs a random-effects meta-analysis with heterogeneity and bias diagnostics, producing a PRISMA-aligned report. The paper's central claim is that this automated pipeline produces pooled effect estimates close to those of expert-conducted meta-analyses; in the main demonstration the estimated Hedges' g was 0.143 versus 0.020 in the human benchmark, a difference of 0.123, within a tolerance of ±0.20 used in reanalysis studies. If correct, the system would make quantitative evidence synthesis dramatically cheaper and faster (about half an hour and roughly $1.5 per run in the reported case) and could support near-real-time 'living' meta-analyses that update as new studies appear.

Core claim

The central claim is that an end-to-end agentic LLM system can carry out a complete meta-analysis—search, screening, full-text eligibility, statistical extraction and validation, effect-size computation, and random-effects synthesis—and arrive at quantitative conclusions close to those of expert reviewers. In the headline application, the system retrieved 28 candidate records, included 8 studies with 20 effect sizes, and estimated a pooled Hedges' g of 0.143 (95% CI [0.059, 0.226]); the human benchmark reported g = 0.020 (95% CI [-0.048, 0.093]), a gap of 0.123 that falls within the paper's stated tolerance region of ±0.20. The system also produced similar heterogeneity (I² = 88.3% vs 75.97%

What carries the argument

The load-bearing mechanism is a modular multi-agent pipeline in which specialised LLM agents handle reasoning (planning, eligibility, extraction, validation, relevance selection) while deterministic statistical modules compute effect sizes and run the meta-analysis. The critical component is the extraction-and-filter chain: a statistical extractor first builds a structured 'paper map' and pulls numbers only for pre-identified outcomes; a validation agent then verifies every extracted value verbatim (or within rounding) against the source text, marking absent core statistics as 'hallucinated'; and a relevance filter agent classifies each result by outcome type and a fixed priority rank (behav

Load-bearing premise

The benchmark meta-analysis that the system is compared against is truly independent of how the system was designed.

What would settle it

Run AutoSynthesis against a pre-registered, expert-conducted meta-analysis in a different discipline (e.g., clinical medicine) where the review team has no connection to the developers, with prompts frozen and no manual corrections; if the pooled Hedges' g deviates by more than ±0.20 from the expert estimate, or if the system includes fewer than half of the benchmark's studies, the central claim of generalizable automated meta-analysis fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A full meta-analysis can be produced in a few hours and at negligible monetary cost, removing the main barrier to quantitative evidence synthesis for small teams and under-resourced fields.
  • The modular workflow lets researchers update an existing review by starting after planning and retrieval, so newly published studies can be folded in continuously—'living' meta-analysis becomes practical.
  • Because every intermediate decision and extracted value is logged and auditable, automated syntheses could make evidence reviews more transparent and reproducible than many manual efforts.
  • The system's built-in diagnostics (Egger's test, trim-and-fill, leave-one-out, risk-of-bias tools) mean that even non-specialists can produce methodologically standard meta-analyses.
  • If the ±0.12 agreement is seen as a tolerance bound, automated synthesis is credible as a rapid check of whether a new result would change the pooled conclusion of an existing review.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: the benchmark is not fully independent—two of the benchmark's authors overlap with the system's developers, and the relevance-filter prompt contains a hand-coded outcome-priority list that encodes the benchmark's selection logic. The observed ±0.12 agreement may therefore overstate how well the system would perform on an unfamiliar research question without similar prompt tuning.
  • My inference: the system's reliance on verifiable verbatim numbers means it will likely fail on poorly reported studies, statistics in figures, or non-standard reporting—so its practical niche may be well-reported, machine-readable literature rather than the messy full corpus of published research.
  • My inference: a cheaper and safer use is continuous updating of already-validated meta-analyses, where the protocol is fixed and the marginal task is extraction and pooling of new studies; this is where the automation risk is lowest.
  • My inference: the 'corrected recall' adjustment, in which a missing study was later judged relevant by the benchmark team, moves the target after the run; a pre-registered evaluation without post-hoc corrections would be a stiffer test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper presents AutoSynthesis, a multi-agent LLM-based system that attempts to automate the full meta-analysis pipeline: search, screening, eligibility, extraction, effect-size computation, random-effects synthesis, bias assessment, and PRISMA-style reporting. It evaluates the system on one benchmark task—LLM persuasion—against a published meta-analysis by Hölbling et al. (2025). The main quantitative comparison reports AutoSynthesis pooled Hedges' g = 0.143 (95% CI [0.059, 0.226], p < 0.001) versus the benchmark g = 0.020 (95% CI [−0.048, 0.093], p = 0.53), and a study-level correlation of r = 0.69 (p = 0.085, n = 7). The paper claims 'similar' pooled estimates and emphasizes scalability, transparency, and low cost.

Significance. If the central claim were established, AutoSynthesis would be a valuable contribution: an end-to-end automated meta-analysis pipeline with deterministic statistical computation, audit logs, and an estimated cost of about $1.5 per run. The system's modular design and explicit separation of LLM reasoning from statistical calculation are strengths. However, the headline result rests on a single validation benchmark that is not independent of the system's design: two authors of the benchmark are also authors of this paper, and the relevance-filter prompt encodes an outcome-priority ranking that matches the benchmark's selection logic. Moreover, the reported agreement is weaker than the abstract suggests: the pooled difference exceeds the stated ±0.12 bound, the confidence intervals lead to different significance conclusions, and the study-level correlation is not statistically significant. The architecture may be promising, but the current evidence does not support the strong claim of close agreement with expert meta-analyses.

major comments (6)
  1. [Abstract and §2.3] The claim 'within ±0.12 Hedges' g' is contradicted by the paper's own results. Table S2 gives Δg = 0.123, which exceeds 0.12. The text then switches to a ±0.20 tolerance from reference [39], but that standard is from a reanalysis context, not from a comparison between a new automated pipeline and an expert meta-analysis using different evidence sets. The two pooled CIs overlap only in a narrow region [0.059, 0.093] and yield different conclusions (p<0.001 vs p=0.53). This discrepancy undermines the central claim of 'similar' pooled estimates.
  2. [§4.2 and Supplementary 'Relevance filter agent' prompt] The validation benchmark is not independent of the system. Hölbling et al. [12] is co-authored by Maier and Feuerriegel, two of the present authors. More importantly, the relevance-filter prompt hard-codes a priority ranking (behavior=1, compliance/persuasion=2, attitude=3, intention=4, ...) that mirrors the benchmark's outcome-selection logic. Because the final pooled estimate depends on which extracted rows are selected, the observed agreement may reflect tuning to this benchmark rather than generalizable automated meta-analysis. The paper should either provide an external benchmark or clearly present this as a case study, not as validation.
  3. [§2.2] Study selection agreement is modest: 5/7 recall (71.4%) and 62.5% precision. The corrected recall/precision of 85.7%/87.5% is obtained after a 'qualitative review by the lead author of the benchmark meta-analysis'—who is also a co-author of the present paper. This subjective adjudication is not an objective measure and changes the headline numbers substantially. With only 7 benchmark studies, the result is fragile.
  4. [§2.3 and Fig. 4a] The study-level agreement is not statistically significant (r = 0.69, p = 0.085, n = 7), and the OLS slope β = 1.62 indicates systematic inflation of effect sizes. The paper characterizes this as 'moderate-to-strong' association, which overstates the evidence. Additionally, the pooled comparison is not apples-to-apples: AutoSynthesis used 8 studies with 20 effect sizes versus the benchmark's 7 studies with 12 effect sizes. The difference in pooled estimates therefore reflects not only effect-size computation but also retrieval, eligibility, and outcome-selection decisions.
  5. [§4.1, Table S4; Fig. 2h] Multiple effect sizes from the same study are treated as independent observations in the random-effects meta-analysis (e.g., S7, S8, S19, S20 each contribute multiple rows). No multilevel or robust-variance adjustment is described. With k=20 effect sizes from n=8 studies and I²=88.3%, the pooled effect and its confidence interval may be materially biased by this dependence. The paper lists multiple-effect-size handling as a strength, but it does not account for the resulting non-independence in the statistical synthesis.
  6. [§1, §2, §4.2] The introduction claims 'several automated meta-analyses across different scientific domains, including human–AI interaction, education, and psychology' and that 'Across all reference cases, the pooled effect estimates... were within ±0.12'. However, only one benchmark (Hölbling et al.) is presented in the main text and supplementary materials. No results are shown for education or psychology, so the generality claim is unsupported by the evidence in the manuscript.
minor comments (6)
  1. [Abstract / §1] 'within ±0.12' should be corrected to 0.123 or the claim should be revised, since the reported Δg exceeds the stated bound.
  2. [§2.1 vs Fig. 2h / Table S2] The text reports p<0.001, while Fig. 2h and Table S2 report p=0.0008. These should be reconciled.
  3. [§2.3] The statement that the two confidence intervals 'overlapped to a large extent' is inaccurate; the overlap is only [0.059, 0.093], a small fraction of either interval.
  4. [Fig. 2i caption] 'ROINS-I' should be 'ROBINS-I'.
  5. [Table S2 header] 'bechmark' is a typo for 'benchmark'.
  6. [§4.2] The description of Hölbling et al. says k=12 effect sizes, but §2.3 and Table S2 say the benchmark has k=12; consistency is fine, but the earlier text in §2.1 says k=11 effect sizes. Please correct this inconsistency.

Circularity Check

2 steps flagged

The pooled-effect 'prediction' is partly by construction: the relevance-filter prompt hard-codes benchmark-like outcome priorities, and the validation benchmark is co-authored by the present team and adjudicated by its own lead author.

specific steps
  1. fitted input called prediction [Supplementary Materials, 'Relevance filter agent' system prompt]
    "6. PRIORITY RANK (1 = highest) 1=behavior, 2=compliance/persuasion, 3=attitude, 4=intention, 5=resistance, 6=experiential, 7=mechanism, 8=descriptive"

    The relevance-filter agent is the gate that determines which extracted statistical rows enter the meta-analysis (Table S3: 'classifies extracted rows and selects a single winning result per study–outcome group'). The priority list hard-codes an outcome hierarchy (behavior > compliance/persuasion > attitude > intention) that matches the outcome taxonomy used in the benchmark meta-analysis [12] (see Table S5: behavioral persuasion/compliance, accuracy, attitude change, intention). Because the pooled Hedges' g is computed only over the selected rows, the 'predicted' pooled estimate is not an independent out-of-sample result: a designer who knows the benchmark's selection logic can encode it into this list, making agreement with the benchmark partly by construction. No independent validation o

  2. self citation load bearing [§4.2 Evaluation]
    "Finally, because one author of the present study overlaps with the original lead of the review team, we can complement the quantitative benchmark with a qualitative assessment of where AUTOSYNTHESIS agrees with, diverges from, or misses decisions made in the human benchmark."

    The benchmark [12] is co-authored by two authors of the present paper (Maier and Feuerriegel). It is the only quantitative benchmark for the headline claim. Moreover, the overlapping author/judgment is used to re-adjudicate eligibility decisions ('a qualitative review by the lead author of the benchmark meta-analysis indicated that this study was not directly relevant'), producing corrected recall/precision of 85.7%/87.5%. The 'ground truth' is thus not independent of the system's designers: the same team built the system, wrote the prompts, and judged borderline cases in the benchmark.

full rationale

The central validation is the comparison of AutoSynthesis's pooled estimate (g=0.143) with the published benchmark [12] (g=0.020). Two load-bearing features weaken the independence of that comparison. First, the relevance-filter prompt includes a hand-designed priority rank (behavior=1, compliance/persuasion=2, attitude=3, intention=4, ...) that is not derived from first principles and mirrors the benchmark's outcome-selection taxonomy. Since the pooled estimate is computed only after this filter selects which extracted rows survive, the agreement is partly by construction: the system is guided toward the benchmark's choices rather than rediscovering them from the literature. Second, the benchmark itself is not independent of the authors: [12] shares two co-authors with the present paper, and the paper explicitly leverages the overlapping review-team membership for qualitative adjudication that improves the reported overlap metrics. The system's effect-size arithmetic and REML pooling are deterministic and not circular, and the paper appropriately discloses training-data contamination and small benchmark size as limitations; but those do not cure the selection-gate and benchmark-independence issues. Separately, the abstract's claim of 'within ±0.12 Hedges' g' is not met by the reported Δg=0.123; the paper then switches to a ±0.20 tolerance from [39]. This is a reporting inconsistency rather than a circularity, but it further undermines the headline. Overall, the result is a partial circularity: the 'prediction' is partly shaped by benchmark-specific priors, so the score is 6 rather than 8 or 10.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim depends on a small set of hand-chosen design parameters (priority ranking, retrieval cap, tolerance) and on domain assumptions about LLM verification fidelity, benchmark independence, and conversion validity. No new physical or conceptual entities are introduced. The most serious load-bearing assumption is the independence of the benchmark, since it shares authors with the system and the prompts appear tailored to it.

free parameters (4)
  • Outcome priority ranking = behavior=1, compliance/persuasion=2, attitude=3, intention=4, resistance=5, experiential=6, mechanism=7, descriptive=8
    Hand-designed in the relevance-filter prompt (Supplementary Materials); determines which extracted effect size becomes the winning result per study, directly shaping the pooled estimate.
  • Agreement tolerance = ±0.20 Hedges' g
    Applied post hoc in §2.3 and Discussion to declare g=0.020 vs g=0.143 as 'broadly similar'; sourced from prior reanalysis work [39].
  • Search retrieval cap = 40 papers per database
    Set in §4.1 and §4.3 to limit cost; changes which candidate studies enter the evidence base and therefore the final effect estimate.
  • LLM reasoning effort = medium (gpt-5.4-mini)
    Configuration choice in §4.3; no ablation is provided, so the effect of this choice on extraction quality is unknown.
axioms (5)
  • domain assumption LLM self-verification ('verbatim or within rounding') is sufficient to prevent hallucinated statistics from entering the meta-analysis
    The validation agent is the same class of LLM that performed extraction and is instructed to accept values that appear verbatim or within rounding (§4.1); shared failure modes are not tested.
  • ad hoc to paper The Hölbling et al. (2025) benchmark is an appropriate, independent ground truth for the evaluation
    §4.2 explicitly states that one author of the present study overlaps with the original lead of the review team; the benchmark is the only quantitative reference used.
  • standard math The custom REML, Egger's test, and trim-and-fill implementations are correct
    §4.1 states these were implemented directly in Python following Viechtbauer [50] and Duval/Tweedie [29], but no test suite or validation against established packages is reported.
  • domain assumption Converting diverse reported statistics (t-tests, F-values, odds ratios, regression coefficients) to Hedges' g is valid for the included studies
    The analysis agent converts 'whatever statistics are available' (Fig. 1; §4.1); small samples and non-normal outcomes can bias such conversions, and no per-study sanity checks are reported.
  • domain assumption Retrieval from five public API sources (arXiv, Semantic Scholar, CrossRef, OSF Preprints, PubMed) is representative of the relevant literature
    §4.1 notes proprietary databases such as Web of Science and Scopus are absent, and one benchmark study was missed because its full text was inaccessible; the evidence base is therefore incomplete by construction.

pith-pipeline@v1.3.0-alltime-deepseek · 21178 in / 12185 out tokens · 101468 ms · 2026-08-01T23:43:46.353136+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of AutoSynthesis: An agentic system for automated meta-analysis." pith.science (2026). https://pith.science/paper/OYXJ5XTN

@misc{pith2026260715247,
  author       = {Pith},
  title        = {Pith review of: AutoSynthesis: An agentic system for automated meta-analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OYXJ5XTN}},
  note         = {Machine review of arXiv:2607.15247}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Evidence synthesis is crucial for turning primary research into reliable knowledge for science, medicine, education, and policy. Yet, quantitative evidence synthesis remains largely manual and difficult to scale. Here, we introduce AutoSynthesis, an end-to-end multi-agent system for automated meta-analysis. Given a research question in natural language, AutoSynthesis formulates a search strategy, retrieves scientific literature, screens candidate studies, assesses full-text eligibility, extracts quantitative statistics, computes standardized effect sizes, and finally performs random-effects meta-analysis. AutoSynthesis further supports heterogeneity analysis to examine how effect sizes vary across moderators, as well as risk-of-bias assessment. As output, AutoSynthesis produces a transparent report aligned with PRISMA guidelines. In our application, AutoSynthesis screened over 28 studies and extracted more than 20 quantitative claims. The pooled effect estimates produced by AutoSynthesis are similar to Hedges' $g$ of expert-conducted meta-analyses, indicating close agreement with manual evidence synthesis. Together, these results show that AutoSynthesis can make quantitative evidence synthesis more scalable, thereby supporting evidence-based decision-making across disciplines.

Figures

Figures reproduced from arXiv: 2607.15247 by Francesco Pierri, Gerardo Vitagliano, Moein Taherinezhad, Sebastian Maier, Stefan Feuerriegel.

Figure 1
Figure 1. Figure 1: Overview of the AUTOSYNTHESIS framework. A multi-agent framework includ￾ing: a, The workflow begins by transforming the user’s research question into a structured review protocol and retrieving candidate studies from multiple scientific databases using LLM-generated search queries. b, Retrieved articles undergo full-text retrieval, eligibility assessment, and study mapping, where independent studies and re… view at source ↗
Figure 2
Figure 2. Figure 2: Demonstration of automated evidence synthesis using AUTOSYNTHESIS. To demonstrate the capa￾bilities of AUTOSYNTHESIS, we tasked the framework with synthesizing evidence on the persuasive power of large language models, for which evidence is contested following the research question of the recent meta-analysis by Hölbling et al. (2025) [12]. a, The input to AUTOSYNTHESIS is a user-provided research question… view at source ↗
Figure 3
Figure 3. Figure 3: Evaluation of the literature retrieval and study selection. a, The funnel summarizing study selection process by reporting the number of studies retrieved, screened, eligible, and finally included. b, Agreement between the studies included by AUTOSYNTHESIS and those included in the benchmark meta-analysis. 2.3 AUTOSYNTHESIS recovers accurate meta-analytic effect size estimates We next evaluated whether AUT… view at source ↗
Figure 4
Figure 4. Figure 4: Agreement between the meta-analytics estimates produced by AUTOSYNTHESIS and the manual benchmark. a, Comparison of study-level standardized effect sizes (Hedges’ g) extracted by AUTOSYNTHESIS and those reported in the published human benchmark (gray points). The diagonal line indicates perfect agreement. Reported is the Pearson’s correlation co￾efficient (r) and the slope of the ordinary least squares (OL… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 8 linked inside Pith

  1. [1]

    & Stewart, G

    Gurevitch, J., Koricheva, J., Nakagawa, S. & Stewart, G. Meta-analysis and the science of research synthesis.Nature555, 175–182 (2018)

    2018

  2. [2]

    J.et al.The prisma 2020 statement: An updated guideline for reporting systematic reviews.BMJ372(2021)

    Page, M. J.et al.The prisma 2020 statement: An updated guideline for reporting systematic reviews.BMJ372(2021). https://www.bmj.com/content/372/bmj.n71.full.pdf

    2020

  3. [3]

    Higgins, J. P. T., Thomas, J., Chandler, J.et al. Cochrane Handbook for Systematic Reviews of Interventions(Cochrane, 2024), version 6.5 edn

    2024

  4. [4]

    Collaboration, O. S. Estimating the reproducibility of psychological science.Science349, aac4716 (2015). https://www.science.org/doi/pdf/10.1126/science.aac4716

    doi:10.1126/science.aac4716 2015

  5. [5]

    K., Davidson, I

    Flake, J. K., Davidson, I. J., Wong, O. & Pek, J. Construct validity and the validity of replication studies: A systematic review.American Psychologist77, 576–588 (2022)

    2022

  6. [6]

    Chen, L., Yan, R. & Hu, Y . City walk or nature walk? Evidence-based psychological and physiological outcomes – a systematic review and meta-analysis.Urban Forestry & Urban Greening106, 128726 (2025)

    2025

  7. [7]

    & Feuerriegel, S

    Holzner, N., Maier, S. & Feuerriegel, S. Generative AI and creativity: A systematic literature review and meta-analysis (2025). URL https://arxiv.org/abs/2505.17241

    Pith/arXiv arXiv 2025

  8. [8]

    & Kamari, A

    Arifin, Z., Sukarmin, S., Saputro, S. & Kamari, A. The effect of inquiry-based learning on students’ critical thinking skills in science education: A systematic review and meta-analysis. Eurasia Journal of Mathematics, Science and Technology Education21, em2592 (2025)

    2025

  9. [9]

    & Huang, R

    Tlili, A., Saqer, K., Salha, S. & Huang, R. Investigating the effect of artificial intelligence in education (AIEd) on learning achievement: A meta-analysis and research synthesis.Infor- mation Development41, 825–842 (2025). 28

    2025

  10. [10]

    Bayat, K., Pooyan, P., Chhablani, J., Kanavi, M. R. & Ahmadieh, H. Retinal and choroidal alterations in heart failure: A systematic review and meta-analysis of oct and oct-a findings with emphasis on HFrEF.Ophthalmology and Therapy14, 2631–2651 (2025)

    2025

  11. [11]

    L.et al.The risk of long covid symptoms: a systematic review and meta- analysis of controlled studies.Nature Communications16(2025)

    O’Mahoney, L. L.et al.The risk of long covid symptoms: a systematic review and meta- analysis of controlled studies.Nature Communications16(2025)

    2025

  12. [12]

    & Feuerriegel, S

    Hölbling, L., Maier, S. & Feuerriegel, S. A meta-analysis of the persuasive power of large language models.Scientific Reports15(2025)

    2025

  13. [13]

    W., Capers, P

    Borah, R., Brown, A. W., Capers, P. L. & Kaiser, K. A. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the prospero registry.BMJ Open7, e012545 (2017)

    2017

  14. [14]

    & Chalmers, I

    Bastian, H., Glasziou, P. & Chalmers, I. Seventy-five trials and eleven systematic reviews a day: How will we ever keep up?PLoS Medicine7, e1000326 (2010)

    2010

  15. [15]

    & Zschech, P

    Feuerriegel, S., Hartmann, J., Janiesch, C. & Zschech, P. Generative AI.Business & Infor- mation Systems Engineering66, 111–126 (2023)

    2023

  16. [16]

    Gottweis, J.et al.Accelerating scientific discovery with co-scientist.Nature(2026)

    2026

  17. [17]

    URL https://arxiv.org/abs/2606.13670

    Holtdirk, T.et al.Automated reproducibility assessments in the social and behavioral sci- ences using large language models (2026). URL https://arxiv.org/abs/2606.13670. 2606. 13670

    Pith/arXiv arXiv 2026

  18. [18]

    URL https://arxiv.org/abs/2504.08066

    Yamada, Y .et al.The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search (2025). URL https://arxiv.org/abs/2504.08066. 2504.08066

    Pith/arXiv arXiv 2025

  19. [19]

    & Kumar, G

    Syriani, E., David, I. & Kumar, G. Screening articles for systematic reviews with chatgpt. Journal of Computer Languages80, 101287 (2024). 29

    2024

  20. [20]

    Guo, E.et al.Automated paper screening for clinical reviews using large language models: Data analysis study.Journal of Medical Internet Research26, e48996 (2024)

    2024

  21. [21]

    & Moreno-Garcia, C

    Ofori-Boateng, R., Aceves-Martins, M., Wiratunga, N. & Moreno-Garcia, C. F. Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review.Artificial Intelligence Review57(2024)

    2024

  22. [22]

    Forster, K.et al.Assessing corporate sustainability with large language models: evidence from europe.Nature Communications17(2026)

    2026

  23. [23]

    I.et al.Empowering meta-analysis: Leveraging large language models for scientific synthesis (2024)

    Ahad, J. I.et al.Empowering meta-analysis: Leveraging large language models for scientific synthesis (2024). URL https://arxiv.org/abs/2411.10878. 2411.10878

    Pith/arXiv arXiv 2024

  24. [24]

    good enough

    Li, L., Mathrani, A. & Susnjak, T. What level of automation is “good enough”? a benchmark of large language models for meta-analysis data extraction.Research Synthesis Methods17, 671–692 (2026)

    2026

  25. [25]

    & Shariatnia, M

    Rouzrokh, P., Khosravi, B., Rouzrokh, P. & Shariatnia, M. Lattereview: A multi-agent framework for systematic review automation using large language models (2025). URL https://arxiv.org/abs/2501.05468

    arXiv 2025

  26. [26]

    URL https://arxiv.org/abs/2505.20310

    Xu, W.et al.Manalyzer: End-to-end automated meta-analysis with multi-agent system (2025). URL https://arxiv.org/abs/2505.20310

    arXiv 2025

  27. [27]

    URL https://arxiv.org/abs/2603.22327

    Padarha, S.et al.Evaluating AI-based scientific knowledge synthesis with epidemiological systematic reviews (2026). URL https://arxiv.org/abs/2603.22327

    Pith/arXiv arXiv 2026

  28. [28]

    F.et al.LLM-assisted systematic review of large language models in clinical medicine.Nature Medicine32, 1152–1159 (2026)

    Chen, S. F.et al.LLM-assisted systematic review of large language models in clinical medicine.Nature Medicine32, 1152–1159 (2026)

    2026

  29. [29]

    & Tweedie, R

    Duval, S. & Tweedie, R. Trim and Fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis.Biometrics56, 455–463 (2000). 30

    2000

  30. [30]

    D., Schneider, M

    Egger, M., Smith, G. D., Schneider, M. & Minder, C. Bias in meta-analysis detected by a simple, graphical test.BMJ315, 629–634 (1997)

    1997

  31. [31]

    Sterne, J. A. C.et al.RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJl4898 (2019)

    2019

  32. [32]

    A.et al.ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions.BMJi4919 (2016)

    Sterne, J. A.et al.ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions.BMJi4919 (2016)

    2016

  33. [33]

    Iversen, I. A. & Quintana, D. S. The value of living systematic reviews.Nature Human Behaviour10, 647–649 (2026)

    2026

  34. [34]

    Tyler, C.et al.AI tools as science policy advisers? the potential and the pitfalls.Nature622, 27–30 (2023)

    2023

  35. [35]

    & West, R

    Salvi, F., Horta Ribeiro, M., Gallotti, R. & West, R. On the conversational persuasiveness of gpt-4.Nature Human Behaviour9, 1645–1653 (2025)

    2025

  36. [36]

    G., Muldowney, S., Eichstaedt, J

    Bai, H., V oelkel, J. G., Muldowney, S., Eichstaedt, J. C. & Willer, R. Llm-generated messages can persuade humans on policy issues.Nature Communications16(2025)

    2025

  37. [37]

    C.et al.The potential of generative ai for personalized persuasion at scale.Scientific Reports14(2024)

    Matz, S. C.et al.The potential of generative ai for personalized persuasion at scale.Scientific Reports14(2024)

    2024

  38. [38]

    & Margetts, H

    Hackenburg, K. & Margetts, H. Evaluating the persuasive influence of political microtar- geting with large language models.Proceedings of the National Academy of Sciences121 (2024)

    2024

  39. [39]

    Nature652, 135–142 (2026)

    Aczel, B.et al.Investigating the analytical robustness of the social and behavioural sciences. Nature652, 135–142 (2026). 31

    2026

  40. [40]

    R., von Krogh, G

    Shrestha, Y . R., von Krogh, G. & Feuerriegel, S. Building open-source ai.Nature Computa- tional Science3, 908–911 (2023)

    2023

  41. [41]

    Brodeur, A.et al.Ai-assisted teams outperform ai-led teams but not human-only teams in assessing research reproducibility in quantitative social science.Proceedings of the National Academy of Sciences123(2026)

    2026

  42. [42]

    Miao, J., Pritchard, J. K. & Zou, J. The agentic garden of forking paths (2026). URL https://arxiv.org/abs/2607.01507

    Pith/arXiv arXiv 2026

  43. [43]

    Bertran, M., Fogliato, R. & Wu, Z. S. Many ai analysts, one dataset: Navigating the agentic data science multiverse.Proceedings of the National Academy of Sciences123(2026)

    2026

  44. [44]

    Langchain

    Chase, H. Langchain. https://github.com/langchain-ai/langchain (2022)

    2022

  45. [45]

    URL https://arxiv.org/abs/2210.03629

    Yao, S.et al.React: Synergizing reasoning and acting in language models (2022). URL https://arxiv.org/abs/2210.03629

    Pith/arXiv arXiv 2022

  46. [46]

    URL https://arxiv.org/abs/2409.18839

    Wang, B.et al.Mineru: An open-source solution for precise document content extraction (2024). URL https://arxiv.org/abs/2409.18839

    Pith/arXiv arXiv 2024

  47. [47]

    LlamaIndex (2022)

    Liu, J. LlamaIndex (2022). URL https://github.com/jerryjliu/llama_index

    2022

  48. [48]

    Introducing mistral ocr 3

    Mistral AI. Introducing mistral ocr 3. https://mistral.ai/news/mistral-ocr-3/ (2025). Accessed: 2026-07-12

    2025

  49. [49]

    Hedges, L. V . Distribution theory for glass’s estimator of effect size and related estimators. Journal of Educational Statistics6, 107–128 (1981)

    1981

  50. [50]

    Bias and efficiency of meta-analytic variance estimators in the random- effects model.Journal of Educational and Behavioral Statistics30, 261–293 (2005)

    Viechtbauer, W. Bias and efficiency of meta-analytic variance estimators in the random- effects model.Journal of Educational and Behavioral Statistics30, 261–293 (2005). 32

    2005

  51. [51]

    J., Higgins, J

    Deeks, J. J., Higgins, J. P., Altman, D. G. & on behalf of the Cochrane Statistical Meth- ods Group. Analysing data and undertaking meta-analyses (2019). URL http://dx.doi.org/10. 1002/9781119536604.ch10

    2019

  52. [52]

    V ., Higgins, J

    Borenstein, M., Hedges, L. V ., Higgins, J. P. T. & Rothstein, H. R.Introduction to Meta- Analysis(Wiley, 2009)

    2009

  53. [53]

    & Baker, R

    Jackson, D., Bowden, J. & Baker, R. How does the dersimonian and laird procedure for random effects meta-analysis compare with its more efficient but harder to compute counter- parts?Journal of Statistical Planning and Inference140, 961–970 (2010)

    2010

  54. [54]

    Openrouter

    OpenRouter.ai. Openrouter. https://openrouter.ai (2026). Accessed: 2026-07-16

    2026

  55. [55]

    Nature Human Behaviour(2026)

    Feuerriegel, S.et al.A reporting checklist for large language models in behavioural science. Nature Human Behaviour(2026)

    2026

  56. [56]

    queries": [

    Anthropic. Claude for science: An AI workbench for scientific research. https://www. anthropic.com/news/claude-science-ai-workbench (2025). Accessed: 2026-07-13. 33 Acknowledgments Funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under the National Research Data Infrastructure – NFDI 27/1-2026, project number 460037581 is a...

    2025

  57. [57]

    Preregistered outcomes

  58. [58]

    Main experimental contrasts

  59. [59]

    -------------------------------------- OUTPUT PHILOSOPHY -------------------------------------- You are building a CLEAN SYNTHESIS DATASET, not a complete theory map

    Central claims in abstract/results/discussion If uncertain whether an effect is central: prefer inclusion -- it is better to over-extract than to miss a relevant 42 effect. -------------------------------------- OUTPUT PHILOSOPHY -------------------------------------- You are building a CLEAN SYNTHESIS DATASET, not a complete theory map. Fewer highly rele...

  60. [60]

    Only accept a number as verified if it appears VERBATIM (or within rounding) in the paper text -- in the Results section, tables, or figure captions

  61. [61]

    Do NOT accept numbers from Introduction, Related Work, or Discussion sections that describe OTHER studies

  62. [62]

    If a value is wrong but the correct value is in the text, issue a ’corrected’ verdict and provide the corrected value

  63. [63]

    If a value cannot be located anywhere in the text, mark it in failed_stats

  64. [64]

    Mark the result ’hallucinated’ only if the CORE numbers (test statistic, p-value, or means) are absent from the paper text entirely

  65. [65]

    Call ‘record_validation‘ once per extracted result row

  66. [66]

    Relevance filter agent Relevance filter agent: System prompt You are a meta-analysis outcome classifier

    Be precise: 3.77 and 3.78 are NOT the same. Relevance filter agent Relevance filter agent: System prompt You are a meta-analysis outcome classifier. Your ONLY job is to classify each row -- you do NOT decide which rows to keep. A separate algorithm will select winners based on your classifications. For EVERY row call classify_and_select with accurate values for:

  67. [67]

    OUTCOME CLASS primary | secondary | manipulation_check | process_variable | exploratory | robustness_check | descriptive

  68. [68]

    CRITICAL -- PARENTHETICAL QUALIFIER RULE: Outcome names that share the same base name but differ only by a parenthetical qualifier are sub-conditions of the SAME construct

    CONSTRUCT FAMILY A normalised label for the underlying construct being measured. CRITICAL -- PARENTHETICAL QUALIFIER RULE: Outcome names that share the same base name but differ only by a parenthetical qualifier are sub-conditions of the SAME construct. Strip the qualifier. Examples: ’Compliance rate (overall)’, ’Compliance rate (truthful)’, 44 ’Complianc...

  69. [69]

    GRANULARITY LEVEL Within the construct_family group, is this row: - ’aggregate’ -- covers all conditions/full sample (keyword: ’overall’, ’total’, ’combined’, or no qualifier) - ’sub_condition’ -- splits by condition, content type, time point, sub-scale (keywords: ’truthful’, ’deceptive’, ’immediate’, ’delayed’, etc.) - ’standalone’ -- neither (single uni...

  70. [70]

    COMPARISON ARM From group_1_label / group_2_label, classify the comparison arm: - ’target_arm’ -- treatment is the LLM / AI intervention of interest - ’non_target_arm’ -- treatment is a human or other non-AI comparator - ’primary’ -- only one arm exists

  71. [71]

    False for manipulation checks, process variables unrelated to the main outcome, etc

    IS SYNTHESIS RELEVANT True if the construct directly answers the synthesis objective. False for manipulation checks, process variables unrelated to the main outcome, etc

  72. [72]

    Do not skip any

    PRIORITY RANK (1 = highest) 1=behavior, 2=compliance/persuasion, 3=attitude, 4=intention, 5=resistance, 6=experiential, 7=mechanism, 8=descriptive Call classify_and_select once for EVERY row. Do not skip any. Set is_winner=True for all rows -- the selection algorithm overrides this. Analysis agent Analysis agent: Interpretation prompt (templated per resul...

    2024