Pith. sign in

REVIEW 3 major objections 4 minor 26 references

EviDAG stages literature-grounded LLM judgments into auditable causal DAGs, with every citation a verified abstract span.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 06:25 UTC pith:JG777HZ6

load-bearing objection Solid systems paper: EviDAG's real contribution is the auditable evidence trail and frozen PubMed snapshot, but the headline recall/F1 numbers rest on a scoring rule that drops the exposure–outcome pair, and there's no code/data release. the 3 major comments →

arxiv 2607.21859 v2 pith:JG777HZ6 submitted 2026-07-23 cs.AI

EviDAG: Auditable Causal DAG Authoring with Biomedical Literature

classification cs.AI
keywords causal DAGbiomedical literaturelarge language modelsevidence groundingverifiable citationsauditable workflowscausal reasoningDAG authoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that authoring a causal DAG from biomedical literature can be automated without giving up auditability. The proposed system, EviDAG, resolves free-text study variables, freezes a reproducible corpus of retrieved abstracts, asks an LLM to make structured pairwise causal judgments over that corpus, and assembles the judgments into a constraint-checked graph in which every edge carries confidence, provenance, and a verbatim evidence excerpt. The pivotal quantitative claim is that on DAGs derived from published biomedical studies the workflow reaches mean directed-graph recall of 0.866 in a cold run and 0.905 with study context (directed F1 0.664 and 0.714), while every EviDAG citation is an exact abstract span; the same model asked to produce the graph directly produced zero verifiable citations. If this holds, the bottleneck of DAG construction shifts from manual literature review to expert review of machine-suggested, evidence-linked edges.

Core claim

The central claim, stated on the paper's own terms, is that a staged workflow—concept resolution with ontology enrichment, a frozen literature snapshot, evidence-grounded pairwise causal reasoning, and deterministic graph assembly under acyclicity and user constraints—can reconstruct causal DAGs from free-text study concepts at high edge recall on a literature-derived cohort while retaining a verifiable evidence trail for every proposed edge. The evaluation reports mean graph directed F1 of 0.664 (cold) and 0.714 (with study context) on three reference DAGs drawn from published studies, with recall of 0.866 and 0.905 respectively. It also reports that 86–87% of literature-cohort false-positi

What carries the argument

The load-bearing mechanism is the provenance-linked edge record: each proposed edge bundles a direction, sign, confidence, rationale, provenance channel (retrieved literature, ontology, temporal ordering, user constraint, solver decision, or model prior), and a verbatim excerpt copied from a retrieved abstract. The pipeline gains leverage by restricting LLM influence to per-pair structured judgments over a fixed, run-specific corpus and to a bounded graph-level disambiguation step, while deterministic components enforce acyclicity, temporal order, and user constraints. This separation is what makes the output auditably replayable in principle.

Load-bearing premise

The headline recall figures assume that edges missing from the deliberately parsimonious published reference DAGs are literature-supported relations the references chose not to model, and that dropping the exposure–outcome estimand pair from both scored edge sets is a neutral scoring choice; if either assumption fails, the recall-and-auditability claim overstates what the evaluation supports.

What would settle it

Inspect every surplus edge in the literature-cohort runs: independently verify that the cited abstract actually reports a causal relation for that pair; if most do not, the 'outside scope' interpretation collapses and precision is genuinely poor. Likewise, re-run scoring with the exposure–outcome pair included in both system and reference edge sets: if the gap to the direct LLM baseline widens or recall drops sharply, the reported recall is an artifact of the exclusion rule.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Biomedical analysts can draft a literature-based DAG from free-text study concepts in one pass and review each proposed edge's evidence card instead of starting from blank paper.
  • Every accepted or rejected edge can be defended to collaborators or reviewers with a citation that is verifiable by construction, because it is an exact span of a retrieved abstract.
  • The system's recall orientation makes it a useful generator of candidate edges and candidate unmeasured common causes, with precision left for expert pruning.
  • On-demand adjustment-set computation means downstream confounder adjustment inherits the same audit trail as graph construction.
  • The direct-LLM comparison shows that fluent graph output without an evidence trail is not enough for scientific use; the paper's workflow demonstrates a concrete alternative.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the verbatim-citation property suggests a general design rule for retrieval-grounded LLM workflows—restrict generated claims to exact spans from a frozen corpus—so that audit failures become automatically detectable.
  • One testable extension the paper leaves implicit: using the per-edge citation-backed status as a confidence feature, then applying a provenance-preserving pruning pass over the assembled graph, could recover the precision lost to the parsimonious-reference comparison without sacrificing verifiability.
  • The quadratic per-pair retrieval cost implies the approach will need candidate-pair pruning for graphs beyond roughly a dozen variables; the citation-verification rate could serve as the signal for which pairs deserve full retrieval.
  • Because the exclusion of the exposure–outcome estimand pair and of named-latent exclusions are scoring choices, a reasonable next validation would re-score with those edges included, to show the headline recall is not an artifact of the rule.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents EviDAG, a browser-based system for authoring causal DAGs from biomedical literature. It describes a staged pipeline: UMLS-assisted concept resolution, construction of a frozen PubMed evidence snapshot, LLM-based pairwise causal judgments with verbatim evidence extraction, deterministic constraint-checked graph assembly, and an interactive review and export interface. The evaluation uses seven synthetic compact DAGs and three published literature DAGs (A, B, C). The headline literature-cohort results are mean graph directed F1 of 0.664 (cold) and 0.714 (with study context), with recall 0.866/0.905 and precision 0.577/0.625. The paper emphasizes that all EviDAG citations are verified verbatim abstract spans, whereas the direct and pairwise LLM baselines produce no verifiable citations. The authors frame the contribution as auditable, evidence-preserving DAG curation rather than causal discovery.

Significance. If the claims hold, EviDAG is a valuable systems contribution. The traceability and reproducibility architecture — frozen PubMed snapshot, PRISMA-style search provenance, per-edge evidence cards, deterministic acyclicity and constraint enforcement, and a headless evaluation mode — is a genuine step beyond prompting an LLM to emit a DAG. The citation-verification result is striking: even if the baseline comparison is imperfect, the construction of EviDAG guarantees that citations come from retrieved abstracts, and the reported 0-of-211 and 0-of-812 counts for baselines make the contrast concrete. The paper is also honest about limitations (PubMed abstracts only, fallible LLM judgments, quadratic cost). The main weakness is the quantitative structural evaluation: the most consequential edge in a study DAG — the exposure–outcome pair — is excluded from the literature-cohort metrics, and the favorable interpretation of low precision rests on an untested assumption about reference-DAG scope. These issues are fixable with additional analyses and more careful claim calibration, but they currently make the abstract's 'high edge recall' claim stronger than the evaluation supports.

major comments (3)
  1. [Section 4 (Table 1); Appendix A, 'Scoring-rule rationale'] The exposure–outcome estimand pair is excluded, in both orientations, from both the system and reference edge sets for all three literature DAGs. For a study-design tool this is the most consequential edge type. Dropping the pair from both sets removes the possibility of scoring a true positive or a false negative on the primary relation, and the net effect on the reported recall (0.866/0.905) is unknown without per-pair results. Please report a sensitivity analysis with the pair included, and/or per-pair outcomes for the exposure–outcome edge for A, B, and C. At minimum, qualify the abstract's 'high edge recall' as applying to the surrounding structure only.
  2. [Section 4 ('Error-analysis edges'); Appendix A] The interpretation that 86% (cold) and 87% (with context) of literature-cohort false positives are 'reported relations outside the reference DAG's chosen scope' is a post-hoc hypothesis, not a demonstrated finding. A verified verbatim citation shows that a relation is mentioned in some retrieved abstract, but not that it is causal, direct rather than mediated, or within the target study's causal structure. This interpretation is used to contextualize the low precision (0.577/0.625) in Table 1 and the comparison in Table 4. Supporting it with per-edge expert adjudication, or explicitly labeling it as a hypothesis, is necessary before the precision gap can be set aside.
  3. [Section 4 (Table 4); Appendix A, 'LLM-only baselines'] The traceability comparison to the LLM-only baselines may not be apples-to-apples. The baselines were asked to produce a DAG directly with no retrieval, and the text does not state whether they were prompted to provide citations or quoted spans at all. The 0-of-211 and 0-of-812 verbatim-citation counts then reflect the prompted output contract rather than a direct comparison of citation behavior. To support the claim that such evidence trails are 'absent from LLM-only baselines,' specify the baseline prompts exactly, or add a condition in which baselines are asked to provide citations and quotes so the output contract matches EviDAG.
minor comments (4)
  1. [Section 4, Table 2] The literature cohort contains only three DAGs, with wide variation (e.g., A cold precision 0.431 vs. B 0.812). The per-DAG medians are reported without any measure of spread across the three temperature-0 runs. Consider adding per-run values or variability estimates, and temper the language from 'cohort' to 'case studies' or expand the benchmark.
  2. [Appendix A, 'Inclusion-policy comparison'] Table 3 shows that literature-cohort directed F1 drops from 0.664 (all-tiers) to 0.486 (high-only) in the cold condition. Because the headline relies on including low-confidence edges, the paper should discuss how users are expected to use the confidence tiers in practice and why all-tiers is the appropriate headline rather than a boundary case.
  3. [Section 4, 'Standard evaluation configuration'] The model is pinned as 'claude-sonnet-4-6' at temperature 0, but no model version date or API snapshot is given. For reproducibility, specify the exact model version and, if possible, release the frozen PubMed snapshots and evaluation harness.
  4. [Appendix A, 'Scoring-rule rationale'] The exclusion of non-specific latent placeholders (e.g., U*) is reasonable but also removes edges from both system and reference sets. It would be helpful to state how many edges are affected per reference DAG, since this modifies the metric denominators.

Circularity Check

0 steps flagged

No significant circularity: graph recall/F1 are measured against external frozen-PubMed reference DAGs with no fitted parameters; the only by-construction element (verbatim citations) is acknowledged and does not drive the graph metrics.

full rationale

EviDAG's central evaluation is not circular. No parameter is fitted to the reference DAGs: the benchmark uses a fixed standard configuration (Anthropic claude-sonnet-4-6 at temperature 0, UMLS enrichment, per-DAG PubMed cutoff set to the day before each source study's online publication date), and the assembled graph is scored against externally published reference DAGs. The headline recall/F1 therefore measure an external reconstruction task rather than a quantity encoded in the inputs. No load-bearing self-citation exists: the references are to external methods and tools (DAGitty, DoWhy, UMLS, PubMed, etc.), and no 'uniqueness theorem' or prior own-work ansatz is invoked to force the design. The only explicitly by-construction property is the verbatim-citation guarantee (Table 4: 'by construction yes'; 'every EviDAG citation is an extracted-and-verified verbatim span'). The paper itself labels this a design constraint, not a fitted prediction, and it does not enter the graph directed-F1/recall computation, so it does not make the central recall claim circular. Appendix A's exclusion of the exposure–outcome estimand pair from study-derived literature DAG scoring is a measurement-scope decision with unknown direction of effect on recall; it is a validity/interpretation limitation, not a definitional equivalence or a fitted-parameter renaming. Accordingly, no circular step is exhibited.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim depends mostly on modeling and evaluation choices rather than on invented entities. The main free parameters are scoring exclusions (estimand pair, latent placeholders), retrieval controls (PubMed cutoff, per-pair cap), the lenient mediation policy, and unreported confidence thresholds. The axioms are the validity of parsimonious published DAGs as references, abstract-only evidence sufficiency, LLM judgment reliability, the scope-vs-error interpretation of false positives, and the default no-unmeasured-confounding assumption. No new particles, forces, dimensions, or physical entities are introduced.

free parameters (6)
  • Estimand-pair exclusion (study DAGs) = excluded from metrics
    The exposure–outcome pair is removed from both system and reference edge sets for literature DAGs (Appendix A). Reported recall 0.87/0.91 excludes the one edge each study was designed to estimate; keeping it could change headline figures.
  • Non-specific latent placeholder exclusion = U* excluded
    Edges incident to U* are dropped because EviDAG reconstructs latent confounders positionally; removes a class of unavoidable mismatches from scores.
  • PubMed cutoff date = day before source study online publication
    Set per literature DAG to prevent retrieval of the reference paper itself; choice determines whether the reference's own framing is available as evidence.
  • Per-pair retrieval cap = not specified
    Bounded number of records per concept pair with truncation under PubMed Best Match; the cap determines which abstracts the LLM sees and is not quantified in the paper.
  • Mediation policy = lenient
    Assembly retains a direct edge unless evidence fully accounts for it via indirect paths; the policy affects how many edges survive assembly and was chosen for the headline runs.
  • Confidence-tier thresholds = not reported
    high/mid/low tiers define inclusion policies (high-only, high+mid, all-tiers); without thresholds only the all-tiers policy is fully specified.
axioms (5)
  • domain assumption Published reference DAGs are adequate evaluation targets for literature-based DAG reconstruction.
    The paper uses three study DAGs as references while admitting they 'do not necessarily reflect causal ground truth' and are 'often deliberately parsimonious'; scoring system output against them equates reference reconstruction with usefulness.
  • domain assumption PubMed abstracts suffice as an evidence base for pairwise causal judgments.
    All evidence is abstract-level; the system never reads full text, so causal claims that only appear in full-text articles are invisible (Limitations, Section 6).
  • domain assumption The LLM's structured judgments are reliable enough that deterministic constraints can bound their errors.
    Pipeline correctness depends on Claude Sonnet 4.6 producing valid structured judgments; the paper acknowledges model fallibility but still relies on it as the core reasoning module.
  • ad hoc to paper Edges absent from the parsimonious reference DAGs are out-of-scope relations rather than errors.
    Used in Section 4 error analysis to reinterpret 86–87% of false positives as 'reported relations outside the reference DAG's chosen scope'; the paper does not test whether the reference authors considered and rejected those edges.
  • domain assumption Unmeasured confounding is absent unless users explicitly admit latent common causes.
    Identification shell and adjustment sets assume candidate latents are unconfounded by default (Section 2, footnote 2), which is a standard but nontrivial causal sufficiency assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 10330 in / 13202 out tokens · 112866 ms · 2026-08-01T06:25:25.818664+00:00 · methodology

0 comments
read the original abstract

Constructing causal directed acyclic graphs (DAGs) is a core step in biomedical causal analysis, yet it remains a largely manual process. Analysts must connect study variables to prior literature, evaluate uncertain causal claims, and preserve sufficient provenance for expert review. We present EviDAG, a browser-based system for authoring causal DAGs as auditable, evidence-linked artifacts from biomedical literature. Given free-text descriptions of study concepts, EviDAG creates a reproducible literature snapshot, uses an LLM-based reasoning module to generate structured pairwise causal judgments, links literature-supported judgments to verbatim evidence excerpts, and assembles the judgments into a constraint-checked graph. Each proposed edge includes confidence estimates, provenance, and a reviewable rationale. The interface supports study specification, progress monitoring, evidence review, graph comparison, adjustment-set computation, and export. In evaluations against both compact benchmark DAGs and reference DAGs derived from published literature, EviDAG achieves high edge recall on the literature-based cohort while retaining verifiable evidence trails absent from LLM-only baselines. EviDAG thus reduces the burden of causal DAG curation while making the resulting assumptions auditable, supporting the design, analysis, and interpretation of biomedical studies.

Figures

Figures reproduced from arXiv: 2607.21859 by Bo Wang, Fan-Yu Yen, Jordan W. Smoller, Michael R. Steigman, Yi-han Sheu, Yu Zhou.

Figure 1
Figure 1. Figure 1: EviDAG system flow. LLM calls occur inside bounded backend stages; the interface renders persisted artifacts rather than recomputing causal state. porting Items for Systematic Reviews and Meta￾Analyses (PRISMA) (Page et al., 2021); down￾stream modules consume it read-only. Pairwise causal reasoning. For each concept pair, EviDAG first ranks evidence by study-design tier, using recency to break ties. It ext… view at source ↗
Figure 2
Figure 2. Figure 2: EviDAG interface. The left panel shows the study-submission form with temporal constraints. The right panel shows the completed causal DAG above a partially visible evidence card for the selected edge; the full card includes confidence, provenance, supporting citation, exact passage, and rationale. this interpretation, 86% of literature-cohort false￾positive edges under the cold setting (87% with study con… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    , title =

    Greenland, Sander and Pearl, Judea and Robins, James M. , title =. Epidemiology , year =

  2. [2]

    Pearl, Judea , title =

  3. [3]

    Spirtes, Peter and Glymour, Clark and Scheines, Richard , title =

  4. [4]

    and Xing, Eric P

    Zheng, Xun and Aragam, Bryon and Ravikumar, Pradeep K. and Xing, Eric P. , title =. Advances in Neural Information Processing Systems , year =

  5. [5]

    and Liskiewicz, Maciej and Ellison, George T

    Textor, Johannes and van der Zander, Benito and Gilthorpe, Mark S. and Liskiewicz, Maciej and Ellison, George T. H. , title =. International Journal of Epidemiology , year =

  6. [6]

    2020 , eprint =

    Sharma, Amit and Kiciman, Emre , title =. 2020 , eprint =

  7. [7]

    Causal Reasoning and Large Language Models: Opening a New Frontier for Causality , year =

    K. Causal Reasoning and Large Language Models: Opening a New Frontier for Causality , year =. doi:10.48550/arXiv.2305.00050 , url =. 2305.00050 , archivePrefix =

  8. [8]

    2024 , eprint =

    Darvariu, Victor-Alexandru and Hailes, Stephen and Musolesi, Mirco , title =. 2024 , eprint =. doi:10.48550/arXiv.2405.13551 , url =

  9. [9]

    Large Language Models for Constrained-Based Causal Discovery

    Cohrs, Kai-Hendrik and Varando, Gherardo and Diaz, Emiliano and Sitokonstantinou, Vasileios and Camps-Valls, Gustau , title =. 2024 , eprint =. doi:10.48550/arXiv.2406.07378 , url =

  10. [10]

    2024 , eprint =

    Jiralerspong, Thomas and Chen, Xiaoyin and More, Yash and Shah, Vedant and Bengio, Yoshua , title =. 2024 , eprint =. doi:10.48550/arXiv.2402.01207 , url =

  11. [11]

    2024 , eprint =

    Zhang, Yuzhe and Zhang, Yipeng and Gan, Yidong and Yao, Lina and Wang, Chen , title =. 2024 , eprint =. doi:10.48550/arXiv.2402.15301 , url =

  12. [12]

    Causal Discovery from Data Assisted by Large Language Models

    Barakati, Kamyar and Molak, Alexander and Nelson, Chris and Zhang, Xiaohang and Takeuchi, Ichiro and Kalinin, Sergei V. , title =. 2025 , eprint =. doi:10.48550/arXiv.2503.13833 , url =

  13. [13]

    Nucleic Acids Research , year =

    Bodenreider, Olivier , title =. Nucleic Acids Research , year =

  14. [14]

    Nucleic Acids Research , year =

    Wei, Chih-Hsuan and Allot, Alexis and Leaman, Robert and Lu, Zhiyong , title =. Nucleic Acids Research , year =

  15. [15]

    Proceedings of the 18th BioNLP Workshop and Shared Task , year =

    Neumann, Mark and King, Daniel and Beltagy, Iz and Ammar, Waleed , title =. Proceedings of the 18th BioNLP Workshop and Shared Task , year =. doi:10.18653/v1/W19-5034 , url =

  16. [16]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , year =

    Beltagy, Iz and Lo, Kyle and Cohan, Arman , title =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , year =. doi:10.18653/v1/D19-1371 , url =

  17. [17]

    Bioinformatics , year =

    Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo , title =. Bioinformatics , year =

  18. [18]

    and McKenzie, Joanne E

    Page, Matthew J. and McKenzie, Joanne E. and Bossuyt, Patrick M. and Boutron, Isabelle and Hoffmann, Tammy C. and Mulrow, Cynthia D. and Shamseer, Larissa and Tetzlaff, Jennifer M. and Akl, Elie A. and Brennan, Sue E. and Chou, Roger and Glanville, Julie and Grimshaw, Jeremy M. and Hr. The. BMJ , year =

  19. [19]

    and Fiszman, Marcelo , title =

    Rindflesch, Thomas C. and Fiszman, Marcelo , title =. Journal of Biomedical Informatics , year =

  20. [20]

    , title =

    Kilicoglu, Halil and Shin, Dongwook and Fiszman, Marcelo and Rosemblat, Graciela and Rindflesch, Thomas C. , title =. Bioinformatics , year =

  21. [21]

    BMC Bioinformatics , year =

    Mihaila, Claudiu and Ohta, Tomoko and Pyysalo, Sampo and Ananiadou, Sophia , title =. BMC Bioinformatics , year =

  22. [22]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , year =

    Wadden, David and Lin, Shanchuan and Lo, Kyle and Wang, Lucy Lu and van Zuylen, Madeleine and Cohan, Arman and Hajishirzi, Hannaneh , title =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , year =. doi:10.18653/v1/2020.emnlp-main.609 , url =

  23. [23]

    Advances in Neural Information Processing Systems , year =

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and Kuttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rocktaschel, Tim and Riedel, Sebastian and Kiela, Douwe , title =. Advances in Neural Information Processing Systems , year =

  24. [24]

    and Gascoyne, Randy D

    Boyle, Terry and Gallagher, Richard P. and Gascoyne, Randy D. and Connors, Joseph M. and Le, Nhu D. and Spinelli, John J. , title =. Cancer Epidemiology, Biomarkers & Prevention , year =

  25. [25]

    and Gerberich, Susan Goodwin and Ryan, Andrew D

    Reiner, Anne M. and Gerberich, Susan Goodwin and Ryan, Andrew D. and Mandel, Jeffrey , title =. Journal of Occupational and Environmental Medicine , year =

  26. [26]

    and Skovlund, Eva and Houthuijs, Danny and Aasvang, Gunn Marit , title =

    Evandt, Jorunn and Oftedal, Bente and Krog, Norun Hjertager and Skurtveit, Svetlana and Nafstad, Per and Schwarze, Per E. and Skovlund, Eva and Houthuijs, Danny and Aasvang, Gunn Marit , title =. Environmental Health , year =