Pith. sign in

REVIEW 2 major objections 4 minor 21 references

Domain knowledge graphs let LLMs recover hidden causal factors and annotate free text well enough for statistical algorithms to build accurate causal graphs in specialist domains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 03:41 UTC pith:DYKVTGU7

load-bearing objection Solid engineering extension of COAT that uses domain KGs for latent factors and annotation clues; gains are real on the authors’ synthetic benchmarks but those benchmarks plant the clues by design. the 2 major comments →

arxiv 2607.09348 v1 pith:DYKVTGU7 submitted 2026-07-10 cs.CL

DKCD: Domain Knowledge-Enhanced Causal Discovery from Unstructured Data

Xin Li , Jin Li , Shoujin Wang , Kun Yu , Fang Chen This is my paper
classification cs.CL
keywords causal discoveryunstructured datalarge language modelsdomain knowledge graphslatent factorshealthcareFCI algorithmfactor annotation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Pure LLM pipelines that turn clinical free text into causal graphs fail in two places: they miss latent factors that never appear by name, and they annotate factor values inconsistently because they lack domain-grounded reasoning. DKCD attacks both failures by retrieving subgraphs from a domain knowledge graph that match the observable factors the LLM can already extract, then using that knowledge to propose the missing latent factors and to write short causal clues that guide scoring of every sample. With a more complete factor set and cleaner structured table, a classical algorithm such as FCI recovers a graph much closer to the ground truth. On two synthetic medical datasets the method raises node and adjacency F1 scores and lowers extended structural Hamming distance across four different LLM backbones. A sympathetic reader cares because unstructured text dominates high-expertise fields yet existing causal tools still demand clean tables.

Core claim

DKCD demonstrates that injecting retrieved domain-knowledge subgraphs into LLM reasoning recovers latent causal factors that surface-text methods miss and supplies causal clues that make factor-value annotation reliable enough for classical structure-learning algorithms to produce substantially more accurate causal graphs from unstructured domain text.

What carries the argument

The three-stage DKCD pipeline—Knowledge Mining (observable-factor extraction, KG subgraph retrieval, semantic matching), Knowledge-guided Causal Reasoning (latent-factor discovery plus causal-clue generation), and Causal Structure Discovery (FCI on the completed annotation table)—is the mechanism that converts incomplete free text into a usable causal graph.

Load-bearing premise

The synthetic patient notes are written so that latent factors leave recoverable clinical clues; if real notes hide those factors more deeply or noisily, the reported recovery gains shrink.

What would settle it

Run the identical pipeline on a corpus of real de-identified clinical notes that have expert-validated ground-truth causal graphs and no deliberately planted latent-factor language; if node recall for latent factors and ESHD no longer beat COAT, the central claim fails for realistic text.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Latent factors such as kidney disease or genetic risk that never appear by name can still be recovered when a domain knowledge graph is available.
  • Annotation errors that previously poisoned conditional-independence tests are reduced by the causal clues, raising adjacency recall and F1.
  • The same pipeline works across GPT-4o, Gemini 2.5 Flash, Grok-3 and LLaMA 3-70B, indicating the gain is not model-specific.
  • Domain knowledge graphs become a practical prerequisite for trustworthy causal discovery from specialist free text.
  • Ablations show both latent-factor recovery and causal clues contribute; removing either degrades the final graph.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same retrieval-plus-clue pattern could transfer to finance or education once comparable domain knowledge graphs exist.
  • Performance on real EHR notes without deliberately planted latent clues remains the decisive open test of whether the synthetic gains hold.
  • Causal clues might later serve as soft constraints or priors inside score-based structure learners, not only as annotation aids.
  • Semantic-matching quality (top-r) is a tunable bottleneck; better domain embeddings could further reduce noise.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes DKCD, a three-module framework (Knowledge Mining, Knowledge-guided Causal Reasoning, Causal Structure Discovery) that retrieves subgraphs from domain knowledge graphs, uses them with LLMs to surface latent causal factors and generate causal clues for annotation, then runs FCI on the resulting structured table. It claims this addresses incomplete latent-factor identification (CH1) and unreliable annotation (CH2) that limit prior LLM-only pipelines such as COAT. On two author-constructed medical synthetic datasets (Diabetes, Respiratory; 400 notes each), multi-LLM experiments (GPT-4o, Gemini 2.5 Flash, Grok-3, LLaMA 3-70B) report higher node precision/recall/F1 and improved adjacency metrics / lower ESHD versus Zero-shot LLM, META and COAT, with supporting ablations and a top-r parameter study.

Significance. If the gains transfer beyond the synthetic setting, the work would supply a practical bridge between unstructured clinical (or other high-expertise) text and classical causal discovery algorithms, and the multi-backbone tables plus clean ablations (w/o CC, w/o LF, w/o SM, w/o SP) already constitute a useful empirical baseline for the community. The explicit use of real-world KGs (DiaKG, literature-derived respiratory triples) and the release of the first domain-specific unstructured-to-DAG benchmarks are concrete contributions. The result remains conditional on the realism of the planted latent clues; under that caveat the paper is a solid incremental advance rather than a foundational breakthrough.

major comments (2)
  1. [§4.1, Appendix B.1, Tables 1–2] §4.1 and Appendix B.1 (Prompts B.1/B.2): the synthetic notes are generated from the same ground-truth DAGs that later serve as evaluation targets, and the generation prompts explicitly instruct the LLM to “embed natural clinical clues that imply the following latent factors without explicitly labeling them.” Consequently the measured lift in NR/NF (Tables 1–2) and the recovery of red latent nodes (Figures 3, 11) largely tests re-extraction of deliberately planted surface cues that match the retrieval KG, rather than recovery of truly implicit factors from ordinary clinical prose. This is load-bearing for the central claim of solving CH1; either a quantitative comparison against real EHR notes (or a controlled “clue-difficulty” ablation) or a substantially stronger Limitations discussion with estimated optimism bias is required before the claim can be accepted at face value.
  2. [§3.3, Table 3] §3.3–3.4 and Algorithm 1: causal clues H^(k) are generated by the same LLM that later performs annotation, yet no analysis is provided of whether these clues introduce systematic annotation bias (e.g., over-emphasizing KG-supported edges). The ablation “w/o CC” shows only modest AF/ESHD degradation; a controlled experiment that injects deliberately incorrect clues would be needed to confirm that the clues improve rather than merely regularize the scoring table S that is fed to FCI.
minor comments (4)
  1. [§3.2, Eq. (4)] Figure 1 caption and §3.2: the semantic-matching formula (Eq. 4) uses all-MiniLM-L6-v2; a short sensitivity check with a domain-adapted encoder would strengthen the claim that the retrieval step is robust.
  2. [Tables 1–2] Tables 1–2 report means ± std over three runs; given the stochasticity of LLM decoding, five or more seeds (or temperature=0) would make the error bars more informative.
  3. [Appendix B.4] Appendix B.4: factor-name alignment is performed manually; a brief inter-annotator agreement statistic would clarify reproducibility of the NP/NR/NF numbers.
  4. [Figures 1, 9, 11] Typos: “Challenege 2” (Fig. 1), “Aisa Travel” (Fig. 11 and B.9), “Cadiometabolic” (Fig. 9); also inconsistent capitalization of “Hyperuricemia/HUA”.

Circularity Check

1 steps flagged

Synthetic notes plant recoverable latent-factor clues by design (Prompt B.1/B.2), so CH1 recovery gains are partly by construction of the evaluation data rather than pure discovery.

specific steps
  1. other [§4.1 + Appendix B.1.1/B.1.2 + Prompt B.1/B.2]
    "Embed natural clinical clues that imply the following latent factors without explicitly labeling them: • Neuropathy: numbness, tingling, burning sensation, or reduced sensation in the feet. • Obesity: elevated body weight... • ChronicKidneyDisease: albuminuria, reduced eGFR... Only when the reconstructed graph is structurally consistent with the ground-truth causal graph are the structured samples used to generate descriptions..."

    The unstructured evaluation texts are generated from the same ground-truth DAGs that later serve as the recovery target, and the generation prompts deliberately plant recoverable surface cues for every latent factor. Consequently, success on NP/NR/NF (and the downstream graph metrics) partly measures re-extraction of author-planted clues rather than discovery of truly implicit factors from ordinary clinical prose; the measured lift over baselines is therefore not fully independent of the data-construction process.

full rationale

The DKCD pipeline itself (KG retrieval + LLM latent-factor proposal + causal-clue annotation + FCI) is not definitionally circular: the domain KGs are external (DiaKG or literature-extracted), FCI is a standard off-the-shelf algorithm, and no parameter is fitted to the target graph and then re-predicted. The only mild circularity is in the evaluation design. Ground-truth DAGs first generate structured scores; those scores are verified by FCI; only then are unstructured notes produced by prompts that explicitly instruct the LLM to embed surface clinical clues for exactly the latent factors later scored as ‘discovered’. Relative gains over COAT/META remain informative, but absolute claims about latent-factor identification (CH1) and the resulting ESHD/AR/AF lifts are partially forced by the planted cues. No self-citation uniqueness theorem or ansatz smuggling is load-bearing. Score 3 reflects this single, non-central evaluation circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

The central empirical claim rests on a small set of free design choices (top-r, sample size for factor proposal), standard causal-discovery assumptions, and the untested premise that the synthetic generation process fairly represents real clinical implication. No new physical entities are postulated; the only invented intermediate is the “causal clue” object used for annotation guidance.

free parameters (2)
  • top-r knowledge statements per observable factor = 20
    Chosen by grid search on the diabetes set (Table 6); r=20 is used for all main results and directly affects both latent-factor recall and noise.
  • number of seed samples for observable-factor proposal = ≤20
    Empirically set to ≤20 per outcome group following COAT; not ablated.
axioms (3)
  • domain assumption Domain knowledge graphs contain the latent factors and relations needed for correct causal recovery, and semantic matching with all-MiniLM-L6-v2 retrieves them without systematic bias.
    Invoked throughout §3.2–3.3; never independently validated on held-out clinical text.
  • domain assumption FCI’s faithfulness and causal Markov assumptions hold for the annotated scoring tables produced by the LLM.
    Standard for any FCI application; stated in §3.4.
  • ad hoc to paper LLM-generated causal clues improve annotation accuracy without introducing new systematic errors.
    Supported only by the ablation “w/o CC”; no external validation of clue quality.
invented entities (1)
  • causal clues no independent evidence
    purpose: Intermediate natural-language statements of plausible factor dependencies used solely to guide LLM annotation of the scoring table.
    Defined in §3.3 and Prompt B.5; no independent existence outside the pipeline.

pith-pipeline@v1.1.0-grok45 · 29472 in / 2585 out tokens · 36726 ms · 2026-07-13T03:41:13.813737+00:00 · methodology

0 comments
read the original abstract

Causal discovery from unstructured data is a challenging yet underexplored task in high-expertise domains such as healthcare, finance, and education. Existing methods typically leverage the general knowledge of large language models (LLMs) to identify causal factors from unstructured data and annotate them into structured data for causal graph construction. However, they remain limited by two key challenges (CHs): (CH1) insufficient identification of latent factors, which are implicit in the data yet essential for causal discovery, due to the lack of domain-specific knowledge; and (CH2) unreliable factor annotation, caused by the lack of domain-grounded reasoning, which propagates errors to the resulting causal graphs. To address these challenges, we introduce a novel Domain Knowledge-enhanced Causal Discovery framework (DKCD) for causal discovery from unstructured data in high-expertise domains with three interconnected components: (1) Knowledge Mining: It retrieves relevant domain knowledge based on observable factors to support subsequent causal reasoning. (2) Knowledge-guided Causal Reasoning: Reasoning with relevant knowledge, it discovers latent causal factors to address CH1 and generates key causal clues for more accurate data annotation to address CH2. (3) Causal Structure Discovery: It constructs the final causal graphs based on a more complete factor set and accurate annotations. Experiments on two domain-specific datasets show that DKCD significantly improves both causal factor identification and causal graph construction.

Figures

Figures reproduced from arXiv: 2607.09348 by Fang Chen, Jin Li, Kun Yu, Shoujin Wang, Xin Li.

Figure 1
Figure 1. Figure 1: Overview of the DKCD framework. DKCD leverages domain knowledge graphs to guide latent factor [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The COAT result on the diabetes dataset. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ground-truth and discovered causal graphs on the diabetes dataset. Latent factors are marked in red. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ground truth causal graph and the causal graph reconstructed by the FCI algorithm from the generated [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ground-truth causal graph, generated structured dataset, and the causal graph reconstructed by the FCI [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An example of the causal discovery process of DKCD on the Diabetes dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Causal graphs discovered with GPT-4o on the Diabetes dataset. Latent factors are marked in red. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Causal graphs discovered with Gemini 2.5 Flash on the Diabetes dataset. Latent factors are marked in red. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Causal graphs discovered with Grok-3 on the Diabetes dataset. Latent factors are marked in red. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Causal graphs discovered with LLaMA 3-70B on the Diabetes dataset. Latent factors are marked in red. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Causal graphs discovered with GPT-4o on the Respiratory dataset. Latent factors are marked in red. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Causal graphs discovered with Gemini 2.5 Flash on the Respiratory dataset. Latent factors are marked in [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Causal graphs discovered with Grok-3 on the Respiratory dataset. Latent factors are marked in red. [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Causal graphs discovered with LLaMA 3-70B on the Respiratory dataset. Latent factors are marked in [PITH_FULL_IMAGE:figures/full_fig_p025_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 1 canonical work pages

  1. [1]

    10.1001/jamanet- workopen.2024.25981

    Use of generative ai to identify helmet status among patients with micromobility-related injuries from unstructured clinical notes.Jama Network Open, 7(8):e2425981. 10.1001/jamanet- workopen.2024.25981. Maxwell Caplin and Freda Festenstein. 1975. Relation between lung cancer, chronic bronchitis, and airways obstruction.British Medical Journal, 3(5985):678...

    work page doi:10.1001/jamanet- 2024

  2. [2]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others

    Springer. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261. Victor-Alexandru...

    Pith/arXiv arXiv 2025

  3. [3]

    Gaël Gendron, Jože M Rožanec, Michael Witbrock, and Gillian Dobbie

    Rethinking of bert sentence embedding for text classification.Neural Computing and Applications, 36(32):20245–20258. Gaël Gendron, Jože M Rožanec, Michael Witbrock, and Gillian Dobbie. 2024. Counterfactual causal inference in natural language with large language models.arXiv preprint arXiv:2410.06392. Chang Gong, Chuzhe Zhang, Di Yao, Jingping Bi, Wen- bi...

    Pith/arXiv arXiv 2024

  4. [4]

    Steffen L Lauritzen and David J Spiegelhalter

    Causal reasoning in large language mod- els: A knowledge graph approach.arXiv preprint arXiv:2410.11588. Steffen L Lauritzen and David J Spiegelhalter. 1988. Local computations with probabilities on graphical structures and their application to expert systems. Journal of the Royal Statistical Society: Series B (Methodological), 50(2):157–194. Jin Li, Shou...

    Pith/arXiv arXiv 1988

  5. [5]

    Ana Rita Nogueira, Andrea Pugnana, Salvatore Rug- gieri, Dino Pedreschi, and João Gama

    Comprehensive review and empirical evalu- ation of causal discovery algorithms for numerical data.arXiv preprint arXiv:2407.13054. Ana Rita Nogueira, Andrea Pugnana, Salvatore Rug- gieri, Dino Pedreschi, and João Gama. 2022. Meth- ods and tools for causal discovery and causal infer- ence.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery...

    Pith/arXiv arXiv 2022

  6. [6]

    same semantics, different structure

    Efficient latent variable causal discovery: Combining score search and targeted testing.arXiv preprint arXiv:2510.04263. Daniela Rodrigues, Noemi Kreif, Anna Lawrence-Jones, Mauricio Barahona, and Erik Mayer. 2022. Re- flection on modern methods: constructing directed acyclic graphs (dags) with domain experts for health services research.International Jou...

    arXiv 2022

  7. [7]

    Xiaohan Zou

    Springer. Xiaohan Zou. 2020. A survey on application of knowl- edge graph. InJournal of Physics: Conference Se- ries, volume 1487, page 012016. IOP Publishing. A Algorithm Algorithm 1:The DKCD Framework Input:D,X s,G KG,f obs,M,L,C. Output:V,S,G

    2020

  8. [8]

    Propose observable factors:V o ←f obs(Xs)

  9. [9]

    (b) Verbalize subgraph: S (k) ={s 1,

    For eachx k ∈ D: (a) Retrieve subgraph:G (k) rel ⊆ G KG. (b) Verbalize subgraph: S (k) ={s 1, . . . , snk }. (c) Semantic matching: S (k) vi ←M(v i,S (k)), ∀vi ∈ V o. (d) Build matched context: eS (k) ={S (k) vi | vi ∈ V o}. (e) Discover latent factors:V l ←L(V o, eS (k)). (f) Generate causal clues: H ←L(V o ∪ Vl,G (k) rel ). (g) Annotate factors:V ← V o ...

  10. [10]

    Discover causal graph:G← C(S,V)

  11. [11]

    ReturnV,S, G. B More Details about Experiments We construct two medical-domain datasets: the Diabetic Patient Condition Descriptions Dataset (Diabetes Dataset) and the Respiratory Patient Con- dition Descriptions Dataset (Respiratory Dataset). Each dataset contains 400 patient condition descrip- tions, a set of high-level factors, and a correspond- ing do...

    2024

  12. [12]

    All experimental results are reported as the mean and standard deviation over three independent runs

    to the annotated scoring table to infer the fi- nal causal graph structure. All experimental results are reported as the mean and standard deviation over three independent runs. All experiments are conducted on a server with two Intel Xeon 6346 CPUs, 256GB RAM, and two NVIDIA A40 GPUs, with all LLMs accessed via API calls. B.4 Evaluation Metrics To evalua...

    2024

  13. [13]

    Tuberculosis → Lung Disease:Tuberculosis infection can lead to severe pulmonary damage and respiratory disease

  14. [14]

    Lung Disease → Dyspnea:Progressive lung disease is strongly associated with breathing difficulty and dyspnea

  15. [15]

    Lung Disease → Chest X-ray:Patients with suspected lung disease commonly undergo chest X-ray examinations for clinical assess- ment

  16. [16]

    A 28-year-old pregnant woman who smokes

    Tuberculosis → Asia Travel:Recent travel to TB-endemic regions increases the likelihood of tuberculosis exposure. A 28-year-old pregnant woman who smokes... A 35-year-old pregnant woman, non-smoker ... A 39-year-old male, who smokes and consumes... Unstructured Data Causal Factor Identification SexPregnancy Diabetic Observable Factors Obesity HUA CKDRisk ...

  17. [17]

    Pregnancy → Diabetic: Pregnancy can influence glucose metabolism and increase the risk of diabetes related conditions

  18. [18]

    Obesity → Diabetic: Obesity is a major risk factor for diabetes, as excess body weight is strongly associated with insulin resistance and impaired glucose regulation

  19. [19]

    Risk Genes → Diabetic: Genetic risk factors can increase susceptibility to diabetes by affecting metabolic regulation and disease predisposition

  20. [20]

    CKD → HUA: Chronic kidney disease can impair uric acid excretion, contributing to hyperuricemia

  21. [21]

    HUA → Diabetic: Hyperuricemia is associated with metabolic dysfunction and may contribute to diabetes- related progression. Annotation Relevant Knowledge Obe Diab RG CKD Preg HUASex Knowledge Mining SexPregnancy Diabetic Observable Factors Figure 6: An example of the causal discovery process of DKCD on the Diabetes dataset. Table 6: Impact of the top- r r...