Causal Reasoning in Pieces: Modular In-Context Learning for Causal Discovery

Kacper Kadziolka; Saber Salehkaleybar

REVIEW 3 major objections 7 minor 18 references

Causal Reasoning in Pieces: Modular In-Context Learning for Causal Discovery

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reasoning models reach 83.8 percent F1 on causal discovery when the PC algorithm is split into four prompts.

desk verdict A useful pipeline result that is likely real, but the headline numbers need error bars, a validation split, and a contamination check before the absolute gains are believable. read the letter →

arxiv 2507.23488 v1 pith:NADE2IOJ submitted 2025-07-31 cs.AI

Kacper Kadziolka , Saber Salehkaleybar This is my paper · ORCID

classification cs.AI

keywords causaldiscoverylargelanguagemodelsin-contextlearningPCalgorithmCorr2Causebenchmarkreasoningpromptdecompositionzero-shot

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that state-of-the-art reasoning LLMs can do causal discovery from natural-language conditional-independence statements, and that their ability is sharply amplified when the PC algorithm is split into four separate in-context prompts. On the Corr2Cause benchmark, the four-stage pipeline lifts OpenAI o3-mini's F1 from 66.28 with a single combined prompt to 83.83, against 33.38 for the best published conventional baseline, with no fine-tuning. If that is right, prompt decomposition, not extra training or model scale alone, is the key to turning LLM reasoning traces into reliable algorithmic inference. The paper also shows that the decomposition concentrates errors in the later orientation and hypothesis-evaluation stages, and that reasoning models outperform conventional ones by revisiting and correcting their own intermediate conclusions.

What carries the argument

The load-bearing machinery is the PC algorithm recast as a four-stage prompt pipeline: undirected skeleton extraction from correlation and independence statements, v-structure identification using separation sets, edge orientation via Meek's rules subject to cycle avoidance and v-structure preservation, and hypothesis evaluation against the Markov equivalence class expressed as a CPDAG. Each stage is a separate persona-framed prompt whose output is parsed into a canonical Python object and fed into the next prompt, so the model's internal reasoning is restarted four times instead of once. The CPDAG is the intermediate object that lets the final stage answer true only when the hypothesis holds in every valid DAG, which is exactly what the benchmark asks.

What would settle it

Permute the variable names and re-parse the independence statements of the Corr2Cause test set while keeping every causal relation identical, then run the four-stage o3-mini pipeline; if F1 collapses toward the 33.38 conventional baseline, the reported gain is mostly benchmark memorization.

Watch

Extended reading notes

Core claim

On its own terms, the paper reports that reasoning-first LLM families, OpenAI o3-mini and DeepSeek-R1, already beat every published Corr2Cause result in a zero-shot single prompt that embeds the full PC algorithm, and that a four-module version of the same prompt, skeleton extraction, v-structure identification, Meek-rule orientation, and hypothesis evaluation, raises performance to an F1 of 83.83 for o3-mini, with precision 90.91 and accuracy 95.32. The authors attribute the gain to stage-wise re-activation of the model's reasoning trace, structured intermediate artifacts passed between prompts via parsers, and a persona that anchors each stage in the PC algorithm. They further claim that the pipeline delivers consistent improvements for every model tested and that the later stages are the main bottleneck.

Load-bearing premise

The result presupposes that o3-mini and the DeepSeek-R1 API were not trained on Corr2Cause, so their answers reflect causal reasoning rather than memorized examples; the paper reports no contamination check.

Editorial extensions

If this is right

If the result holds, causal discovery from verbalized independence statements no longer requires fine-tuning: a reasoned prompt decomposition matches or beats trained baselines.
The four-stage blueprint transfers to any task with a known algorithmic decomposition, since each stage gets focused attention and an inspectable intermediate artifact, at the cost of extra tokens and latency.
Stage-wise scoring identifies the bottleneck: skeleton and v-structure stages are nearly perfect, while Meek-rule orientation and hypothesis evaluation drive the remaining errors, so those are the stages where better prompts or verification would pay off.
The reported token-to-F1 improvement implies a practical budgeting rule: accuracy can be traded against cost by choosing how many pipeline stages to run.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the within-model gain from single-prompt to pipeline is largely immune to benchmark-contamination worries, because both conditions use the same model and API, so the modularity benefit is a cleaner signal than the absolute F1.
Beyond the paper: a cheap falsification test would freeze the four prompts and shuffle variable names and rephrase the independence statements; if the pipeline F1 drops far below 83.83, the model is exploiting surface statistics rather than causal structure.
Beyond the paper: the same stage architecture should apply to other constraint-based causal-discovery algorithms such as FCI, where the extra stages for latent confounders would give an immediate test of generality.
Beyond the paper: the claim that longer reasoning traces cause the gains predicts that artificially capping per-stage token budgets, or suppressing the model's self-check behavior, should degrade later stages more than skeleton extraction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a credible engineering paper with a useful result, but the headline numbers are not yet trustworthy because the evaluation protocol is under-specified. The staged-prompting idea is not new — PC-SubQ already did staged prompting for the PC algorithm — but the application to reasoning-specialist models and the stage-wise error analysis are new. The paper does several things well: it releases code and prompt templates, it reports a consistent within-model improvement for every pipeline over its baseline (all three models gain), and it is candid about limitations, including the single benchmark and the missing confidence intervals, which Appendix E explicitly defers to a final version. The reasoning-trace analysis is a nice addition.

The soft spots are exactly where the reader puts them. The o3-mini pipeline F1 of 83.83 rests on one run with no error bar. Appendix B says there were three baseline runs but only one pipeline run. Appendix D describes iterative prompt engineering but never defines a development/validation split, so the prompts may have been selected on the same 1,162-sample test split whose scores are then reported. And there is no contamination check for the API models; Corr2Cause is public and has been used in prior LLM evals, so the absolute numbers could be inflated. The within-model pipeline-versus-baseline gap is less vulnerable to contamination, and the fact that it holds across all three models is reassuring. But the "latent causal-reasoning ability" interpretation rests on the absolute comparison with BART MNLI, which is exactly the comparison contamination would inflate.

My view: the central argument survives in a weaker form — staged prompting helps reasoning models on this benchmark. That is worth publishing once error bars, held-out prompt selection, and a leakage check are added. As written, it is a solid workshop paper or a conditional acceptance with required revisions. I would not cite the absolute numbers yet, but I would read the revision and expect the authors can address these concerns without much trouble.

Referee Report

3 major / 7 minor

Summary. The paper studies causal discovery on the Corr2Cause benchmark using reasoning-specialist large language models (DeepSeek-R1-70B, DeepSeek-R1 API, and OpenAI o3-mini). It proposes a four-stage modular in-context pipeline that decomposes the PC algorithm into skeleton extraction, v-structure identification, Meek-rule orientation, and hypothesis evaluation, with Python parsers passing structured intermediate outputs between stages. Table 2 reports an F1 of 83.83 for o3-mini in the pipeline, compared with 66.28 for the single-prompt baseline and 33.38 for the best published conventional baseline (BART MNLI), and the paper interprets these gains as evidence that reasoning-specialist LLMs possess latent causal-reasoning ability that careful prompt decomposition can unlock. The paper also analyzes token usage, stage-wise errors, and qualitative reasoning-trace differences.

Significance. The proposed modular pipeline is a simple and potentially useful empirical contribution: it requires no fine-tuning, it is grounded in a well-known algorithm, and the stage-wise error analysis and token-cost comparison are informative. The authors promise to release code, prompt templates, and evaluation scripts, which would aid reproducibility. However, the headline claims currently rest on single-run point estimates from one public benchmark, prompt engineering was performed iteratively on that benchmark without a defined validation split, and no contamination analysis is provided for the API models. If these evaluation gaps are closed, the paper could be a solid empirical contribution to LLM-based causal discovery; in its current form the evidence is not strong enough to support the strength of the conclusions.

major comments (3)

[Section 4.1, Table 2, Appendices B and E] The headline pipeline results are single-run point estimates. Appendix E explicitly states that bootstrap confidence intervals were computed only for the DeepSeek-R1 API single-prompt baseline (mean F1 0.6527, 95% CI [0.6461, 0.6616]) and that standard deviations for the remaining experiments will be reported 'in the final version.' Without run-to-run variability for the o3-mini pipeline result (83.83) and the 66.28 single-prompt baseline, the 17.55-point improvement cannot be assessed for statistical significance, and the token-efficiency comparison in Figure 3 is similarly based on unreplicated measurements. The authors should provide multiple runs and bootstrap intervals for all cells in Table 2, or at minimum for the headline baseline-versus-pipeline comparisons.
[Section 4.1 and Appendix D] The paper states that the stage-wise prompts were 'refined through a careful, iterative prompt engineering process' but never defines a development/validation split. Evaluation is performed on the full 1,162-sample test split of Corr2Cause. If prompt variants were selected by their scores on this same split, the reported F1 is optimistically biased, and the claim that the framework is a generalizable blueprint is not supported. The authors should report how many prompt variants were tried, define a held-out development split for prompt selection, and evaluate the final prompts on a separate test split or through a nested protocol.
[Section 4.1 and Appendix B] No contamination or leakage check is performed for the closed API models. Corr2Cause is a publicly hosted Hugging Face dataset that has been used in earlier LLM evaluations, and the training corpora of o3-mini and DeepSeek-R1 are undisclosed. If those models memorized benchmark instances, the absolute gains over BART MNLI and the 'native causal reasoning' interpretation would be inflated. The authors should add a perturbation analysis (for example, variable-identifier substitution or paraphrasing, as in the original Corr2Cause study) and/or a leakage probe. The within-model pipeline-versus-baseline difference would be less affected by contamination, so such a check would help bound the main claim.

minor comments (7)

[Abstract and Section 5] The phrase 'nearly three-fold improvements' overstates Table 2: 83.83 versus 33.38 is a factor of 2.51, not three.
[Appendix B] The model name 'LlaMa3.3-70B' should be written as 'LLaMA-3.3-70B'.
[Appendix D, Listings 3 and 4] The prompt templates use 'Casual skeleton' where 'Causal skeleton' is meant.
[Section 4.3] The text repeatedly uses 'common model' where 'conventional model' is meant; this makes the failure analysis harder to read.
[Appendix E] The statement 'Using five bootstrap samples (R = 5, B = 1000)' is confusing: if B = 1000 resamples, R = 5 likely refers to five independent bootstrap repetitions, but this should be stated explicitly.
[Section 3.5] PC-SubQ is discussed quantitatively in the text but its published F1 values are not included in Table 2, making the claimed comparison difficult for the reader to verify.
[References] The reference to Meek (2013) appears to be a misdating of the standard citation Meek (1995) for causal-inference rules with background knowledge; please verify and correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claims are empirical evaluations on an external benchmark, grounded in the external PC algorithm, with no equation-level reduction or self-citation chain.

full rationale

The paper makes no formal derivation whose output is equivalent to its input. The central result is an empirical F1 comparison on the external Corr2Cause benchmark. The modular pipeline is explicitly grounded in the PC algorithm (Spirtes et al., 2001) and its four stages are standard decompositions of that algorithm, not quantities defined in terms of the reported scores. No fitted parameter is renamed as a prediction; the prompt templates were refined only with 'programmatic JSON-schema checks to ensure correct, schema-compliant outputs' (Appendix D), which is output-format validation, not label fitting. There are no self-citations by the present authors, so no self-citation chain is load-bearing. The final hypothesis-evaluation stage mirrors the benchmark's own definition of validity within a Markov equivalence class, but that is the task definition, not a circular derivation. Appendix E explicitly reports standard deviations only for the DeepSeek-R1 API baseline and defers others to a 'final version'; Section 5 concedes that evaluation is confined to a single benchmark. These are legitimate validity and reproducibility limitations, as are the absence of a contamination check for the API models and the single pipeline run, but they do not make any claim circular by construction. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No fitted numerical parameters enter the core comparison; the main tuning choices are hyperparameters (temperature) and prompt design. The central claims rest on standard causal-discovery assumptions from the benchmark and on two unstated methodological assumptions: no training-data contamination for the API models, and prompt tuning that did not use the test split. These assumptions are not verified in the text.

free parameters (1)

DeepSeek-R1 sampling temperature = 0.1 (tested 1.0, 0.6, and 0.1)
The paper reports results at three temperatures for the self-hosted 70B model and uses 0.1 for the headline pipeline runs; this selection is benchmark-driven and could affect reported F1.

assumptions (4)

domain assumption The causal Markov condition and faithfulness hold for the distributions underlying Corr2Cause, so d-separation faithfully reflects conditional independence.
Invoked in Section 3.1 as correctness conditions of PC; inherited from the benchmark's construction.
domain assumption The Corr2Cause benchmark's premise statements and ground-truth labels were generated correctly.
The paper relies on the benchmark's construction (Section 2.2) without independent verification.
domain assumption The large API models were not trained on the Corr2Cause benchmark.
Implicit in Section 4.1 and Appendix B; no contamination check is reported, and the benchmark is public.
ad hoc to paper Prompt engineering was performed on data separate from the evaluation split.
Appendix D describes iterative prompt refinement but does not state a held-out validation split, so the assumption that test performance was not used to select prompts is unstated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Causal Reasoning in Pieces: Modular In-Context Learning for Causal Discovery." pith.science (2026). https://pith.science/paper/NADE2IOJ

@misc{pith2026250723488,
  author       = {Pith},
  title        = {Pith review of: Causal Reasoning in Pieces: Modular In-Context Learning for Causal Discovery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NADE2IOJ}},
  note         = {Machine review of arXiv:2507.23488}
}

read the original abstract

Causal inference remains a fundamental challenge for large language models. Recent advances in internal reasoning with large language models have sparked interest in whether state-of-the-art reasoning models can robustly perform causal discovery-a task where conventional models often suffer from severe overfitting and near-random performance under data perturbations. We study causal discovery on the Corr2Cause benchmark using the emergent OpenAI's o-series and DeepSeek-R model families and find that these reasoning-first architectures achieve significantly greater native gains than prior approaches. To capitalize on these strengths, we introduce a modular in-context pipeline inspired by the Tree-of-Thoughts and Chain-of-Thoughts methodologies, yielding nearly three-fold improvements over conventional baselines. We further probe the pipeline's impact by analyzing reasoning chain length, complexity, and conducting qualitative and quantitative comparisons between conventional and reasoning models. Our findings suggest that while advanced reasoning models represent a substantial leap forward, carefully structured in-context frameworks are essential to maximize their capabilities and offer a generalizable blueprint for causal discovery across diverse domains.

Figures

Figures reproduced from arXiv: 2507.23488 by the authors.

**Figure 2.** Stage-wise in-context pipeline for causal discovery. Each vertical panel represents a [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Efficiency trade-off on the CORR2CAUSE benchmark. Scatter points denote F1 scores against average token usage for both baseline (circles) and pipeline (triangles) settings. Solid lines connect each model’s baseline and pipeline points as trend lines, highlighting how increased token investment in the pipeline correlates with F1 improvements. (33.38), reflecting the limitations of conventional language models. In con… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: (Left) Boxplot of conditional independence sets metrics—number of tests and maximum [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Graphical and formal definitions of the four fundamental causal relationships underpinning [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 16 canonical work pages

[1]

Bagheri, A., Alinejad, M., Bello, K., and Akhondi-Asl, A. (2024). C ^2 P : Featuring large language models with causal reasoning

work page 2024
[2]

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

2020
[3]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

DeepSeek-AI (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

work page 2025
[4]

Jin, Z., Liu, J., Lyu, Z., Poff, S., Sachan, M., Mihalcea, R., Diab, M., and Schölkopf, B. (2024). Can large language models infer causation from correlation?

work page 2024
[5]

Liu, X., Wu, Z., Wu, X., Lu, P., Chang, K.-W., and Feng, Y. (2024). Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data

work page 2024
[6]

Meek, C. (2013). Causal inference and causal explanation with background knowledge

work page 2013
[7]

Openai o3-mini: Pushing the frontier of cost-effective reasoning

OpenAI (2025). Openai o3-mini: Pushing the frontier of cost-effective reasoning. https://openai.com/research/openai-o3-mini. Accessed: April 5, 2025

work page 2025
[8]

Pearl, J. (1988). The morgan kaufmann series in representation and reasoning. In Probabilistic Reasoning in Intelligent Systems , page i. Morgan Kaufmann, San Francisco (CA)

work page 1988

Show all 18 references

[9]

Scheines, R. (2005). An introduction to causal inference. Technical report, Carnegie Mellon University

2005
[10]

W., Doucet, A., Gretton, A., and Chiappa, S

Sgouritsa, E., Aglietti, V., Teh, Y. W., Doucet, A., Gretton, A., and Chiappa, S. (2024). Prompting strategies for enabling large language models to infer causation from correlation. arXiv preprint arXiv:2412.13952

2024 arXiv
[11]

Spirtes, P., Glymour, C., Scheines, R., and Heckerman, D. (2001). Causation, Prediction, and Search . Adaptive Computation and Machine Learning. The MIT Press, Cambridge, MA, 2 edition. Special Collection: CogNet

2001
[12]

Y., Kraft, A., Jin, L., Cai, C., Hosseini, A., Xu, T., Zhang, Z., Hong, L., Chi, E

Tsai, A. Y., Kraft, A., Jin, L., Cai, C., Hosseini, A., Xu, T., Zhang, Z., Hong, L., Chi, E. H., and Yi, X. (2024). Leveraging llm reasoning enhances personalized recommender systems

2024
[13]

Tseng, Y.-M., Huang, Y.-C., Hsiao, T.-Y., Chen, W.-L., Huang, C.-W., Meng, Y., and Chen, Y.-N. (2024). Two tales of persona in llms: A survey of role-playing and personalization

2024
[14]

Uhler, C., Raskutti, G., B \"u hlmann, P., and Yu, B. (2013). Geometry of the faithfulness assumption in causal inference . The Annals of Statistics , 41(2):436 -- 463

2013
[15]

Wan, G., Lu, Y., Wu, Y., Hu, M., and Li, S. (2025). Large language models for causal discovery: Current landscape and future directions

2025
[16]

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2023). Chain-of-thought prompting elicits reasoning in large language models

2023
[17]

L., Cao, Y., and Narasimhan, K

Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models

2023
[18]

S., and Kersting, K

Zečević, M., Willig, M., Dhami, D. S., and Kersting, K. (2023). Causal parrots: Large language models may talk causality but are not causal

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Bagheri, A., Alinejad, M., Bello, K., and Akhondi-Asl, A. (2024). C ^2 P : Featuring large language models with causal reasoning

work page 2024

[2] [2]

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

2020

[3] [3]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

DeepSeek-AI (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

work page 2025

[4] [4]

Jin, Z., Liu, J., Lyu, Z., Poff, S., Sachan, M., Mihalcea, R., Diab, M., and Schölkopf, B. (2024). Can large language models infer causation from correlation?

work page 2024

[5] [5]

Liu, X., Wu, Z., Wu, X., Lu, P., Chang, K.-W., and Feng, Y. (2024). Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data

work page 2024

[6] [6]

Meek, C. (2013). Causal inference and causal explanation with background knowledge

work page 2013

[7] [7]

Openai o3-mini: Pushing the frontier of cost-effective reasoning

OpenAI (2025). Openai o3-mini: Pushing the frontier of cost-effective reasoning. https://openai.com/research/openai-o3-mini. Accessed: April 5, 2025

work page 2025

[8] [8]

Pearl, J. (1988). The morgan kaufmann series in representation and reasoning. In Probabilistic Reasoning in Intelligent Systems , page i. Morgan Kaufmann, San Francisco (CA)

work page 1988