REVIEW 3 major objections 4 minor 29 references
Do Large Language Models Reason Causally Like Us? Even Better?
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read On collider causal reasoning tasks, GPT-4o, Gemini-Pro, and Claude-3 outperform human reasoners by normative standards, chiefly because they lack the associative bias that skews human judgment.
desk verdict Solid empirical finding on LLM causal reasoning, but the 'more normative than humans' claim rests on a benchmark fit to each agent's own judgments. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the collider causal graph $C_1 \rightarrow E \leftarrow C_2$ with binary variables, embedded in three cover stories (meteorology, economics, sociology). The normative benchmark is a causal Bayes net whose free parameters -- cause priors $w_C$, causal strengths $w_{C_1,E}$ and $w_{C_2,E}$, and an exogenous weight $w_E$ -- are fit to each agent's own judgments through the logistic generating function $p(E=1 \mid C_1,C_2) = 1/(1+\exp(-(C_1 w_{C_1,E} + C_2 w_{C_2,E} + w_E)))$. The psychological comparison model is the mutation sampler, an MCMC process model that produces associative bias when sampling is truncated; its chain-length parameter $\lambda$ measures how much an agent's inferences are influenced by prototype starting states.
What would settle it
Present the same three LLMs with a collider cover story in which the two causes are explicitly described as depending on a common background factor, so that independence is no longer normatively correct. If GPT-4o, Gemini-Pro, and Claude still produce flat independence lines, their collider performance reflects a fixed assumption rather than genuine normative inference.
Extended reading notes
Core claim
In a collider structure $C_1 \rightarrow E \leftarrow C_2$, the two causes are independent, yet humans judge $p(C_1=1 \mid C_2=1) > p(C_1=1 \mid C_2=0)$, a Markov violation called associative bias. The paper reports that GPT-4o, Gemini-Pro, and Claude-3 do not show this bias: they rate the two causes as independent, producing flat lines where humans produce rising slopes. When a causal Bayes net is fit to each agent's judgments and used as a normative standard, these three LLMs show higher correlations with the normative predictions and lower prediction error than humans do. The model fits attribute the advantage to the absence of associative bias rather than to a superior grasp of collider diagnostics; no LLM fully satisfies both conditions of explaining away, and Gemini-Pro is the only model to show the core condition that observing an alternative cause lowers the other cause's probability.
Load-bearing premise
The load-bearing premise is that a causal Bayes net whose parameters are fit to each agent's own judgments can serve as an independent normative benchmark for how normative that agent is.
Editorial extensions
If this is right
- If the fitted CBN is the accepted normative standard, GPT-4o, Gemini-Pro, and Claude-3 are less biased than humans on collider independence, meaning they would not over-infer a correlation between two independent causes in the same situation.
- Because no model fully captures explaining away, LLM-based diagnostic systems may discount alternative causes incorrectly in settings such as medical or fault diagnosis.
- GPT-3.5's systematic failures show that basic predictive inference on colliders should not be assumed across LLMs; capability varies by model generation.
- LLMs fitted roughly equal causal strengths for the two causes despite domain knowledge, suggesting that in these tasks they, like humans, relied more on the instructed structure than on prior knowledge.
- The wider response range of GPT-4o, Gemini, and Claude relative to humans may be an artifact of the human slider's default of 50, so part of the apparent human bias could be procedural rather than cognitive.
Reading between the lines
- As a direct extension, newer or differently prompted LLMs could be screened for explaining away using the same 11 inference tasks; the paper's results predict that later models will more closely satisfy both conditions (i) and (ii).
- The absence of associative bias might be a double-edged sword: in real-world colliders where causes are correlated through unobserved confounders, the same models would be normatively wrong, and their flat lines would hide that misspecification.
- The fitted-CBN benchmark is partly circular because the same judgments that define the benchmark are used to fit it; an independent benchmark would derive parameters from the prompt's causal language itself, allowing a cleaner separation of normative competence from calibration.
- Since the mutation sampler, a process model of human causal reasoning, did not fit the LLMs, a process-level account of how LLMs produce these judgments is still open; the flat independence lines could come from token-level statistical regularities about 'independent causes' in the training data rather than from causal Bayes net computation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper compares human causal likelihood judgments from Rehder and Waldmann (2017) with those of four LLMs (GPT-3.5, GPT-4o, Claude-3-Opus, Gemini-Pro-1.5) on 11 query tasks over a collider graph C1→E←C2, embedded in three cover-story domains. The authors report that GPT-4o, Gemini-Pro, and Claude produce sensible predictive and diagnostic inferences, respect the independence of the two causes (flat lines in Fig. 2c), and show higher correlations with fitted causal Bayes net predictions than humans, whereas GPT-3.5 is markedly worse. They conclude that the better LLMs are often more normatively aligned than humans, largely because they lack the human associative bias, while noting that none fully captures explaining away.
Significance. If established, the finding would be a useful empirical data point for assessing LLM reasoning: it would show that current instruction-following models can apply an experimentally taught collider structure, avoid a robust human bias (Markov violations), and yet still miss subtler normative patterns such as explaining away. The paper's direct behavioral displays (Fig. 2c), the use of pre-existing human data, and the attempt to connect results to computational models are strengths. Its significance is attenuated, however, by the fact that the 'normative' CBN benchmark is fit to each agent's own judgments, so the headline normative-ordering claim needs reframing or additional validation.
major comments (3)
- [Results, CBN Model Fitting (Table 2)] The normative CBN benchmark is not independent of the agent being scored. In the 'CBN Model Fitting' section, the parameters θ_M are 'treated as free parameters and fit to the data' (per subject for humans, per condition for LLMs), and the joint distribution is defined as p(C1,C2,E)=p(E|C1,C2)p(C1)p(C2), which builds in the independence of the two causes. Consequently, the R and loss values in Table 2 measure how well an independence-assuming logistic model can reproduce an agent's judgments, not how close those judgments are to a fixed normative target. The flat independence lines in Figure 2c for GPT-4o, Gemini-Pro, and Claude make their high fit scores largely a restatement of that flatness, while humans' Markov violations guarantee lower fit on those tasks. The abstract's 'more normatively aligned' and the Discussion's claim that these LLMs 'exhibited more accurate causal reasoning than the humans' therefore need support from a benchmark whose parameters are not fit to the agent being evaluated, or the claim should be weakened to 'more consistent with the CBN family.' The direct Figure 2c result remains valid evidence about associative bias. Table 3 inherits the same issue because its baseline is the same fitted 3-parameter CBN.
- [Results, Effect-Absent Diagnostic Inference (Fig. 2e) / Discussion] The treatment of GPT-4o's very low effect-absent ratings as normative relies on an unstated deterministic assumption. The text notes that 'if the causal relations are assumed to be deterministically sufficient and necessary' then p(C1=1|E=0) should be zero, but the cover stories only say that one variable 'causes' another; they do not state deterministic sufficiency or necessity. The fitted logistic CBN with finite causal strengths also predicts positive probabilities for the cause when the effect is absent. Thus, calling GPT-4o 'most consistent with deterministically sufficient causal relations' (Results) and 'aligned most closely with normative inferences' (Discussion) is not a self-contained normative result; it depends on a particular interpretation of the causal mechanism that the prompt does not specify. This matters because the effect-absent tasks are included in the overall CBN fit that supports the ordering of agents.
- [Results, Comparison of LLMs and Humans (Fig. 2c)] The central behavioral claim that GPT-4o, Gemini-Pro, and Claude 'reasoned normatively' by respecting the independence of causes is supported only by visual inspection. The text states that they assigned uniform likelihoods 'regardless of the status of the alternative cause,' but no slope estimate, equivalence test, or statistical comparison with the human slope is reported, despite the bootstrapped confidence intervals shown in Figure 2. Because this flat-line result is the main non-circular evidence for the paper's headline conclusion, the authors should quantify it and test the human-LLM difference.
minor comments (4)
- [Procedure / Footnote 3] The procedural differences between humans (slider defaulting to 50, two-phase learning, graph visible) and LLMs (single long prompt, open numeric response) are acknowledged only in a footnote; these differences could affect response range and flatness, and deserve a fuller treatment in the main text.
- [Table 2] The table reports averaged R values without confidence intervals or a note on how averaging over subjects versus conditions affects the comparison; given that R is a correlation and each fit uses only 11 tasks per condition or subject, these estimates are quite uncertain.
- [Results, Comparison of LLMs and Humans] The statement that 'all LLMs except GPT-3.5 provided sensible judgments for all inference tasks' is too strong, since Claude-3 shows a negative slope from Task VI to VII in Fig. 2d, which violates explaining away proper; consider softening to 'provided sensible judgments for most tasks.'
- [Discussion] There is a typo in the Discussion: 'associative bias that plagues human casual reasoning' should be 'causal reasoning.'
Circularity Check
The 'more normatively aligned than humans' claim rests on fitting a CBN to each agent's own judgments and then treating that fitted model as the normative benchmark; the resulting fit scores partly reduce to the model's built-in independence assumption.
-
fitted input called prediction
[Results: CBN Model Fitting; Table 2 and Discussion]
"Since agents received only verbal descriptions, the CBN’s parameters θ_M were treated as free parameters and fit to the data. ... The correlations between the observed judgments and those predicted by the fitted CBNs were substantial ... if CBNs are accepted as the normative standard, these LLMs exhibited more accurate causal reasoning than the humans."
The normative benchmark is not independent of the data being scored. The CBN's parameters are fit to each agent's own judgments (per subject for humans, per condition for LLMs), and the R values in Table 2 are correlations between those judgments and the fitted model's predictions of the same judgments. The model's factorization p(C1,C2,E)=p(E|C1,C2)p(C1)p(C2) hard-codes the independence of causes, which is precisely the normative principle used to diagnose associative bias. Thus LLMs that produce flat independence lines (Fig. 2c) fit this model well by construction, while humans with Markov violations fit worse.
full rationale
The paper's central empirical finding that GPT-4o, Gemini-Pro, and Claude do not exhibit the associative bias is directly supported by Figure 2c, which shows flat judgments across the independence-of-causes tasks; that evidence is not circular. However, the broader and load-bearing claim that these models are 'more normatively aligned' than humans is quantified through CBN model fitting in which the CBN's parameters are free and fit to each agent's own judgments. The same fitted model is then used as the 'normative standard' (Discussion: 'we compared LLMs to the normative inferences of fitted CBNs'), and the R and loss values in Table 2 measure self-consistency with the model family rather than distance to a fixed normative target. Because the CBN family enforces p(C1,C2)=p(C1)p(C2), high fit is almost guaranteed for agents whose judgments are flat on the independence tasks, so the model-fitting comparison partly encodes the very conclusion it is used to support. The mutation-sampler comparison inherits this issue because its null model is the same fitted CBN and its chain-length parameter is also fit to each dataset. No other circular steps were found: the human data are empirical, and the self-citations (Rehder & Waldmann 2017; Davis & Rehder 2020) are not the sole justification for the main empirical patterns. Overall, partial circularity in the normative-superiority argument warrants a score of 6 rather than higher, since an independent non-circular result (flat independence lines) does support part of the claim.
Assumptions & free parameters
free parameters (4)
- w_C (cause prior probability) =
0.436 to 0.897 across agents (Table 2)
- w_C,E or w_C1,E/w_C2,E (causal strength) =
e.g., 1.55 (Gemini), 1.66 (GPT-4o), 0.60 (GPT-3.5), 1.06 (Humans) for 3-parameter model (Table 2)
- w_E (exogenous influence on effect) =
1.76 to 1.97 across agents (Table 2)
- lambda (mutation sampler chain length) =
3.7 (humans), 26.0 (GPT-4o), 38.6 (Gemini), 37.9 (GPT-3.5), 43.9 (Claude) (Table 3)
assumptions (4)
- domain assumption Causal Bayes net structure with independent causes: p(C1,C2,E)=p(E|C1,C2)p(C1)p(C2)
- domain assumption Logistic generating function p(E=1|C1,C2)=1/(1+exp(-(C1 w_C1,E + C2 w_C2,E + w_E)))
- domain assumption LLM judgments at temperature 0.0 are deterministic and representative of the model's reasoning
- domain assumption Human data from Rehder and Waldmann (2017) is directly comparable to LLM judgments despite procedural differences (learning phase vs. single prompt, slider vs. number)
Cite this review
Pith. "Pith review of Do Large Language Models Reason Causally Like Us? Even Better?." pith.science (2026). https://pith.science/paper/OF65BMU5
@misc{pith2026250210215,
author = {Pith},
title = {Pith review of: Do Large Language Models Reason Causally Like Us? Even Better?},
year = {2026},
howpublished = {\url{https://pith.science/paper/OF65BMU5}},
note = {Machine review of arXiv:2502.10215}
}
read the original abstract
Causal reasoning is a core component of intelligence. Large language models (LLMs) have shown impressive capabilities in generating human-like text, raising questions about whether their responses reflect true understanding or statistical patterns. We compared causal reasoning in humans and four LLMs using tasks based on collider graphs, rating the likelihood of a query variable occurring given evidence from other variables. LLMs' causal inferences ranged from often nonsensical (GPT-3.5) to human-like to often more normatively aligned than those of humans (GPT-4o, Gemini-Pro, and Claude). Computational model fitting showed that one reason for GPT-4o, Gemini-Pro, and Claude's superior performance is they didn't exhibit the "associative bias" that plagues human causal reasoning. Nevertheless, even these LLMs did not fully capture subtler reasoning patterns associated with collider graphs, such as "explaining away".
Figures
Reference graph
Works this paper leans on
-
[1]
abdin2024phi APACrefauthors Abdin, M. , Aneja, J. , Awadalla, H. , Awadallah, A. , Awan, A A. , Bach, N. others APACrefauthors \ 2024 . Phi-3 technical report: A highly capable language model locally on your phone, 2024 Phi-3 technical report: A highly capable language model locally on your phone, 2024 . URL https://arxiv. org/abs/2404.14219
arXiv 2024
-
[2]
bottou2023borges APACrefauthors Bottou, L. \ Sch \"o lkopf, B. APACrefauthors \ 2023 . Borges and AI Borges and AI . arXiv preprint arXiv:2310.01425
arXiv 2023
-
[3]
bramley2015conservative APACrefauthors Bramley, N R. , Lagnado, D A. \ Speekenbrink, M. APACrefauthors \ 2015 . Conservative forgetful scholars: How people learn causal structure through sequences of interventions. Conservative forgetful scholars: How people learn causal structure through sequences of interventions. Journal of Experimental Psychology: Lea...
work page 2015
-
[4]
bubeck2023sparks APACrefauthors Bubeck, S. , Chandrasekaran, V. , Eldan, R. , Gehrke, J. , Horvitz, E. , Kamar, E. others APACrefauthors \ 2023 . Sparks of artificial general intelligence: Early experiments with GPT -4 Sparks of artificial general intelligence: Early experiments with GPT -4 . arXiv preprint arXiv:2303.12712
arXiv 2023
-
[5]
cheng1997covariation APACrefauthors Cheng, P W. APACrefauthors \ 1997 . From covariation to causation: A causal power theory. From covariation to causation: A causal power theory. Psychological review 104 2 367
work page 1997
-
[6]
cruz2020explainingaway APACrefauthors Cruz, N. , Hahn, U. , Fenton, N. \ Lagnado, D. APACrefauthors \ 2020 . Explaining away, augmentation, and the assumption of independence Explaining away, augmentation, and the assumption of independence . Frontiers in Psychology 11 502751
work page 2020
-
[7]
davis2020mutation APACrefauthors Davis, Z J. \ Rehder, B. APACrefauthors \ 2020 . A process model of causal reasoning A process model of causal reasoning . Cognitive Science 44 e12839
work page 2020
-
[8]
glymour2003learning APACrefauthors Glymour, C. APACrefauthors \ 2003 . Learning, prediction and causal Bayes nets Learning, prediction and causal bayes nets . Trends in cognitive sciences 7 1 43--48
work page 2003
Show all 29 references
-
[9]
, Wang, Z
gunter2024apple APACrefauthors Gunter, T. , Wang, Z. , Wang, C. , Pang, R. , Narayanan, A. , Zhang, A. others APACrefauthors \ 2024 . Apple intelligence foundation language models Apple intelligence foundation language models . arXiv preprint arXiv:2407.21075
2024
-
[10]
, Xie, Y
jiang2024peek APACrefauthors Jiang, B. , Xie, Y. , Hao, Z. , Wang, X. , Mallick, T. , Su, W J. Roth, D. APACrefauthors \ 2024 . A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners A peek into token bias: Large language models are not yet genuine reasone...
2024 arXiv
-
[11]
, Chen, Y
jin2023cladder APACrefauthors Jin, Z. , Chen, Y. , Leeb, F. , Gresele, L. , Kamal, O. , Lyu, Z. others APACrefauthors \ 2023 . Cladder: A benchmark to assess causal reasoning capabilities of language models Cladder: A benchmark to assess causal reasoning capabilities of langua...
2023 arXiv
-
[12]
\ Busemeyer, J R
johnson2016computational APACrefauthors Johnson, J G. \ Busemeyer, J R. APACrefauthors \ 2016 . A computational model of the attention process in risky choice. A computational model of the attention process in risky choice. Decision 3 4 254--280
2016
-
[13]
u gelgen, J. , Priesemann, V. \ Sch \
kekiC2023evaluating APACrefauthors Keki \'c , A. , Dehning, J. , Gresele, L. , von K \"u gelgen, J. , Priesemann, V. \ Sch \"o lkopf, B. APACrefauthors \ 2023 . Evaluating vaccine allocation strategies using simulation-assisted causal modeling Evaluating vaccine allocation str...
2023
-
[14]
, Willig, M
keshmirian2024biased APACrefauthors Keshmirian, A. , Willig, M. , Hemmatian, B. , Hahn, U. , Kersting, K. \ Gerstenberg, T. APACrefauthors \ 2024 . Biased Causal Strength Judgments in Humans and Large Language Models Biased causal strength judgments in humans and large languag...
2024
-
[15]
, Ness, R
kiciman2023causal APACrefauthors K c man, E. , Ness, R. , Sharma, A. \ Tan, C. APACrefauthors \ 2023 . Causal reasoning and large language models: Opening a new frontier for causality Causal reasoning and large language models: Opening a new frontier for causality . arXiv prep...
2023 arXiv
-
[16]
, Ullman, T D
lake2017building APACrefauthors Lake, B M. , Ullman, T D. , Tenenbaum, J B. \ Gershman, S J. APACrefauthors \ 2017 . Building machines that learn and think like people Building machines that learn and think like people . Behavioral and Brain Sciences 40
2017
-
[17]
, Griffiths, T
lieder2012burn APACrefauthors Lieder, F. , Griffiths, T. \ Goodman, N. APACrefauthors \ 2012 . Burn-in, bias, and the rationality of anchoring Burn-in, bias, and the rationality of anchoring . Advances in neural information processing systems Advances in neural information pro...
2012
-
[18]
, Alizadeh, K
mirzadeh2024gsm APACrefauthors Mirzadeh, I. , Alizadeh, K. , Shahrokhi, H. , Tuzel, O. , Bengio, S. \ Farajtabar, M. APACrefauthors \ 2024 . Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models Gsm-symbolic: Understanding the limitatio...
2024 arXiv
-
[19]
\ Krakauer, D C
mitchell2023debate APACrefauthors Mitchell, M. \ Krakauer, D C. APACrefauthors \ 2023 . The debate over understanding in AI's large language models The debate over understanding in AI's large language models . Proceedings of the National Academy of Sciences 120 13 e2215907120
2023
-
[20]
\ Larrick, R P
morris1995one APACrefauthors Morris, M W. \ Larrick, R P. APACrefauthors \ 1995 . When one cause casts doubt on another: A normative analysis of discounting in causal attribution. When one cause casts doubt on another: A normative analysis of discounting in causal attribution....
1995
-
[21]
, King, N
nori2023capabilities APACrefauthors Nori, H. , King, N. , McKinney, S M. , Carignan, D. \ Horvitz, E. APACrefauthors \ 2023 . Capabilities of GPT -4 on medical challenge problems Capabilities of GPT -4 on medical challenge problems . arXiv preprint arXiv:2303.13375
2023 arXiv
-
[22]
APACrefauthors \ 2023
obaid2023machine APACrefauthors Obaid, O I. APACrefauthors \ 2023 . From machine learning to artificial general intelligence: A roadmap and implications From machine learning to artificial general intelligence: A roadmap and implications . Mesopotamian Journal of Big Data 2023 81--91
2023
-
[23]
APACrefauthors \ 1995
pearl1995bayesian APACrefauthors Pearl, J. APACrefauthors \ 1995 . From Bayesian networks to causal networks From bayesian networks to causal networks . Mathematical models for handling partial knowledge in artificial intelligence Mathematical models for handling partial knowl...
1995
-
[24]
APACrefauthors \ 2024
rehder2024inhibitory APACrefauthors Rehder, B. APACrefauthors \ 2024 . Extending a Rational Process Model of Causal Reasoning: Assessing Markov Violations and Explaining Away with Inhibitory Causal Relations Extending a rational process model of causal reasoning: Assessing mar...
2024
-
[25]
\ Waldmann, M R
rehder2017failures APACrefauthors Rehder, B. \ Waldmann, M R. APACrefauthors \ 2017 . Failures of explaining away and screening off in described versus experienced causal learning scenarios Failures of explaining away and screening off in described versus experienced causal le...
2017
-
[26]
, Chen, X
pmlr-v202-shi23a APACrefauthors Shi, F. , Chen, X. , Misra, K. , Scales, N. , Dohan, D. , Chi, E H. Zhou, D. APACrefauthors \ 2023 23--29 Jul . Large Language Models Can Be Easily Distracted by Irrelevant Context Large language models can be easily distracted by irrelevant con...
2023
-
[27]
, Goodman, N
vul2014one APACrefauthors Vul, E. , Goodman, N. , Griffiths, T L. \ Tenenbaum, J B. APACrefauthors \ 2014 . One and done? Optimal decisions from very few samples One and done? optimal decisions from very few samples . Cognitive Science 38 4 599--637
2014
-
[28]
, Hagmayer, Y
waldmann2006beyond APACrefauthors Waldmann, M R. , Hagmayer, Y. \ Blaisdell, A P. APACrefauthors \ 2006 . Beyond the information given: Causal models in learning and reasoning Beyond the information given: Causal models in learning and reasoning . Current Directions in Psychol...
2006
-
[29]
, Zecevic, M
willig2023causal APACrefauthors Willig, M. , Zecevic, M. , Dhami, D S. \ Kersting, K. APACrefauthors \ 2023 . Causal parrots: Large language models may talk causality but are not causal Causal parrots: Large language models may talk causality but are not causal . arXiv preprin...
2023 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.