Pith. sign in

REVIEW 2 major objections 5 minor 13 references

Context, not pretraining mix, decides LLM causal direction

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:08 UTC pith:7FH335PF

load-bearing objection The inference-time suppression result is real and clean, but the training-mixture claim rests only on an OBS-only sweep, so the abstract overreaches until they test α under probe contexts. the 2 major comments →

arxiv 2607.29484 v1 pith:7FH335PF submitted 2026-07-31 cs.CL cs.LG

Evidence-Type Competition: When Can Interventional Data Teach Language Models Causal Direction?

Xining Xun This is my paper
classification cs.CL cs.LG
keywords causal directionlanguage modelsinterventional dataSimpson's paradoxevidence-type competitionactivation patchingin-context learningnoise floor
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests a common assumption: that adding more interventional data to a language model's pretraining improves its causal reasoning. In controlled synthetic worlds where the observed correlation and the true causal effect point in opposite directions, the training mixture has no effect on causal direction — the model learns the size of the effect but copies the sign from whatever evidence is present in the context at inference time. Purely observational contexts produce sign reversals in 29 of 50 worlds, mixed contexts in 19, and aligned interventional probes alone in only 4. Erasing the observational records from the context instantly restores the suppressed ability, so the causal knowledge lives in the weights and the context merely switches it on or off. If correct, this separates model capability from deployment behavior and redirects data-mixture optimization toward context engineering.

Core claim

On the paper's own terms, the discovery is a magnitude–direction duality: in a six-level sweep of the interventional training fraction, the model's Simpson-slope magnitude grows monotonically while its sign stays wrong throughout. What actually governs the sign is the evidence type at inference: under an identical training recipe, a purely observational context induces systematic sign reversal in 29/50 worlds, a mixed context in 19/50, while aligned interventional probes alone give 41/50 correct with reversals at the measurement noise floor. Erasing observational evidence from the context immediately releases the suppressed interpolation ability (ratio_true = +0.56, matching a probe-only mod

What carries the argument

The central object is a family of Simpson's-paradox structural causal models — worlds in which the observational correlation and the true causal effect of X on Y have opposite signs by construction — used both as training data and as evaluation probes. The carrying mechanism is the evidence-type competition: contexts contain observational records, interventional probes, or a mixture, and the paper measures a contextual dose gradient by varying the fraction of observational evidence at inference time. Two supporting instruments are a least-squares-interpolator baseline that quantifies the sampling noise floor of probe-based evaluation (26% sign errors single-shot, 9% after averaging eight con

Load-bearing premise

The mechanistic reversal counts assume the transformer's sign extraction is no noisier than a perfect least-squares interpolator on the same probe evidence; if the model is noisier than that simulated 9% floor, the 'mechanistic' reversal counts could be partly measurement noise.

What would settle it

Rerun the identical 50-world protocol on freshly generated Simpson worlds with the same checkpoint: if the probe-only arm produces more than about 10 reversals (clearly above the Bin(50, 0.09) floor) or the observational-only arm fails to produce a significant reversal rate, the evidence-type claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Interventional pretraining data is necessary but not sufficient: a probe-only training context yields causal direction, but any conflicting observational context can switch it off.
  • Inference-time context engineering can release already-learned causal ability without retraining: erasing observational records restores the suppressed slope to probe-only levels.
  • Evaluation protocols for causal reasoning in LLMs must control the evidence type in the context, because the same checkpoint can show 29/50 reversals under observational context and 4/50 under pure probes.
  • The suppression persists as a rate at 0.93B scale (31.8% vs 6% reversals in the probe-only arm), even though absolute gains contract four-fold.
  • Activation patching indicates the switch is distributed across the middle layers' observational rows, not a single circuit: the release is near-complete from layers 0–6 and vanishes exactly at the final layer.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the capability/switch picture generalizes to natural language, then standard causal-reasoning benchmarks may be measuring context evidence composition rather than causal competence; probe-only or conflict-free contexts could be a better test of what a model knows.
  • A direct design consequence: pretraining mixtures that never pair observational and interventional records in the same context might prevent the suppression from being learned at all, since the paper shows the shortcut deepens during training when both are present.
  • A testable extension: the E5 dose curve predicts a sharp transition between one and four observational records; checking whether a single mismatched record is sufficient to flip a model's answer in open-domain QA could translate the graded switch into a practical warning signal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper trains small GPT-style transformers on synthetic structural causal models (SCMs) with Simpson-trap worlds, where observational correlation and causal effect have opposite signs. It varies (i) the fraction α of interventional records in pretraining and (ii) the evidence type in the inference-time context (pure observational, mixed, pure probes). The main claim is that increasing α does not improve causal-direction sign accuracy; instead, the evidence type in the context at inference determines whether interventional evidence is used. The paper reports a four-arm comparison (Baseline, C1, C2, C2-curr2) on 50 worlds, inference-time erasure and content-manipulation experiments (E1, E5), scaling results at 0.93B, activation-patching localization, and an external audit on CLadder, along with a noise-floor analysis of probe-based evaluation.

Significance. If the central claim holds, it challenges the common intuition that adding interventional data to pretraining directly improves causal reasoning, and it reframes the problem as one of evidence selection at inference time. The paper's strengths include the fully controlled synthetic environment with computable ground truth, paired statistics on identical world sets, a detailed negative-result registry, reproducible code fingerprints, and a genuinely falsifiable sign-randomization experiment that is run and honestly reported (including its OOD failure). The noise-floor analysis is a useful methodological contribution, though its application requires calibration against the actual model as discussed below.

major comments (2)
  1. [§5.1 and Abstract] The claim that training mixture is not the operative variable is not established by the α sweep, because the sweep is performed only under a purely observational (OBS) context. In a Simpson-trap world, the OBS context does not contain the identifying interventional evidence, so a flat sign-accuracy curve across α is expected a priori regardless of whether the model can use interventional evidence. To support the headline generalization 'not the training mixture... but the evidence type in the context,' the paper must either run an α sweep under PROBE or MIXED contexts (e.g., α=0 vs. α=1 evaluated on pure probes) or explicitly restrict the claim to OBS-context evaluations. Without this, the abstract's 'What governs whether interventional evidence is used is not the training mixture' overclaims what the experiment can show.
  2. [§4, Eq. (5)–(6); Fig. 3] The reversal-count null is calibrated on a perfect least-squares interpolator, not on the transformer being tested. The statement that C2's 4/50 reversals 'sit on the null' and are 'measurement noise' (§5.2) assumes that the model's sign-extraction error rate equals that of the simulated perfect interpolator. This is not guaranteed: the model may be better or worse than the LS baseline at extracting signal from the same probe evidence. The correct calibration requires a model-specific noise floor, e.g., by resampling independent context draws for each world under C2 (or by using worlds whose causal sign is oracle-indeterminate) and measuring the model's own reversal rate. As written, the binomial test against p0=0.09 does not separate mechanistic reversal from sampling noise for this model, and the abstract's 'reversals at the measurement noise floor' is not supported.
minor comments (5)
  1. [§5.2] The paper uses 'paired t, p=5.6e-8' for comparing C2 vs. C1 slope ratios. Since ratio_true is likely non-normal and the paper already uses Wilcoxon signed-rank for other comparisons, either justify the t-test or replace it with a non-parametric statistic to avoid normality assumptions.
  2. [§5.6 vs. Appendix F] Appendix F states that C1 is reported under the in-distribution nprobe=2 protocol 'throughout (§5.2, §5.5)', but §5.6 reports E5 results with nprobe=4 on the C1 checkpoint. The relationship is confusing: please clarify whether E5 is deliberately run out-of-distribution and why the 'throughout' wording excludes §5.6, or update the cross-reference.
  3. [§5.3, E1b] E1b uses only 9 worlds (7/9 hit). This is a striking paired result, but the small n is not acknowledged in the main text; report a confidence interval or exact paired test result to quantify the uncertainty.
  4. [Throughout] Unify the notation for the ratio variable: 'ratio true' (with space) appears in Eq. (3) and in several places, while 'ratio_true' appears in abstracts and tables. Consistency would improve readability.
  5. [§5.5, Table 2] The row 'C2 1B' reports parameter count 0.93B in the text but the table simply says '1B'. Add a footnote or clarify in the caption that '1B' means 0.93B parameters.

Circularity Check

0 steps flagged

No significant circularity: the central claims are empirical, benchmarked against an external oracle floor, and include a falsified OOD prediction.

full rationale

The paper's main derivation chain is genuinely empirical rather than definitional. The Simpson-trap worlds are constructed with an independent ground truth: Eq. (1) defines the trap by requiring Corrobs(X,Y)*eff(X,Y)<0, but eff is computed by large-sample Monte Carlo, not by the model under test. The alpha-sweep and four-arm comparison are measured behaviors under fixed protocols, not parameters fitted to the target claim. The nearest candidate for circularity is the noise-floor argument (§4, Eqs. 5-6): p0=0.09 is obtained by simulating a perfect least-squares interpolator on the same generator, which is an external oracle rather than a fit to the transformer, and C2's 4/50 reversal count being 'on the null' is a statistical non-rejection, not an identity. The CLadder positive-effect prior is a measured consequence of single-signed training data, not an input assumption, and the sign-randomized retraining (§5.8) makes a pre-registered falsifiable prediction (b→0 with a intact) that the paper reports as failing (b=+0.21±0.02), which is the opposite of circular. The self-reported limitations (§7) explicitly flag external-validity gaps such as the OBS-only alpha sweep and the unknown mechanism of the OOD default; these are scope limitations, not definitional equivalences. There are no load-bearing self-citations or imported uniqueness theorems. The skeptic's point about the alpha sweep being OBS-only is an external-validity concern, not a circularity, and does not raise the score.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The paper's central claims rest on the SCM/do-calculus formalism, the Simpson-trap world construction, and measurement assumptions that LS slope over grid predictions is a faithful proxy for causal direction and that the perfect-interpolator noise floor is an upper bound. No new physical entities are introduced; the main free choices are evaluation thresholds (τ0=0.05) and evidence-averaging count (nrep=8), both disclosed.

free parameters (2)
  • hit magnitude gate τ0 = 0.05
    Chosen threshold in Eq. 4 determining hit vs flat vs reversed; reversal counts shift with this choice. Boundary-noise analysis for |slope|<0.15 acknowledges sensitivity, but no systematic sweep is reported.
  • evidence-averaging repetitions nrep = 8
    Chosen to reduce the simulated single-shot sign-error floor from 26% to 9% (§4, Appendix C); all final results use nrep=8. The floor p0=0.09 used in binomial tests depends on this choice.
axioms (6)
  • standard math SCM with do-calculus semantics: interventional records are drawn from P(V|do(V_j=v)) and identify the causal effect (Pearl 2009).
    Used throughout §3 to define worlds, evidence formats, and oracle slopes.
  • domain assumption Simpson-trap condition Corr_obs(X,Y) * eff(X→Y) < 0 with eff > 0 in 50/50 evaluation worlds.
    §3 Eq. 1 and the generator define the eval family; all hit/reversal counts assume the true effect is positive.
  • domain assumption Least-squares slope of the model's five grid predictions is a faithful measure of causal-direction competence.
    §3 Eq. 3 and hit criterion Eq. 4; the entire reversal analysis is based on this proxy.
  • domain assumption The perfect least-squares interpolator's sign-error rate (simulated from the oracle generator) is a valid noise floor / upper bound on measurement error for the trained model.
    §4 and Appendix C; C2's residual reversals are declared noise because they sit on this null.
  • domain assumption CLadder v1 CPT ground truth can be recomputed by exact enumeration and transplanted to the symbolic probe protocol.
    §5.7; agreement on 9/10 families, one exception in Appendix.
  • domain assumption Activation patching of residual-stream states in a small decoder causally localizes the context switch.
    Appendix H; G1 passes at row level but G2 sparse circuit fails, so only sufficiency is claimed.

pith-pipeline@v1.3.0-daily-deepseek · 12962 in / 19781 out tokens · 214787 ms · 2026-08-03T06:08:29.522605+00:00 · methodology

0 comments
read the original abstract

Interventional data is widely regarded as the gold standard for teaching models causal reasoning. We test this assumption in a fully controlled synthetic environment pitting observational correlation against causal effect, and find it fails instructively. In Simpson's-paradox worlds, where the two have systematically opposite signs, increasing the fraction of interventional samples in pretraining does not improve causal direction: the magnitude of the model's do()-response grows monotonically, yet its sign is copied from the observational context. What governs whether interventional evidence is used is not the training mixture but the evidence type present in the context at inference time. Under an identical training recipe, a purely observational context induces systematic sign reversal in 29/50 worlds, a mixed context in 19/50, while aligned interventional probes alone yield 41/50 correct. Erasing observational evidence from the context immediately releases the suppressed causal interpolation ability (ratio_true = +0.56); a four-state content manipulation shows the switch is content-mediated and graded. The suppression is stable across training seeds (11/11 strong reversals persist on a matched-protocol second seed) and robust as a rate at 0.93B parameters (31.8% vs. 6% reversals in the matched probe-only arm), even as absolute gains shrink four-fold. An external audit on CLadder exposes a learned positive-effect prior with a two-layer structure: sign-randomized retraining removes it in-distribution but not out-of-distribution. We summarize: the capability lives in the weights; the switch lives in the context, and activation patching localizes the switch to the middle layers' observational rows. We further quantify the sampling noise floor of probe-based causal evaluation and an evidence-averaging protocol that cuts sign errors from 26% to 9%.

Figures

Figures reproduced from arXiv: 2607.29484 by Xining Xun.

Figure 1
Figure 1. Figure 1: Overview (schematic). A: a hidden confounder makes observational correlation negative while the true effect of X on Y is positive (sampled world, ρ = 0.9; full tour in Appendix A). B: with observational records in context, the answer follows the observational sign (19/50 reversed); erasing them at inference time releases the suppressed interpolation (E1b: ratio true = +0.56); doubling the probe dose alone … view at source ↗
Figure 2
Figure 2. Figure 2: Main result. (a) Hit/flat/reversed counts over 50 worlds; (b) per-world [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Reversal counts vs. the noise floor: Binomial(50, 0.09) null with the four arms’ reversal [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: One-sided reading on the 50 CLadder confounding worlds. Blue circles: positive-ATE [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: One sampled evaluation world (ρ = 0.9), end to end. (a) Structure: the generator injects a confounded triple (Z, X, Y ) with X = ρA tanh(Z) + p 1 − ρ 2 U and Y = −ρB tanh(Z) + C tanh(X) + 0.3U; red dashed edges are the confounding paths, the green edge is the causal path. (b) An observational sample from the same world shows a strong negative trend (corr = −0.86). (c) Interventional probes do(X) on the sam… view at source ↗
Figure 6
Figure 6. Figure 6: (a) Release ratio R of zero-condition activation transplants, by layer and row type (s0 dev / s1 all-reversed; nrep = 4). The switch travels with the obs rows (R ≈ +1.0 from L0 through L6, both seeds) and vanishes exactly at L7; black boxes mark the preregistered L2/L3 obs cells. (b) Obs-directed attention mass by layer (s0): reading peaks at L7, where R = 0.00—reading ̸= leverage. 13 [PITH_FULL_IMAGE:fig… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 2 linked inside Pith

  1. [1]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky¨urek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. InThe Eleventh International Conference on Learning Representations (ICLR 2023),

    2023

  2. [4]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. InAdvances in Neural Information Processing Systems 35 (NeurIPS 2022),

    2022

  3. [6]

    Can large language models infer causation from correlation? InThe Twelfth International Conference on Learning Representations (ICLR 2024),

    Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Sch¨olkopf. Can large language models infer causation from correlation? InThe Twelfth International Conference on Learning Representations (ICLR 2024),

    2024

  4. [8]

    Passive learning of active causal strategies in agents and language models

    Andrew Lampinen, Stephanie Chan, Ishita Dasgupta, Andrew Nam, and Jane Wang. Passive learning of active causal strategies in agents and language models. InAdvances in Neural Information Processing Systems 36 (NeurIPS 2023),

    2023

  5. [9]

    Transformers handle endogeneity in in-context linear regression

    Haodong Liang, Krishnakumar Balasubramanian, and Lifeng Lai. Transformers handle endogeneity in in-context linear regression. InThe Thirteenth International Conference on Learning Represen- tations (ICLR 2025),

    2025

  6. [10]

    Entity-based knowledge conflicts in question answering

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021),

    2021

  7. [12]

    Teaching transformers causal reasoning through axiomatic training.arXiv preprint arXiv:2407.07612,

    Aniket Vashishtha, Abhinav Kumar, Atharva Pandey, Abbavaram Gowtham Reddy, Kabir Ahuja, Vineeth N Balasubramanian, and Amit Sharma. Teaching transformers causal reasoning through axiomatic training.arXiv preprint arXiv:2407.07612,

    arXiv

  8. [13]

    Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. InThe Twelfth International Conference on Learning Representations (ICLR 2024),

    2024

  9. [2009]

    Do-PFN: In-context learning for causal effect estimation

    Jake Robertson, Arik Reuter, Siyuan Guo, Noah Hollmann, Frank Hutter, and Bernhard Sch¨olkopf. Do-PFN: In-context learning for causal effect estimation. InAdvances in Neural Information Processing Systems 38 (NeurIPS 2025),

    2025

  10. [2022]

    CLadder: Assessing causal reasoning in language models

    9 Preprint Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojas Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Sch¨olkopf. CLadder: Assessing causal reasoning in language models. InAdvances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track,

    2023

  11. [2023]

    Cresswell, and Rahul G

    Vahid Balazadeh, Hamidreza Kamkari, Valentin Thomas, Benson Li, Junwei Ma, Jesse C. Cresswell, and Rahul G. Krishnan. Amortized causal effect estimation via in-context learning. InAdvances in Neural Information Processing Systems 38 (NeurIPS 2025),

    2025

  12. [2024]

    Causal reasoning and large language models: Opening a new frontier for causality.arXiv preprint arXiv:2305.00050,

    Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality.arXiv preprint arXiv:2305.00050,

    Pith/arXiv arXiv

  13. [2025]

    Ice cream doesn’t cause drowning: Benchmarking LLMs against statistical pitfalls in causal inference.arXiv preprint arXiv:2505.13770,

    Jin Du, Li Chen, Xun Xian, An Luo, Fangqiao Tian, Ganghua Wang, Charles Doss, Xiaotong Shen, and Jie Ding. Ice cream doesn’t cause drowning: Benchmarking LLMs against statistical pitfalls in causal inference.arXiv preprint arXiv:2505.13770,

    Pith/arXiv arXiv