{"id":"da3b1d34-202a-44de-8632-c9500ceead3e","arxiv_id":"2602.10282","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs asked to fill in linear-Gaussian causal equations give inaccurate, unstable, and perturbation-sensitive coefficients; the open-source Linear-LLM-SCM benchmark measures this, with Gemini 2.5 Flash leading on scale-free metrics.","lead":"This paper tests whether AI chatbots can do the quantitative part of causal reasoning: given a diagram showing how variables affect each other, guess the strength (the numbers) of each effect. Across seven real-world datasets the models were often wrong, unstable (same question, different numbers), and distracted by small diagram changes - a warning for using them in medicine or policy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth validity and the Algorithm 2 acceptance gate: BnRep fitted coefficients are treated as causal truth while author-crafted ranges filter proposals; if the GT itself violates the crafted ranges, M1–M4 measure constraint compliance, not causal parameterization quality.","rationale":"Among the candidate concerns, the validity of the reference parameters and the acceptance gate is more load-bearing than missing baselines or possible memorization because it attacks the yardstick itself. RQ1 is framed as comparison to 'real-world ground truths'; if the ground truth is a fitted BN that the framework's own constraints would reject, then the numbers in Tables 3–5 cannot be read as evidence about causal parameterization. The reader's weakest assumption already identifies this. Memorization, by contrast, would inflate apparent accuracy, so it cannot explain the reported failures; a chance baseline would calibrate absolute M2/M4 but would not overturn the qualitative variability finding. Thus the crafted-range/GT compatibility issue is the right single check. My recommendation stays CONDITIONAL: the central descriptive finding is plausible and the framework is honestly scoped, but this one unresolved compatibility question limits the interpretation of the quantitative results.","tokens_in":15783,"tokens_out":7494,"duration_ms":91643,"concrete_test":"Recompute Algorithm 2's C1 for every node of the expenditure DAG using the BnRep ground-truth coefficients and the author-crafted ranges in Table 2; check whether C2 contains C1. If any GT-consistent proposal would be rejected, quantify the effect on M1–M4 by rerunning the benchmark with the acceptance gate disabled (or with ranges derived from the source literature). A material change in M2/M4 would show the 'limitations' conclusion is partly an artifact of the crafted ranges.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs are poor quantitative causal parameterizers requires that the BnRep reference coefficients are trustworthy causal effect sizes and that the pipeline's acceptance gate does not distort the comparison. Section 4.2 adopts BnRep's linear-Gaussian parameters as ground truth without establishing causal provenance; several sources (e.g., expenditure from Tsagris 2022) are learned predictive networks, not necessarily causal graphs. More concretely, Algorithm 2 rejects any proposal whose implied value range C1 is not contained in the node's hard range C2, and for the expenditure DAG these ranges were 'Crafted' by the authors (Table 2, VR column). If the BnRep ground-truth coefficients themselves imply C1 values that fall outside C2, then the acceptance gate would reject the very parameter vector used as ground truth, and M1–M4 would penalize LLMs for not reproducing an inadmissible parameterization. The paper does not report this compatibility check, so a portion of the measured 'failure' could reflect the crafted constraints rather than LLM inability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Linear-LLM-SCM, a framework for eliciting linear-Gaussian structural equation coefficients from LLMs given a DAG, variable descriptions, units, and hard value-range constraints. The framework decomposes the DAG into local parent-child prompts, optionally refines proposals via an iterative acceptance gate (Algorithm 2), and compares LLM outputs against ground-truth parameters from the BnRep repository using four metrics M1–M4. Experiments on seven DAGs with three LLMs (Gemini 2.5 Flash, Llama 3.1 8B, Llama 3.3 70B) are reported for direct estimation (RQ1), unit-tweak robustness (RQ2-I), and spurious-edge robustness (RQ2-II). The authors conclude that LLMs show high variability, scale-sensitive M1, moderate normalized distances, and degradation under structural misspecification, and they caution against deploying LLMs as quantitative causal parameterizers in safety-critical domains.","tokens_in":15983,"tokens_out":3864,"duration_ms":46098,"significance":"If the findings hold, the paper provides a useful, open-sourced benchmark for a relatively underexplored task—quantitative coefficient elicitation for continuous SCMs—and makes a reasonable case that scale-invariant metrics (M3/M4) are more meaningful than raw L2 distance. The framework is model- and DAG-agnostic, and the authors are honest about several limitations, including the linear assumption, limited unit-tweak coverage, and the possibility of non-determinism. The external anchoring of ground truth in BnRep avoids the circularity of self-generated derivations. However, the strength of the central claim depends on three unresolved issues: causal provenance of the BnRep parameters, whether the Algorithm 2 hard-constraint gate is compatible with the ground truth, and the absence of chance-level or statistical baselines for interpreting M2–M4.","major_comments":[{"comment":"Ground-truth validity: The BnRep DAGs are a heterogeneous collection; several entries, e.g., expenditure from Tsagris (2022), are learned predictive networks (FEDHC) rather than established causal graphs. Treating every fitted linear-Gaussian parameter as a causal ground truth conflates conditional association with causal effect size. Since the central claim is about LLMs as 'quantitative causal parameterizers', the benchmark should either restrict to DAGs with explicit causal provenance or add a caveat that it evaluates association-coefficient elicitation. A concrete check would be to compare a subset of coefficients against independent domain-expert values.","section":"Section 4.2, Table 2"},{"comment":"Compatibility of the acceptance gate with ground truth: For the expenditure DAG, value ranges are 'Crafted' by the authors, not taken from the literature. Algorithm 2 rejects any proposal whose implied range C1 is not contained in the node's hard range C2. If the BnRep ground-truth coefficients themselves imply C1 values outside C2, then the gate would reject the ground-truth parameter vector, and M1–M4 would penalize LLMs for not reproducing an inadmissible parameterization. The paper does not report whether the ground-truth coefficients pass this compatibility check. Without this, a nontrivial portion of the measured 'failure' may reflect the crafted constraints rather than LLM inability.","section":"Algorithm 2; Table 2 (VR column)"},{"comment":"Statistical support for degradation under spurious edges: The text states that adversarial conditions generally yield lower M4 values, but the 95% confidence intervals overlap substantially across conditions. For example, Gemini M4 is 7.520±0.200 for O and 7.320±0.271 for S4; Llama 3.3 M4 is 6.560±0.199 for O and 6.680±0.271 for S4. Overlapping intervals do not support a claim of performance degradation. Report paired significance tests, bootstrap difference distributions, or a non-overlap criterion before concluding that spurious edges degrade parameterization.","section":"Section 5.2.2, Table 5"},{"comment":"Missing chance-level baseline: M2–M4 have no null reference. M4 counts the number of nodes with exact effect-size ordering, but the expected value under random ordering depends on the number of parents per node; a score of 1.000 on cachexia1 may be far above chance or simply reflect a single multi-parent node with only two parents. Similarly, M2 and M3 values cannot be judged as 'poor' without knowing the distribution under random coefficients. Add a random-coefficient or permutation baseline for each DAG to calibrate the metrics and support the abstract's claim that LLMs are unreliable.","section":"Section 3.4, Table 3"},{"comment":"Data leakage is invoked but not tested: The authors cite Yang et al. (2023) as motivation for adversarial perturbations to mitigate data leakage, yet they never measure whether direct RQ1 scores are inflated by memorization of published BnRep networks. Given that the same networks appear in the literature, LLMs may be recalling coefficients rather than eliciting them. A control condition using permuted variable names/semantics or held-out DAGs would clarify whether the low accuracy reflects a genuine inability to reason about continuous effect sizes.","section":"Section 4.3, Section 5.1"}],"minor_comments":[{"comment":"The phrase 'seven real-world DAGs effect ground truth' is ungrammatical; should be 'seven real-world DAGs with effect ground truth'.","section":"Abstract"},{"comment":"Index notation in M4 is inconsistent: the surrounding text uses β_{LLM,j,i} and β_{GT,j,i}, but the equation uses β_{LLM,i,j} and β_{GT,i,j}. Unify the subscript order.","section":"Eq. (4)"},{"comment":"When the loop budget n is reached, the algorithm returns the last proposal even if it failed the validation check. The text does not state whether this is the last accepted proposal or simply the final sampled one; clarify to avoid ambiguity in interpreting rejected samples.","section":"Algorithm 2"},{"comment":"The 'VR' column uses 'N (Crafted)' for expenditure, but the fact that the ranges are author-crafted is not explained until the discussion of Algorithm 2. Move or annotate this earlier so readers do not mistake the ranges as literature-sourced.","section":"Table 2"},{"comment":"The interpretation of Llama 3.1 8B's 'determinism' would be stronger if the paper reported repeated runs with different seeds or API call configurations, since temperature=0 is not a guarantee of determinism in all serving stacks.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's open-source framework and external ground-truth anchoring are positive features, and the central claim is plausible. However, the missing compatibility check between the crafted constraints and ground truth, the lack of a chance-level baseline, and the overlapping confidence intervals in the robustness analysis are load-bearing for the abstract's conclusions. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper quantifies something most of us suspected — LLMs are not reliable at turning a DAG plus variable semantics into regression coefficients — and it ships an open, reusable framework to keep measuring that. That is a real contribution. The novelty is honest: the closest work uses sampling or discrete CPDs, so direct symbolic coefficient elicitation in continuous linear-Gaussian SCMs is genuinely new. The DAGs come from BnRep, so ground truth is external, and the M2–M4 metrics sensibly move beyond scale-dependent L2 distance. The robustness probes (spurious edges, unit change) are the right kind of stress test.\n\nThe soft spots are mostly about calibration, not about the core direction.\n\nFirst, there is no baseline. M2 values of 0.45–3.25 and M4 counts mean nothing until you compare against, say, random coefficients within the stated ranges or a heuristic like equal weights. The aggregate conclusion 'LLMs are poor' may survive such a baseline, but the paper doesn't show it. Given the claim is about limitation, this is a load-bearing omission.\n\nSecond, Algorithm 2's acceptance gate can distort what M1–M4 measure. For the expenditure DAG the value ranges were crafted by the authors. If those ranges are inconsistent with the BnRep fitted coefficients, the gate would reject the ground truth itself and the metrics would reward constraint compliance, not causal accuracy. The paper does not report a compatibility check. This needs to be settled.\n\nThird, some robustness conclusions lean on overlapping CIs. The paper acknowledges the Llama 3.3 70B case, but the conclusion then states degradation as a general result. Soften it or add data.\n\nFourth, the ground truth is fitted network parameters, not proven causal effects; calling them 'effect ground truth' is a stretch, especially for DAGs that originated as predictive models. Framing it as comparing against a published fitted model would be more accurate.\n\nFinally, data leakage is cited as motivation but never tested. If the BnRep numbers appear in training corpora, direct RQ1 results could be inflated; the perturbations help, but only partially.\n\nNone of this breaks the central descriptive finding, and the paper is honest about several limitations. It deserves a serious referee. I'd send it out, asking for a baseline, the compatibility check, and tightened conclusions.","headline":"Useful benchmark for a real gap, though the headline claim would be stronger with a baseline and a check that the acceptance gate doesn't reject the ground truth.","tokens_in":16587,"tokens_out":2425,"would_cite":true,"duration_ms":24544,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current LLMs, given only a DAG and variable semantics, cannot reliably estimate linear-Gaussian causal coefficients and quantifies the failure with new scale-invariant metrics.","keywords":["large language models","causal effect estimation","structural causal models","linear-Gaussian SCM","coefficient elicitation","benchmarking","DAG misspecification","robustness"],"falsifier":"Construct a synthetic linear-Gaussian SCM on a familiar DAG with coefficients that could not appear in any training corpus (e.g., random nonzero values), elicit coefficients from the same LLMs under the same prompting, and check whether M2/M3 and M4 stay poor. If the models approximate the held-out coefficients well, the reported failure is an artifact of memorization or constraint crafting; if they still fail, the limitation is robust.","tokens_in":15606,"feed_emoji":"📉","tokens_out":5124,"duration_ms":53687,"temperature":0.7,"pith_summary":"The paper is trying to establish whether large language models can perform quantitative causal reasoning: given a directed acyclic graph and plain-language descriptions of its variables, can an LLM produce the regression coefficients of a linear-Gaussian structural causal model without seeing any observational data? It builds a plug-and-play benchmark that asks the model for one equation per node, aggregates the answers, and compares them against ground-truth coefficients from published Gaussian Bayesian networks using four metrics ranging from raw L2 distance to relative-effect ordering. Across three LLMs and seven real-world DAGs, the estimates are highly variable, sensitive to unit changes, and degrade when spurious edges are added, leading the authors to conclude that current LLMs are unreliable as quantitative causal parameterizers. If the paper is right, LLM-based effect-size elicitation is not yet deployable in safety-critical settings, and progress should be tracked with scale-invariant metrics rather than raw coefficient error.","feed_headline":"LLMs can't reliably set effect sizes from a causal graph","feed_subtitle":"New benchmark measures coefficient accuracy on seven real-world graphs; estimates drift with units and spurious edges.","key_machinery":"The framework's load-bearing mechanism is node-level decomposition with an acceptance gate. The DAG is traversed in topological order; for each node, the LLM receives only the node's direct parents, their units, value ranges, and a phenomenon description, and must return a concrete linear equation with numeric coefficients plus an intercept. An iterative feedback loop (Algorithm 2) computes the implied range of the target from the parents' ranges and the proposal, and rejects proposals whose implied range falls outside the node's stated bounds, re-prompting up to five times. Results are scored by four metrics: M1 raw L2 distance, M2 L2 distance after normalizing each node's coefficient vecto","core_discovery":"The central claim is that LLMs, when given a pre-specified DAG and variable semantics, cannot reliably parameterize linear-Gaussian SCMs: the node-normalized error metric M2 ranges from roughly 0.45 to 3.25 across model-DAG pairs (far from the perfect score of 0), and the relative-ordering metric M4 falls when spurious edges are added, showing models fail to assign near-zero coefficients to irrelevant causes. The authors additionally report that two of the three tested LLMs remain stochastic even at temperature zero, producing non-deterministic coefficient estimates, while the smallest model is deterministic but less accurate. The claim is specifically about direct, symbolically elicited coe","pith_inferences":["The benchmark never measures training-data leakage directly; if the source papers for the seven DAGs appear in the models' training corpora, the direct RQ1 scores could be inflated by memorization, making actual zero-shot elicitation competence lower than reported.","At least one DAG's 'hard constraints' were crafted by the authors rather than taken from the literature; rerunning the framework with different plausible ranges would show whether the acceptance gate, not the model's causal knowledge, drives the scores.","The unit-tweak experiment found that changing micromolar to nanomolar sometimes improved metrics; a clean test would ask the same model for coefficients in both units and check whether the two answers differ by the correct factor, isolating arithmetic scaling from semantic reasoning.","The framework could be extended to elicit residual variances and confidence intervals, which the authors explicitly left out, turning the benchmark into a full distributional-parameterization test."],"forward_implications":["Scale-invariant metrics (M2–M4) should replace or supplement raw L2 distance in future LLM causal benchmarks, since M1 is dominated by variable ranges and misleads cross-DAG comparisons.","Predicting full causal effect sizes from prior knowledge alone is currently unsafe for healthcare or public-policy decisions; any LLM-elicited coefficients would need data grounding or strong uncertainty handling.","Adding spurious edges systematically lowers M4 and raises M2/M3, so downstream causal inference should treat DAG structure as uncertain rather than point-estimated.","Even with temperature set to zero, larger models show non-deterministic estimates, implying reproducibility cannot be assumed for LLM-based parameterization in deployment."],"fun_headline_variants":["LLMs flunk causal coefficient quiz","Benchmark: LLMs unreliable for causal effect sizes","LLMs can't pin down causal strength from DAG","Quantitative causal reasoning? LLMs miss the mark"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark treats the fitted linear-Gaussian coefficients in the source repository as exact ground truth and assumes the hard value ranges supplied (some handcrafted) agree with those fitted effects; if the ranges conflict or if the LLMs have memorized the source papers, the metrics measure agreement with a particular fitted network or retrieval, not causal reasoning.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flunk causal coefficient quiz","Benchmark: LLMs unreliable for causal effect sizes","LLMs can't pin down causal strength from DAG","Quantitative causal reasoning? LLMs miss the mark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1120,"prompt_tokens":709,"completion_tokens":411,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":453,"tokens_out":411,"duration_ms":4414,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T01:11:25.152093+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic linear-Gaussian SCM on a familiar DAG with coefficients that could not appear in any training corpus (e.g., random nonzero values), elicit coefficients from the same LLMs under the same prompting, and check whether M2/M3 and M4 stay poor. If the models approximate the held-out coefficients well, the reported failure is an artifact of memorization or constraint crafting; if they still fail, the limitation is robust.","supporting_citations":[],"review_version":1}