{"id":"71ff4b24-d56b-4b62-9727-80d70c7a04cb","arxiv_id":"2411.15888","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Seven LLMs were evaluated on two new causal modeling tasks; GPT-4-turbo and Llama3-70b were best at distilling causal variables, Mixtral-8x22b was best at detecting interaction entities, and performance depended strongly on domain.","lead":"The paper tests whether large language models can help turn messy text descriptions of causes and effects into clean causal variables, and whether they can spot entities that belong to several variables at once. It compares seven LLMs across eight domains and finds no single model wins both tasks: GPT-4 and Llama3 group better, Mixtral detects interactions better.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed rankings and kappa values rest on LLM-generated labels with no expert gold standard; expert validation is needed before the central 'helpful for causal modeling' claim is supported.","rationale":"The reader identified as the weakest assumption that LLM-generated data serves as ground truth with no human expert validation. My stress-test concurs: this is the single most load-bearing assumption. The entire quantitative comparison—both the absolute claim of 'helpful' performance (kappa 40% and 60%) and the relative model rankings—is built on labels that the evaluated models themselves produced. The paper explicitly acknowledges this limitation, yet the abstract and conclusions state the findings as results rather than as exploratory measures of self-consistency. The proposed concrete test, expert annotation of a subsample, directly addresses this weakness and would decisively determine whether the rankings reflect genuine causal modeling capability or artifacts of self-generated data. If the authors cannot or do not supply such external validation, the central claim is unverified. The reader's verdict of REJECT is consistent with this assessment: the task formulation is valuable, but the quantitative evidence is not sound. I do not see another concern that is more load-bearing; missing confidence intervals and significance tests are secondary, and the domain-dependency observation, while plausible, suffers from the same ground-truth problem. Thus I agree with the reader and recommend no change to the verdict.","tokens_in":12506,"tokens_out":2337,"duration_ms":21698,"concrete_test":"Draw a random sample of, say, 100 positive and 100 negative examples per task from the generated evaluation set used for Tables 1 and 2. Have two or more domain experts independently label each example according to the task definitions, with disagreements resolved (e.g., by a third expert). Use the expert labels as gold standard, then recompute precision, recall, F1, and Cohen's kappa for every LLM on this subset. If the model ordering under expert labels differs from Tables 1 and 2—especially if Mixtral-8×22b is no longer best on Task 2 or GPT-4-turbo/Llama3-70b no longer best on Task 1—the central rankings collapse. The test is decisive because it breaks the circularity by introducing an external reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that LLMs are helpful for causal modeling, and the specific model rankings in Tables 1 and 2, depend entirely on labels constructed from data generated by the same LLMs under evaluation. In Section 3.2 (Figure 2), each model generates causal variables, values, and interaction entities; the authors then sample these into positive and negative examples, and each model evaluates all examples. No human expert or externally validated causal graph is used as ground truth. The paper's own Limitations section states: 'these exact numbers have been derived based on generated data.' This creates a circularity: precision, recall, and Cohen's kappa measure agreement with a machine-generated labeling schema, not correspondence with true causal variable structure. Systematic generation errors—for instance, a model producing semantically similar but distinct entities and labeling them as values of the same variable—would inflate scores for that model, and the cosine-similarity analyses in Figures 4 and 6 suggest that models may indeed be exploiting surface similarity rather than causal semantics. Consequently, the reported superior performance of GPT-4-turbo and Llama3-70b on Task 1 and Mixtral-8×22b on Task 2 is not established as true causal modeling ability; it may reflect self-consistency with each model's own generated ontology.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two tasks for evaluating LLMs' ability to support causal modeling: (Task 1) deciding whether two textual entities are values of the same causal variable, and (Task 2) deciding whether a textual entity represents an interaction of multiple given causal variables. For each task, seven LLMs generate candidate data (causal variables, values, interaction entities) in several domains, the generated data is sampled into positive and negative examples, and all LLMs classify all examples. Performance is measured by precision, recall, F1, and Cohen's kappa, with additional cosine-similarity analysis. The authors claim that GPT-4-turbo and Llama3-70b perform best on Task 1, while Mixtral-8×22b performs best on Task 2, and conclude that LLMs are helpful collaborators for human causal modelers.","tokens_in":12711,"tokens_out":2184,"duration_ms":20337,"significance":"If the central claims were established, the paper would provide a useful step toward benchmarking LLMs for a relatively underexplored component of causal analysis: transforming domain knowledge into causal variables and interaction entities. The task formulations themselves are interesting and connect to established guidelines from causal data science (Suzuki et al., McDonnell, Kuorikoski). However, the empirical evaluation is built on machine-generated labels with no expert-validated gold standard, so the numerical findings and model rankings are not currently supported. The paper does transparently acknowledge this limitation, which is a point in its favor, but the acknowledgement does not repair the circularity.","major_comments":[{"comment":"The evaluation is circular: the ground-truth labels for both tasks are derived from data generated by the very LLMs being evaluated. Section 3.2 (Figure 2) states that each LLM generates causal variables, values, and interaction entities, and these outputs are sampled into positive and negative examples; the same LLMs then classify those examples. The Limitations section concedes that 'these exact numbers have been derived based on generated data' and that the findings need to be compared with domain-specific experts. Consequently, the precision, recall, and kappa values in Tables 1-4 measure agreement with a machine-generated labeling scheme, not correctness with respect to true causal variable structure. The abstract's claim that 'contemporary LLMs are helpful tools' for causal modeling, and the specific rankings (e.g., Table 1: GPT-4-turbo and Llama3-70b with kappa 40% and 39%; Table 2: Mixtral-8×22b with kappa 60%), are not established by the presented evidence.","section":"Section 3.2 and Limitations"},{"comment":"The reported results are single-point estimates with no measures of uncertainty. Each kappa, precision, and recall value appears to be computed once over a pooled dataset, yet the experiment includes multiple domains and multiple generating models. There is no confidence interval, bootstrap, or significance test. For example, in Table 2, Mixtral-8×22b (kappa 60%) and Llama3-70b (kappa 47%) differ by 13 percentage points, but without variance information it is impossible to know whether this difference is meaningful. The paper should either provide per-domain and per-generator breakdowns with uncertainty or refrain from claiming that one model 'stands out' over others.","section":"Tables 1-4"},{"comment":"The cosine-similarity analyses raise a specific confound that is acknowledged but not resolved. Figures 4 and 6 show that several models' predictions align most strongly with classifying entities as belonging to the same variable (Task 1) or as interaction entities (Task 2) when their embedding cosine similarity is high. This suggests that models may be exploiting surface similarity rather than causal semantics. The paper notes this trend for weaker models but does not control for it in the main kappa-based comparison. Without such a control, the superior kappa values of the top performers could reflect better alignment with the (machine-generated) labels rather than genuine causal modeling ability.","section":"Figures 3-6 and Section 4"}],"minor_comments":[{"comment":"The list of domains in Section 4 includes 'Sociology' but the preceding sentence lists '(i) Computer Science ... and (vii) Urban Studies'; the count and the labels are inconsistent. Also, Table 3 and Table 4 omit columns for GPT-3.5-turbo, Mixtral-8×7b, and Mistral-7b without explanation, even though these models are part of the main results.","section":"Section 4"},{"comment":"The author affiliation on the title page reads 'Université Bourgogne Franche-Comté, Besançon, Franc' — the final word should be 'France'.","section":"Abstract and affiliation"},{"comment":"The Discussion interprets the higher precision than recall as potentially caused by Temperature=0, but this is speculative and not supported by any comparison with nonzero temperature. A sentence acknowledging this as an untested hypothesis would be more accurate.","section":"Section 5"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical contribution rests on an evaluation design in which the ground truth is generated by the same models under test. The authors themselves flag the need for expert comparison in the Limitations section, which is honest but also confirms that the reported numbers cannot be interpreted as evidence of causal modeling skill. Fixing this would require a new evaluation protocol with externally validated labels (e.g., expert-constructed causal variables and interaction entities), which is outside the scope of a revision. The task definitions and framing are a reasonable starting point for future work, but as it stands the manuscript does not support its abstract-level claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know: the paper defines two genuinely new evaluation tasks for LLM-assisted causal modeling, and its headline rankings are not supported because the labels come from the same models being tested. The tasks are the real contribution; the numbers are a proof of concept.\n\nWhat is new: Task 1 asks whether two text entities are values of the same causal variable; Task 2 asks whether an entity is an interaction of given causal variables. Both are natural and previously untested steps in building causal models from text, and the paper grounds them in the causal modeling literature (Suzuki, McDonnell, Kuorikoski). The experimental design is coherent: seven models, eight domains, zero-shot, deterministic sampling, and the authors include a cosine-similarity analysis to probe when predictions track surface semantics. Code and data are released, which helps.\n\nThe soft spot is the one the authors admit in the Limitations: 'these exact numbers have been derived based on generated data.' Every model generates the variables and interaction entities, those generations are sampled into positives and negatives, and the same models then classify them. There is no expert-validated gold standard. So precision, recall, and kappa measure agreement with machine-generated labels, not correctness against a known causal structure. The ranking of GPT-4-turbo/Llama3 on Task 1 and Mixtral on Task 2 could just reflect self-consistency. The cosine-similarity plots reinforce this: several models reach peak agreement at high similarity thresholds, suggesting they are doing surface matching. The paper also reports single numbers without confidence intervals or significance tests, so the differences between models are not statistically grounded.\n\nI would not call this a fatal flaw; the framing is transparent and the tasks are worth keeping. But the central claim that some LLMs are 'helpful tools' for causal modeling, and the specific model rankings, need external validation before they can be used. A serious referee should ask for a human/domain-expert labeled subset for each domain, at least a few hundred items, plus variance across repeated sampling.\n\nWho is this for: people building LLM-assisted pipelines for causal modeling, and benchmark designers. The paper deserves a round of peer review. Desk rejection would be wrong, because the tasks are novel and the authors are honest about the limitation. Send it to review with the expectation that the empirical section gets reworked.","headline":"Novel tasks, under-supported numbers: the labels come from the same LLMs being evaluated.","tokens_in":13221,"tokens_out":2110,"would_cite":false,"duration_ms":19414,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models can assist human causal modelers, but the best model depends on the task: GPT-4-turbo and Llama3-70b lead at distilling text into causal variables, while Mixtral-8×22b is best at detecting interaction entities.","keywords":["causal modeling","large language models","causal variables","interaction entities","Cohen's kappa","zero-shot evaluation","domain dependency","causal data science"],"falsifier":"Have two or more human domain experts independently annotate a held-out sample of entity pairs (Task 1) and interaction-entity triples (Task 2) across the same eight domains, then run the same zero-shot prompts and compute Cohen's $\\kappa$ against the human labels; if the absolute values or the model ranking change materially, the paper's conclusion that these LLMs are the helpful tools described here is not supported.","tokens_in":12287,"feed_emoji":"🧠","tokens_out":5682,"duration_ms":44573,"temperature":0.7,"pith_summary":"This paper asks whether off-the-shelf large language models can do the early, human-heavy work of causal modeling: turning free-text domain knowledge into variables and spotting entities that combine several variables at once. It introduces two binary tasks, evaluates seven LLMs across eight domains in a zero-shot setting, and reports that LLMs are helpful but far from conclusive. The best agreement is a Cohen's $\\kappa$ of about 0.40 for grouping entities under one causal variable and about 0.60 for detecting interaction entities, with task-specific winners: GPT-4-turbo and Llama3-70b lead the first task, while the sparse-expert Mixtral-8$\\times$22b leads the second. If the results hold, LLMs could accelerate and broaden human expert causal modeling rather than replace it.","feed_headline":"LLMs can help build causal models—pick the model by task","feed_subtitle":"GPT-4-turbo and Llama3-70b lead at variable distillation; Mixtral-8×22b wins at spotting interaction entities.","key_machinery":"The load-bearing objects are two binary classification tasks defined in the paper. Task 1, distilling causal domain knowledge into causal variables, asks whether two text entities are different values of the same causal variable (e.g., 'Insomniac' and 'Sound sleeper' both belong to 'Sleep quality'). Task 2, detecting interaction entities, asks whether one text entity represents values of two or more given causal variables simultaneously (e.g., 'Diabetic diet plan' spans 'Disease management' and 'Nutritional strategy'). Each LLM generates the dataset for eight domains, positive and negative examples are sampled, and every LLM classifies examples from its own and other models' data in a zero-shot setting with temperature zero; performance is scored by precision, recall, F1, and Cohen's $\\kappa$. A cosine-similarity threshold classifier over text embeddings serves as a non-LLM reference to see whether the tasks reduce to semantic similarity.","core_discovery":"On the paper's own terms, the discovery is that contemporary LLMs already support the two fundamental modeling steps that precede causal reasoning: deciding that two textual entities are values of the same causal variable rather than separate variables, and recognizing an entity as an interaction of several causal variables that should be represented as an artificial variable. The evaluation yields a consistent pattern: modern dense models (GPT-4-turbo, Llama3-70b) perform better at variable distillation, while sparse-expert Mixtral-8$\\times$22b is best at interaction detection, and performance depends strongly on domain, with Health being the easiest for most models. The paper explicitly positions these findings as evidence that LLMs can be used in collaboration with human experts, providing a wider perspective, rather than as standalone causal modelers.","pith_inferences":["Because the evaluation uses LLM-generated data as ground truth, the reported kappas are best read as self-consistency with machine-generated labels; a human-expert-annotated set could produce different absolute values and possibly different rankings.","A natural extension is to measure human-LLM collaboration directly: whether expert modelers who see LLM suggestions produce causal graphs faster or with higher inter-expert agreement than modelers working alone.","The two tasks could form the core of a causal-modeling benchmark for future LLMs, but only after adding human-validated labels and non-English prompts."],"forward_implications":["Off-the-shelf LLMs can already assist human experts in converting textual domain knowledge into causal variables and in flagging interaction entities, though agreement is only fair to moderate ($\\kappa$ up to 0.40 and 0.60).","Model choice should be task-dependent: GPT-4-turbo and Llama3-70b are the strongest at variable distillation, while Mixtral-8$\\times$22b is the strongest at interaction detection.","Domain matters more than raw model size for these tasks: most models score highest on Health data and much lower on Physics, suggesting a reliance on training-domain knowledge.","Simple semantic-similarity baselines do not explain LLM performance, so these tasks require genuine modeling rather than embedding lookup."],"supporting_citations":[{"why":"Supplies the guideline that causal models should contain causal variables, not their realized values; this is the normative basis of Task 1.","marker":"Suzuki et al., 2020"},{"why":"Argues interaction entities can violate transitivity; motivates Task 2's treatment of interaction entities.","marker":"McDonnell [2018]"},{"why":"Recommends representing interactions as artificial variables; informs the paper's target representation.","marker":"VanderWeele and Robins [2009]"},{"why":"Defines modularity in causal diagrams; explains why conflating values with variables is a modeling error.","marker":"Kuorikoski [2012]"},{"why":"Provides the kappa coefficient used as the main agreement measure.","marker":"Cohen [1960]"},{"why":"Prior work on LLM causal inference from correlation; the paper positions its variable-modeling tasks against this.","marker":"Jin et al., 2023"},{"why":"Argument that LLMs recite training-domain causal knowledge; the paper's domain-dependence result aligns with it.","marker":"Zečević et al., 2023"}],"fun_headline_variants":["LLMs aid causal modeling: pick model per task","Dense vs sparse LLMs for causal modeling: task matters","GPT-4 and Llama3 excel at causal variables; Mixtral at interactions","Causal modeling with LLMs: model choice depends on task and domain","LLMs help distill causal knowledge and detect interactions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the positive and negative examples sampled from LLM-generated data as ground-truth labels for both tasks; if those generated groupings contain systematic errors, the measured kappa values and model rankings reflect agreement with machine-generated labels rather than true causal modeling ability.","fun_headline_variants_meta":{"raw":{"variants":["LLMs aid causal modeling: pick model per task","Dense vs sparse LLMs for causal modeling: task matters","GPT-4 and Llama3 excel at causal variables; Mixtral at interactions","Causal modeling with LLMs: model choice depends on task and domain","LLMs help distill causal knowledge and detect interactions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000339,"raw_usage":{"total_tokens":1831,"prompt_tokens":867,"completion_tokens":964,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":876}},"tokens_in":483,"tokens_out":964,"duration_ms":6752,"temperature":1.0,"reasoning_tokens":876,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:47:29.013818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two or more human domain experts independently annotate a held-out sample of entity pairs (Task 1) and interaction-entity triples (Task 2) across the same eight domains, then run the same zero-shot prompts and compute Cohen's $\\kappa$ against the human labels; if the absolute values or the model ranking change materially, the paper's conclusion that these LLMs are the helpful tools described here is not supported.","supporting_citations":[{"cited_title":"Event causality extraction via implicit cause-effect interactions","cited_arxiv_id":null,"evidence_quote":"Prior work on LLM causal inference from correlation; the paper positions its variable-modeling tasks against this."}],"review_version":1}