{"id":"a931944f-8846-4054-a654-fba17a3d817a","arxiv_id":"2412.17970","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A causal reasoning benchmark with graph and tabular data shows open-source LLMs are weak at causal discovery from tables, and reveals that task correlations do not align with the benchmark's three categories.","lead":"CARL-GT is a new benchmark that tests large language models on causal reasoning tasks using causal graphs and numerical tables, spanning graph reasoning, knowledge discovery, and decision making. It evaluates five open-source LLMs and finds they perform poorly, especially at discovering causal relationships from tabular data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 reports 0.00 causal-direction accuracy for models whose unknown answers were supposedly replaced at random; with binary answers and uniform random replacement, 0.00 is impossible, so the headline tabular-weakness result rests on an invalid score.","rationale":"The benchmark is a plausible contribution: it builds on standard synthetic causal graph and tabular data generation, and the zero-shot prompt design is reasonable. However, the central claim that LLMs are weak at causal reasoning from tabular data depends critically on Table 2, and that table is internally inconsistent. The caption promises random replacement for unknown answers, but the reported 0.00 accuracies are impossible under that rule, and Mixtral appears in the table despite the caption excluding it. This is a concrete, checkable correctness risk, and it is more direct than the reader's extraction concern: even before worrying about the unspecified extraction model, the score processing itself violates the stated protocol. The reader's weakest assumption was about two-step extraction; I partially agree, because extraction is also a risk for graph-reasoning and causal-inference metrics, but the immediate load-bearing issue is the unknown-answer scoring rule in the knowledge-discovery metrics. If the corrected numbers still show Llama3 and Mistral at or below chance with proper confidence intervals, the qualitative conclusion survives, which is why I do not move the verdict beyond CONDITIONAL. The paper should be accepted only conditional on correcting Table 2 and documenting refusal handling; the reader already reached this position, so the verdict remains UNCHANGED.","tokens_in":903,"tokens_out":834,"duration_ms":86884,"concrete_test":"Inspect the CARL-GT evaluation script and rerun the knowledge-discovery experiments with unknown-response logging. For each model, record the raw response, mark refusals as refused, and score only responses containing a parseable yes/no; compute accuracy and exact 95% binomial CI on the parseable subset, and report refusal rate separately. If Qwen2/Gemma2/Mixtral corrected accuracies become roughly 0.5 (or their rows are dropped), Table 2's below-chance finding for these models disappears; if Llama3 and Mistral remain below 0.5 with CIs excluding chance, the qualitative weak claim survives, but the specific 0.00 numbers are still invalid. Also print the random-replacement code path and verify that a 0.00 score cannot be produced while the replacement is active.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is that the headline evidence for weak tabular causal reasoning rests on Table 2, whose stated scoring rule makes its reported 0.00 values impossible. The table caption says: When there are unknown answers from LLMs, we replaced by random yes or no. For a binary question with uniform random substitution, expected accuracy is 0.5, not 0.00, regardless of label balance. Yet Qwen2-7B, Gemma2-9B, and Mixtral-8x7B all show Acc. 0.00. Either the replacement was not applied, the replaced answers are scored as incorrect regardless of the random draw, or these entries are placeholder codes. The last row also lists Mixtral despite the caption saying Mixtral is excluded due to GPU memory. The surrounding text says these models cannot provide the answer and reply that there is not enough information; that is a refusal or format-compliance behavior, not a measured 0% causal-direction accuracy. Because the abstract's strongest quantitative claim is built on this table, the score processing must be fixed before the benchmark results can support the conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CARL-GT, a benchmark for evaluating causal reasoning in large language models using synthetic causal graphs and tabular data. The benchmark defines tasks in three categories: causal graph reasoning (adjacency, d-separation, causal direction), knowledge discovery from tables (d-separation, causal direction), and decision making (interventional and counterfactual inference). The authors evaluate five open-source LLMs in a zero-shot setting with serialized tables and graph descriptions, and report that models are particularly weak on table-based knowledge discovery, often performing at or below chance. They also analyze correlations between task performances and conclude that tasks in different categories are more correlated than tasks in the same category. The benchmark code is released.","tokens_in":13885,"tokens_out":5969,"duration_ms":52764,"significance":"If the measurement issues are resolved, CARL-GT is a useful and easily extensible synthetic benchmark for a relatively under-served capability area. The synthetic generation from structural causal models is principled, the release of code and prompts supports reproducibility, and the inclusion of serialized tabular data is practically relevant. However, the headline quantitative claim about table-based causal reasoning rests on Table 2, whose scoring appears internally inconsistent; until that is fixed, the magnitude of the reported weakness is unverified. The task-correlation analysis also needs stronger statistical support.","major_comments":[{"comment":"The caption of Table 2 states that unknown LLM answers are replaced by random yes/no, but the table reports Acc=0.00 for Qwen2-7B, Gemma2-9B, and Mixtral-8x7B. For a binary question, uniform random replacement yields an expected accuracy of 0.5 regardless of label balance, so 0.00 is impossible under the stated procedure. This suggests that the replacement was either not applied, that replaced answers were always scored as incorrect, or that the entries are placeholders. Because this table is the primary evidence for the claim that LLMs are weak in table-based causal reasoning, the scoring procedure must be corrected and the table regenerated or clearly annotated.","section":"Section 4.2, Table 2"},{"comment":"The Mixtral row appears in Table 2 with Acc=0.00 even though the caption says Mixtral results are omitted due to GPU memory limits. This is a direct contradiction. Either the row should be removed or the caption amended; the text in Section 4.2 also states that Mixtral could not answer the causal-direction questions, so the table should distinguish between \"not evaluated\" and \"evaluated but refused.\"","section":"Table 2"},{"comment":"The two-step answer extraction uses an unspecified LLM-based extractor. If the extractor is the same model being evaluated, a model that fails to comply with the output format or refuses to answer could have its verbatim response misparsed, systematically biasing the reported metrics. The paper should specify the extraction model (or state that human annotation was used) and provide evidence that extraction errors do not drive the results, particularly for the 0.00 accuracy values.","section":"Section 4.1, Templates 4.2 and 4.3"},{"comment":"The correlation analysis uses ranking vectors from only five LLMs, yielding only 5 data points for each coefficient. With such a small sample, the reported correlations (e.g., -0.9, 0.9) are highly unstable and not statistically meaningful; the paper even concedes that more LLMs would be needed. Since the abstract presents the task-relationship finding as a contribution, this analysis should be either expanded to more models or explicitly framed as an illustrative observation without the quantitative correlation claim.","section":"Section 4.5, Table 6"}],"minor_comments":[{"comment":"The phrase \"casual reasoning\" appears in the abstract and should be corrected to \"causal reasoning.\"","section":"Abstract"},{"comment":"The sentence \"Therefore, we introducing a benchmark\" should be \"Therefore, we introduce a benchmark.\"","section":"Section 1"},{"comment":"The word \"summerized\" should be \"summarized.\"","section":"Section 3.2"},{"comment":"The spelling of \"MarkDown\" and \"Markdown\" is inconsistent; please use one form throughout.","section":"Section 4.1"},{"comment":"The edge notation in the template (e.g., \"V 1− > V2\") has inconsistent spacing and should be formatted uniformly, e.g., \"V1->V2.\"","section":"Template 4.1"},{"comment":"The MathVista reference appears twice (Lu et al., 2023 and Lu et al., 2024) with the same title and author list; this duplication should be removed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The self-citation of Tu et al. (2024) for data generation is reasonable, but the benchmark's novelty relative to existing causal reasoning benchmarks (e.g., CLadder, CORR2CAUSE) should be sharpened. The inconsistencies in Table 2 are the most pressing issue because they directly undermine the paper's central claim. Please ask the authors to provide the raw response logs or a corrected scoring script during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading for the benchmark, not for the numbers. CARL-GT is a sensible extension of causal-reasoning evaluation to tabular data, with graph reasoning, discovery, and decision-making tasks grounded in SCMs. The task design is competent, and the synthetic data generation follows their prior work (Tu et al. 2024), which is fine.\n\nTwo real problems. First, Table 2 reports 0.00 accuracy for Qwen2-7B, Gemma2-9B, and Mixtral-8x7B on causal direction, with the caption saying unknown answers were replaced by random yes/no. That is impossible: binary uniform random replacement gives expected accuracy 0.5. Either the replacement was never applied, the unknown answers were scored as wrong, or the 0.00 is a placeholder. And Mixtral is listed despite the caption excluding it for GPU memory. This table is the main evidence for the abstract's claim that LLMs are weak with tabular data. The conclusion may still be true, but these numbers don't support it.\n\nSecond, the answer extraction pipeline (Section 4.1) uses an unspecified LLM to convert responses into yes/no or floats. If it's the same model, refusals can be misread as 0.00. The paper doesn't report which model extracts, nor any manual verification. That makes every metric downstream of extraction uncertain.\n\nThe correlation analysis across tasks is also built on rankings of five models—too few to support the claim that cross-category correlations are stronger than within-category. The authors do flag the small N, which is honest, but the claim outruns the data.\n\nWhat's good: the benchmark design is modular and reusable, the prompts are explicit, and the tasks cover the Pearl hierarchy cleanly. A corrected run with proper handling of refusals and a real answer-extraction protocol would make this a useful benchmark. As it stands, treat the quantitative claims as unverified.\n\nFor peer review: yes, send it, because the benchmark deserves scrutiny and the flaws are fixable, but it needs major revision before acceptance. If you're short on time, read Section 3 and skip the experiments.","headline":"The CARL-GT benchmark is a sensible addition to causal-reasoning evaluation, but Table 2's impossible 0.00 accuracies make the main experimental evidence unreliable.","tokens_in":14517,"tokens_out":2222,"would_cite":false,"duration_ms":21527,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models perform at or near chance on causal reasoning when the input is a serialized data table, according to CARL-GT, a benchmark that tests causal graph reasoning, knowledge discovery, and decision-making.","keywords":["causal reasoning","benchmark","large language models","knowledge discovery","d-separation","counterfactual reasoning","tabular data","decision making"],"falsifier":"Run the same CARL-GT prompts with human-verified answer extraction or constrained decoding that forces the output into yes/no or a float, then compare classification scores and accuracy. If the models that currently score zero on table-based causal direction, such as Qwen2-7B, Gemma2-9B, and Mixtral-8x7B, jump above chance, the extraction step is responsible for the reported failure; if they remain at zero, the failure is in the models' causal reasoning itself.","tokens_in":13471,"feed_emoji":"📊","tokens_out":8892,"duration_ms":79535,"temperature":0.7,"pith_summary":"CARL-GT is a benchmark that tests large language models on three families of causal tasks: reading causal graphs, inferring causal structure from observational tables, and estimating intervention and counterfactual quantities. The paper's central finding is that today's open-source LLMs are still weak at these tasks, particularly when they must discover causes from tabular data: on d-separation judgments (whether two variables are independent once a third set is fixed) from tables, several models score near 0.5 on a 0-to-1 scale, the chance level, and on causal-direction questions three of five models return zero accuracy because they decline to answer. The benchmark generates its own graphs and tables with known cause-effect equations, so every question has a ground truth, and it supplies prompts that work without task-specific examples. The reason to care is that it offers a direct way to test whether LLM reasoning transfers from language and math to the structured-data settings real-world decision support actually uses.","feed_headline":"LLMs near chance when inferring causes from tables","feed_subtitle":"A new testbed with ground-truth graphs shows models that ace math still fail at d-separation and causal direction from spreadsheets.","key_machinery":"The central object is the CARL-GT benchmark: synthetic directed acyclic graphs $G=(V,E)$, tabular datasets generated from linear structural causal models defined by those graphs, and task families that ask for adjacency matrices, d-separation relationships, causal directions, intervention distributions, and counterfactual expectations. The design that carries the argument is the pairing of a causal graph with a serialized table for the same underlying data-generation process, so the same causal relationship can be probed both as a verbalized graph question and as a table-only discovery question; this makes the difficulty gap between the two settings interpretable. Ground-truth answers for intervention and counterfactual tasks are obtained by fitting structural causal models to larger samples, and the reported metrics are classification scores for d-separation and direction, F1 scores for graph recovery, and mean absolute error for inference. The prompt templates, with graph data in an expert-analyst text form and tables in Markdown, are part of the benchmark rather than an afterthought.","core_discovery":"The paper claims that current large language models do not yet do reliable causal reasoning, and that the weakness is most visible when the evidence is a serialized table rather than a verbalized graph. On explicitly provided causal graphs, the tested models handle adjacency questions reasonably but struggle with d-separation and direction; on tables alone, d-separation classification hovers at chance and causal-direction questions produce zero accuracy for Qwen2-7B, Gemma2-9B, and Mixtral-8x7B because the models answer that the information is insufficient. The paper also claims that performance across task categories is not what one would expect from a single reasoning ability: graph-based adjacency prediction and counterfactual estimation correlate strongly, while intervention estimation correlates negatively with most other tasks. The contribution is the benchmark itself, offering generated causal graphs, matching tabular data, ground-truth answers, and reusable zero-shot prompts as a reproducible measuring stick for causal reasoning in LLMs.","pith_inferences":["An extension the paper does not run: replace the LLM-based answer extraction with constrained decoding that forces a yes/no or float token, to test whether the near-chance scores are reasoning failures or measurement artifacts.","The correlation analysis rests on five models, so the reported task relationships are fragile; adding more models and per-instance scores could confirm or overturn them.","Because the data are generated by linear Gaussian mechanisms, the benchmark could be extended to nonlinear and discrete systems, and the ordering of task difficulty (adjacency versus d-separation versus direction) would probably change.","The near-universal refusals on table-based causal direction suggest that current pretraining does not instill the inductive bias needed to orient edges; a targeted training objective on synthetic causal tables is a concrete next step the benchmark could monitor."],"forward_implications":["Evaluation of causal ability in LLMs should include table-only tasks, because verbalized-graph results alone overstate causal competence.","Open-source models today are not dependable for zero-shot knowledge discovery from observational data, so deployment in discovery or decision-support settings should not assume causal reliability.","Because question difficulty rises sharply from 10 to 51 nodes in the causal graphs, current models are unlikely to scale to realistic problem sizes without further work.","The synthetic generation gives ground truth for every question, so the benchmark can separate genuine causal computation from memorized facts about causality.","The observed negative correlation between intervention estimation and the other tasks suggests that causal reasoning is not a single competence; improvement work needs to target sub-skills separately."],"supporting_citations":[{"why":"Supplies the causal-graph generation process, the tabular data-generation model, and the selection of d-separation and d-connection question sets.","marker":"Tu et al., 2024"},{"why":"Provides the expert-analyst graph text encoding used in the prompt templates for graph reasoning tasks.","marker":"Fatemi et al., 2024"},{"why":"Basis for the causal-inference question answering design that the benchmark adapts to tabular and graph inputs.","marker":"Jin et al., 2023"},{"why":"Verbalized causal-graph understanding tasks that CARL-GT extends with numerical tables and ground-truth quantities.","marker":"Sirui et al., 2024b"},{"why":"Supplies the causal-reasoning ladder and d-separation criteria that define the decision-making and graph-reasoning tasks.","marker":"Pearl et al., 2016"},{"why":"Defines the Mistral and Mixtral models whose performance is measured in the experiments.","marker":"Jiang et al., 2023"}],"fun_headline_variants":["New benchmark exposes LLMs' weak causal reasoning on tables","LLMs near chance on d-separation from tabular data","CARL-GT: LLMs flop at causal direction from spreadsheets","Causal reasoning test: LLMs ace graphs but fail tables"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pipeline that extracts final answers from each model's free-text response, an unspecified LLM asked to summarize the answer as yes, no, or a float, faithfully preserves the causal reasoning the evaluated model actually performed; if that extraction distorts, discards, or replaces answers, every reported score is suspect.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark exposes LLMs' weak causal reasoning on tables","LLMs near chance on d-separation from tabular data","CARL-GT: LLMs flop at causal direction from spreadsheets","Causal reasoning test: LLMs ace graphs but fail tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1507,"prompt_tokens":989,"completion_tokens":518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":445}},"tokens_in":605,"tokens_out":518,"duration_ms":5553,"temperature":1.0,"reasoning_tokens":445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:07:42.124754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same CARL-GT prompts with human-verified answer extraction or constrained decoding that forces the output into yes/no or a float, then compare classification scores and accuracy. If the models that currently score zero on table-based causal direction, such as Qwen2-7B, Gemma2-9B, and Mixtral-8x7B, jump above chance, the extraction step is responsible for the reported failure; if they remain at zero, the failure is in the models' causal reasoning itself.","supporting_citations":[],"review_version":1}