{"id":"3bc02d4f-b52d-4762-8dc1-26b0d65e29b2","arxiv_id":"2504.13263","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-based agent that selects, configures, and runs causal analysis algorithms reports top F1 scores on synthetic benchmarks, but its gains are not cleanly separated from postprocessing steps.","lead":"Causal-Copilot is an LLM-driven agent that automates causal discovery and inference on tabular and time-series data, from algorithm selection to report generation. It reports higher F1 scores than several standard causal algorithms and a bare GPT-4o baseline across synthetic benchmarks with up to 1000 variables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing ablation: postprocessing may explain Causal-Copilot's edge; baselines never receive bootstrap/LLM refinement, so the 'selection strategy' claim is unsupported.","rationale":"The reader's weakest assumption correctly identifies the missing control: postprocessing is part of Causal-Copilot's reported pipeline but is never applied to baselines and never ablated away. This is the most load-bearing issue because Section 5.2 makes an explicit causal attribution to algorithm and hyperparameter selection, and the bootstrap/LLM refinement stage is the only pipeline component whose exclusion from baselines could plausibly explain large F1 gaps in exactly the scenarios the paper highlights (dense, heterogeneous, compound). The paper deserves credit for a broad benchmark, a live demo, and a reproducible codebase, but those do not replace the missing ablation. The concern is addressable, so a conditional verdict is appropriate; I would not change the reader's CONDITIONAL result. A secondary worry, noted but not developed here, is that the Section 5.1 benchmarking results are used to populate the selection knowledge and then evaluated on data from the same simulator, so cross-distribution generalization is untested; the postprocessing control should be run first because it is the direct test of the paper's stated mechanism.","tokens_in":75907,"tokens_out":5018,"duration_ms":46557,"concrete_test":"Run two ablation arms on the Table 2 and Table 3 scenarios where the gains are largest (Dense, Heterogeneous Domains, Clinical, Financial, time-series non-Gaussian): (1) Causal-Copilot with the Section 2.4 postprocessing module disabled, reporting the raw output of the selected algorithm; (2) each raw baseline (PC, FCI, GES, DirectLiNGAM, GPT-4o-selected method) passed through the same bootstrap and LLM postprocessing code. If selection-only Causal-Copilot still beats postprocessed baselines in those scenarios, the selection claim is supported; if the gaps collapse or reverse, the headline result is a postprocessing artifact and the abstract should be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's Section 5.2 text attributes Causal-Copilot's F1 advantage to its algorithm and hyperparameter selection strategy, but the reported Causal-Copilot numbers include the full Section 2.4 pipeline, while PC, FCI, GES, DirectLiNGAM, and GPT-4o are evaluated without bootstrap edge confidence or LLM graph refinement. No ablation disables postprocessing, and no control applies the same postprocessing to baselines. The bootstrap step alone can add and delete edges, and the LLM step can reintroduce plausible edges, so the gains in dense, heterogeneous, missing-data, and compound scenarios could come from postprocessing rather than from selecting a better algorithm. The GPT-4o baseline is also denied statistical diagnostics and postprocessing, making the comparison a whole-system contrast rather than an evaluation of selection. This does not invalidate Causal-Copilot as a deployed system, but it does invalidate the specific mechanistic claim that the selection strategy is what delivers the superiority; it also leaves the abstract's unqualified 'superior performance' over-scoped, since Table 2's default settings already show PC and GES ahead of Causal-Copilot.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Causal-Copilot, an LLM-orchestrated agent that automates the full causal analysis pipeline: preprocessing, algorithm selection, hyperparameter configuration, execution, postprocessing (bootstrap edge confidence and LLM-guided graph refinement), and report generation. The system integrates over twenty causal discovery and inference algorithms. The paper reports two main experimental contributions: a large preliminary benchmarking of causal discovery algorithms (Section 5.1) that is used to build a knowledge base for algorithm selection, and a comparative evaluation of Causal-Copilot against PC, FCI, GES, DirectLiNGAM, and a GPT-4o baseline on synthetic tabular and time-series data with known ground truth (Section 5.2). The central claim is that Causal-Copilot achieves superior F1 performance thanks to its automated algorithm and hyperparameter selection strategy.","tokens_in":76202,"tokens_out":4768,"duration_ms":41261,"significance":"If validated, the system would be a practically valuable tool for non-experts, and the paper's integrated benchmarking effort is a useful resource for the causal discovery community. The paper ships code, data, and a live demo, and reports results with error bars on synthetic data with known ground truth. However, the current evaluation does not isolate the contribution of the algorithm-selection strategy from the postprocessing module, and the test scenarios overlap heavily with the benchmarking distribution used to build the knowledge base. The central attribution claim is therefore not yet supported, although the system as a whole may still be effective. The paper's claims need to be substantially qualified or defended with additional controlled experiments.","major_comments":[{"comment":"The Causal-Copilot results in Tables 2 and 3 include the full postprocessing module from Section 2.4 (bootstrap edge confidence and LLM-guided graph refinement), whereas the baselines (PC, FCI, GES, DirectLiNGAM, and GPT-4o) are evaluated without any postprocessing. The text in Section 5.2 attributes the F1 advantage to the 'algorithm and hyperparameter selection strategy.' The paper reports no ablation that disables postprocessing for Causal-Copilot, nor any control that applies the same postprocessing to the baselines. Since the postprocessing module can add and delete edges, the observed gains in dense, heterogeneous, missing-data, and compound scenarios could arise entirely from postprocessing rather than from selecting a better algorithm. This missing ablation leaves the central mechanism unidentified and unsupported.","section":"Section 5.2 vs Section 2.4"},{"comment":"The algorithm-selection knowledge base is constructed from the Section 5.1 benchmarking, which uses a specific simulator based on Zheng et al. (2018) with parameter grids over node counts, sample sizes, edge probabilities, noise types, and missing rates. The Section 5.2 evaluation generates test scenarios from the same simulator and with overlapping parameter ranges (compare Appendix C.1 with C.2 and C.3 with C.4). Consequently, the system's predictions are informed by performance data from the same distribution on which it is tested. The paper should quantify the overlap between the benchmark grid and the test grid, and should evaluate on genuinely out-of-distribution settings (e.g., different graph families, non-Erdos-Renyi structures, or parameter values outside the benchmark grid) to support the abstract's claim of a reliable, scalable, and extensible solution.","section":"Appendix C.1, C.2, C.3, C.4 and Section 5.1/5.2"},{"comment":"The abstract claims 'superior performance compared to existing baselines,' but in the default settings of Table 2, Causal-Copilot's F1 is 0.900±0.120 while PC and GES achieve 0.920±0.050 and 0.920±0.090, respectively. The baselines' point estimates are higher and the error bars overlap. This basic scenario directly contradicts the unqualified superiority claim. The claim should be qualified to the specific configurations where Causal-Copilot does show a consistent advantage (e.g., dense graphs, high node counts, data-quality challenges, and compound scenarios).","section":"Table 2, Default Settings row"}],"minor_comments":[{"comment":"The text refers to 'XES' twice; this should be 'XGES.'","section":"Section 5.1.1, Sensitivity to Graph Density"},{"comment":"The row labels 'Super Large (p=100)' and 'Large Scale Extreme Large Node and Sample (p=1000, n=10000)' are confusing; the naming convention should be made consistent and clearer.","section":"Table 2, Node Count rows"},{"comment":"For p=100, Causal-Copilot's F1 is 0.182, which is low in absolute terms; the text calls this 'superior scalability,' but this is only relative to the baselines that fail to complete and should be presented as such.","section":"Table 3, Very Large row"},{"comment":"The metric description says all graph types are converted to DAGs by 'sampling the best DAG representation from the equivalence class,' but the sampling procedure is not specified; this could affect F1 comparisons across methods that output different graph types (CPDAG, PAG, DAG) and should be clarified.","section":"Appendix D"},{"comment":"CORL is described as a reinforcement-learning-based method but is listed under 'Continuous Optimization-based Methods'; the category name is inaccurate for this algorithm.","section":"Section 3.1, Continuous Optimization-based Methods"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-engineered systems contribution, but the evaluation does not currently support the paper's central attribution of performance gains to algorithm selection. The authors should be asked to provide an ablation that separates the postprocessing module from the selection module, and to address the overlap between the benchmarking and evaluation distributions. If they cannot provide out-of-distribution evaluation, the claims in the abstract and Section 5.2 should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a real system and the first full LLM-orchestrated causal-analysis pipeline I am aware of, and the benchmark is broad enough to be useful on its own. But the paper's central claim—that automated algorithm and hyperparameter selection is what makes it superior—is not supported by the experiments as run. The comparison gives Causal-Copilot its full Section 2.4 postprocessing (bootstrap edge confidence plus LLM graph refinement) while PC, FCI, GES, DirectLiNGAM, and GPT-4o get none. There is no ablation with postprocessing off, and no control that applies the same postprocessing to the baselines. The bootstrap step can add and remove edges; the LLM step can reintroduce plausible edges. So the F1 gap could come from graph polishing rather than from choosing the right algorithm. The abstract's 'superior performance' is also over-scoped: in the default tabular setting, PC and GES both score above Causal-Copilot (0.92 vs 0.90).\n\nWhat is genuinely good: the system integrates more than twenty causal discovery and inference methods, supports natural-language interaction, and produces an end-to-end report. The preliminary benchmark in Section 5.1 and Appendix E is a serious empirical map of how algorithms behave across sample size, dimensionality, density, noise, missingness, heterogeneity, and measurement error. That is a contribution worth keeping. Code, GitHub, and demo are provided, and the simulations are described in enough detail to reproduce.\n\nThe soft spots are real but fixable. First, the algorithm-selection knowledge base is built from the Section 5.1 benchmark on the same simulator used in the Section 5.2 evaluation, with overlapping parameter ranges (Appendix C.1 and C.2). That makes the selection results in-sample, not independent prediction. Second, the GPT-4o baseline is a whole-system contrast—it lacks statistical diagnostics and postprocessing—so it cannot isolate the value of the orchestration layer. Third, there is no real-world validation beyond a single case study; everything is synthetic with known ground truth. For a usability claim that is fine, but for 'superior performance' it is not enough.\n\nWho gets value: builders of LLM agents for scientific pipelines, causal discovery practitioners who want a map of algorithm behavior, and anyone deciding whether automated causal analysis is viable. As written I would not accept the superiority claim, but I would send this to a serious referee and ask for the missing ablations plus an out-of-distribution test. With those, the paper could be solid.","headline":"A real LLM-driven causal-analysis pipeline with a valuable benchmark, but the reported gains are not isolated from postprocessing, so the central selection claim is unsupported as written.","tokens_in":76736,"tokens_out":3662,"would_cite":true,"duration_ms":34995,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM-orchestrated pipeline that picks and tunes causal analysis algorithms outperforms single-method baselines, the paper claims.","keywords":["causal discovery","causal inference","large language model","autonomous agent","algorithm selection","hyperparameter optimization","time series","graph refinement"],"falsifier":"Run the full benchmark with the postprocessing module disabled, keeping only preprocessing and LLM-based algorithm selection, and compare F1 against the reported numbers; if scores fall to roughly the level of the raw baseline algorithms or fail to beat the GPT-4o baseline, the selection strategy is not the cause of the advantage.","tokens_in":75736,"feed_emoji":"🤖","tokens_out":6671,"duration_ms":59014,"temperature":0.7,"pith_summary":"Causal-Copilot is an autonomous agent that uses a large language model to run the full causal analysis pipeline, from preprocessing and algorithm selection to execution, postprocessing, and report generation, on both tabular and time-series data. The paper's central claim is that LLM-driven algorithm selection and hyperparameter configuration outperform running any single causal discovery algorithm, such as PC, FCI, GES, or DirectLiNGAM, and also outperform a bare GPT-4o baseline that lacks statistical diagnostics and domain knowledge. The reported F1 scores exceed 0.9 in many clean synthetic settings and remain competitive on dense, high-dimensional, missing, noisy, and heterogeneous data where baseline methods often time out. If correct, this would let non-specialists obtain causal graphs and effect estimates by asking natural-language questions, without hand-tuning method-specific assumptions.","feed_headline":"An LLM agent automates causal discovery and beats raw algorithms","feed_subtitle":"Natural-language causal analysis without manual tuning: results stay strong on dense, missing, and high-dimensional data.","key_machinery":"The load-bearing machinery is the algorithm-selection loop supported by a causality-specific knowledge memory: rules and tagged performance ratings narrow the candidate algorithm set, the LLM ranks and configures the survivors using statistical diagnostics and empirical benchmarks, and execution errors loop back for revision. A second mechanism is the postprocessing module, where bootstrap resampling produces per-edge confidence estimates, high-confidence missing edges are added and weak edges removed, and moderate-confidence edges are sent to the LLM for conceptual plausibility review, with LLM suggestions treated as soft and barred from overriding high-confidence statistical decisions. Together these mechanisms let the system adapt its choice of method to the data at hand and then anchor the final graph statistically.","core_discovery":"On the paper's own terms, the discovery is that a fully automated, LLM-orchestrated workflow can make causal analysis accessible and accurate: the system chooses among more than twenty causal discovery and inference methods, configures their hyperparameters using dataset statistics and a causality-specific knowledge base, executes the methods, and then refines the output graph using bootstrap edge-confidence estimates and plausibility checks. Across the synthetic benchmarks reported, the full pipeline maintains high F1 performance across graph density, node count, sample size, noise type, missing-data rate, measurement error, and heterogeneous domains, while completing tasks at up to 1000 nodes that individual baseline algorithms fail to finish within the time limit. The result is a claim about the whole agent, not about any single module, because the evaluated comparison runs the full pipeline, including its postprocessing, against raw baseline algorithms.","pith_inferences":["A direct subtraction test, applying the same postprocessing to the baseline algorithms, would reveal whether the agent's selection strategy or its refinement module produces the reported gains; this is the crucial experiment the paper does not run.","Because the evaluation is entirely on synthetic data with known ground truth, real observational deployment would need a different validation route, such as resampling consistency or agreement with domain knowledge.","The benchmarking data could be treated as a reusable empirical performance map from data-characteristic vectors to algorithm rankings, independent of the LLM agent.","The LLM refinement step introduces a risk of plausible-sounding but false edges; the paper's restriction of LLM changes to moderate-confidence edges is a testable guardrail whose effect on downstream causal estimates could be measured."],"forward_implications":["Non-specialists could run credible causal discovery and effect estimation by describing their question in natural language, with algorithm choice and hyperparameters set automatically.","Large, dense, missing, or heterogeneous datasets become tractable where a single method times out or collapses, up to roughly 1000 variables in the reported settings.","The benchmarking results themselves form an empirical map of which causal discovery algorithms work under which data conditions, which could be reused beyond the agent.","LLM-based graph refinement can be bounded as a soft postprocessing step, preserving high-confidence statistical decisions while adding domain plausibility."],"supporting_citations":[{"why":"Provides the PC algorithm and the foundational constraint-based causal discovery framework that the system integrates and uses as a baseline.","marker":"(Spirtes et al., 2000)"},{"why":"Supplies GES, the score-based benchmark algorithm the system is compared against for tabular discovery.","marker":"(Chickering, 2002)"},{"why":"Supplies DirectLiNGAM, the linear non-Gaussian baseline method that appears throughout the comparison tables.","marker":"(Shimizu et al., 2011)"},{"why":"Supplies PCMCI, the primary constraint-based time-series baseline used in the temporal evaluation.","marker":"(Runge et al., 2019)"},{"why":"Provides NOTEARS and the structural equation data-generation process used to create the synthetic ground-truth benchmarks.","marker":"(Zheng et al., 2018)"},{"why":"Supplies DYNOTEARS, the time-series structure-learning baseline that is particularly competitive in smaller settings.","marker":"(Pamfil et al., 2020)"},{"why":"Provides the double-machine-learning and doubly robust implementations used for the causal-inference half of the pipeline.","marker":"(Battocchi et al., 2019)"},{"why":"Establishes the large language model capability that the system's orchestration layer relies on.","marker":"(Brown et al., 2020)"},{"why":"Supplies CD-NOD, the heterogeneous-data method the system selects when the data contain multiple domains.","marker":"(Huang et al., 2020)"},{"why":"Supplies XGES, a score-based method that the preliminary benchmarking shows to be a strong general-purpose candidate.","marker":"(Nazaret and Blei, 2025)"}],"fun_headline_variants":["LLM agent automates causal analysis, stays accurate at 1000 nodes","Autonomous causal analysis agent: selects methods, tunes them, explains results","Causal analysis, no PhD required: LLM agent automates it","Handles 1000-node causal analysis where baselines time out"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the F1 advantage comes from the algorithm-selection strategy, but the baseline algorithms run without the system's bootstrap-based edge refinement, and the paper never tests the selection step on its own; if that refinement alone produces the gain, the central claim is not supported.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent automates causal analysis, stays accurate at 1000 nodes","Autonomous causal analysis agent: selects methods, tunes them, explains results","Causal analysis, no PhD required: LLM agent automates it","Handles 1000-node causal analysis where baselines time out"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00131,"raw_usage":{"total_tokens":5345,"prompt_tokens":957,"completion_tokens":4388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":4309}},"tokens_in":573,"tokens_out":4388,"duration_ms":30084,"temperature":1.0,"reasoning_tokens":4309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:12:09.092410+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full benchmark with the postprocessing module disabled, keeping only preprocessing and LLM-based algorithm selection, and compare F1 against the reported numbers; if scores fall to roughly the level of the raw baseline algorithms or fail to beat the GPT-4o baseline, the selection strategy is not the cause of the advantage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides NOTEARS and the structural equation data-generation process used to create the synthetic ground-truth benchmarks."}],"review_version":1}