{"id":"c5aafed1-a235-469a-99a8-6687bc4e22b5","arxiv_id":"2411.17989","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-LLM weighted penalty added to score-based causal discovery gives inconsistent gains on five small benchmarks, with several results worse than the unmodified baseline.","lead":"This paper combines causal graph predictions from several large language models into a penalty term for score-based causal discovery. The authors report that the approach improves some benchmark results but also worsens others, and they do not compare against using a single language model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The multi-LLM collaboration claim is never isolated: no single-LLM baseline appears in Table 1, and the paper's own case study shows GPT-4 alone is perfect on Asia while the multi-LLM-enhanced GES is worse.","rationale":"The reader's conditional verdict identifies a real weakness around the BIC-based weighting and lambda tuning, and I agree the penalty-weighting step is under-justified. My stress-test focuses on a different, more directly load-bearing gap: the paper's claimed contribution is multi-LLM collaboration, but no single-LLM baseline is reported, and the case study shows a single LLM can outperform the multi-LLM-enhanced algorithm. This is not an objection to the broad idea of LLM priors; it is a missing control that the central claim logically requires. The reader's rationale did mention 'no single-LLM baseline' among the issues, so there is partial agreement, but the reader's weakest_assumption field centered on the BIC proxy and lambda, not on this missing comparison. My recommended verdict stays CONDITIONAL/UNCHANGED because the issue is fixable with additional experiments: add single-LLM baselines and a principled lambda rule. If those controls reverse the current table, the claim should be weakened or rejected; if the multi-LLM penalty still wins, the central claim gains real support.","tokens_in":6916,"tokens_out":8281,"duration_ms":83221,"concrete_test":"Re-run the identical pipeline with four additional penalty-source conditions: GPT-3.5 only, GPT-4 only, Gemini only, and an unweighted average of the three LLM graphs, across all datasets in Table 1. Fix lambda by a pre-specified rule (e.g., the same lambda for all datasets, or lambda selected by data-only cross-validation) and report mean plus/minus standard deviation over multiple data subsamples. If the weighted multi-LLM penalty does not strictly dominate the best single-LLM penalty on SHD/TPR/FDR, the multi-LLM collaboration claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty is combining multiple LLMs, but no experiment compares the multi-LLM penalty against a single-LLM penalty. Section 4.2 supplies the missing control and contradicts the claim: GPT-4 alone recovers all 8 Asia edges with no false positives, yet Table 1's Enhanced GES Asia row has SHD 4 and FDR 0.33, so adding GPT-3.5 and Gemini made the final graph worse than following the best single LLM. Several other Table 1 entries also regress (GES on Child: SHD 26 vs 23, FDR 0.62 vs 0.60; NOTEARS on Asia: TPR 0.375 vs 0.5), directly undercutting the abstract's 'improvements across all datasets' claim. Furthermore, the paper specifies no rule for selecting lambda, reports no lambda values, and its own Discussion admits that tuning lambda 'remains a complex task.' Without a ground-truth-free lambda protocol and without single-LLM controls, the reported gains cannot be attributed to the collaboration mechanism rather than to per-dataset tuning or to the best LLM alone. The load-bearing claim requires that the weighted multi-LLM penalty beat the best single-LLM penalty under a fixed, pre-specified protocol; that evidence is absent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework that uses multiple large language models (GPT-3.5, GPT-4, Gemini) to produce candidate causal graphs, combines them with weights derived from the score function, and injects the weighted combination into score-based causal discovery as a regularization term. The augmented score is applied to GES and NOTEARS directly, and to KCRL by replacing expert priors with LLM priors. Experiments on LUCAS, Asia, Earthquake, SACHS, and Child report SHD, TPR, and FDR for GES, NOTEARS, and KCRL with and without the LLM enhancement. The central claim is that the multi-LLM penalty improves accuracy across datasets and algorithms.","tokens_in":7217,"tokens_out":3503,"duration_ms":31657,"significance":"If the claimed improvements were fully supported, this would be a useful practical contribution: LLM-derived priors could replace costly expert knowledge in score-based causal discovery, and the multi-LLM weighting scheme would add a simple, algorithm-agnostic way to fuse imperfect priors. The paper also ships correct but elementary observations: the l1 and l2 penalties are decomposable in the GES sense, and the l2 penalty is differentiable for NOTEARS. The idea is timely and the presentation is generally clear. However, the experimental evidence is the load-bearing part, and it is currently mixed: several Table 1 entries regress relative to the baselines, no single-LLM control is reported, and the lambda selection protocol is unspecified. The significance of the contribution therefore cannot be assessed from the manuscript as it stands.","major_comments":[{"comment":"The abstract and Section 4.2 state that 'the LLM enhancement leads to improvements across all datasets,' but Table 1 contains clear regressions: for Asia, GES SHD worsens from 3 to 4 and FDR from 0.22 to 0.33; for Child, GES SHD worsens from 23 to 26 and FDR from 0.60 to 0.62; for Asia, NOTEARS TPR drops from 0.5 to 0.375 while SHD rises from 4 to 5. The claim should either be restated as dataset-dependent improvement, or the paper should provide an aggregate, statistically grounded comparison (for example, mean or median SHD over datasets with paired significance tests).","section":"Abstract; §4.2, Table 1"},{"comment":"The central novelty of the paper is the collaboration of multiple LLMs, yet no experiment compares the weighted multi-LLM penalty against a single-LLM penalty under the same framework. The case study in Section 4.2 shows that GPT-4 alone recovers all 8 Asia edges with no false positives, while the enhanced GES result in Table 1 has SHD 4 and FDR 0.33, so the multi-LLM penalty is worse than following the best single LLM on that dataset. A control using each LLM individually in the same penalty mechanism is needed to attribute any observed gains to the collaboration rather than to the prior information itself.","section":"§3.3, §4.2"},{"comment":"The hyperparameter lambda is load-bearing, but its selection is not specified. Section 4.3 admits that 'tuning this parameter remains a complex task,' and the manuscript reports no lambda values and no ground-truth-free protocol. If lambda was tuned per dataset using the true graph, the reported improvements are not a fair evaluation of the method. Please report the chosen lambda values and provide a selection protocol that does not use the test graph, or a sensitivity analysis showing that the improvements are robust to reasonable lambda choices.","section":"§3.3, §4.3"},{"comment":"The weights mu_model are computed by scoring each LLM graph with the same score function Sscore on the same dataset that is later used to fit the final graph. This makes the objective function serve as both the fitting criterion and the quality measure for the prior, which can select LLM outputs that happen to fit the data rather than LLM outputs with genuine causal knowledge. This potential circularity should be addressed directly, for example by using a held-out portion of the data for weighting or by comparing against equal weights.","section":"§3.3"}],"minor_comments":[{"comment":"The heading 'Muti-LLM Enhanced GES' contains a typo and should read 'Multi-LLM Enhanced GES.'","section":"§3.3"},{"comment":"The penalty definitions use the notation ||M_hat - M_hat_LLM|| and ||M_hat - M_hat_LLM||^2 without specifying the norm; please state explicitly that this is the Frobenius norm and keep the matrix notation consistent throughout.","section":"§3.3"},{"comment":"The composition K = U ∘ C ∘ R is described as a sequential operation, but the order in which the three stages are applied is ambiguous; please clarify which stage is executed first and whether R (revision) is applied last.","section":"§3.2"},{"comment":"Table 1 reports single values without standard deviations or repeated runs; if the optimization procedures are deterministic, this should be stated explicitly.","section":"§4.2"},{"comment":"The KCRL experiments are not connected to the methodological section: Section 3 derives the penalty only for GES and NOTEARS, so the manuscript should state precisely how the LLM prior is inserted into KCRL rather than only saying that LLM prior knowledge was used.","section":"§4.2"},{"comment":"Several references are incomplete: [4] lacks full publication details and [15] lacks a venue; please complete the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is reasonable and the theoretical parts are sound, but the experimental validation is not yet convincing. The absence of single-LLM controls is especially problematic because the paper's own case study shows that the best single LLM can outperform the multi-LLM-enhanced algorithm. The lambda protocol and the weighting scheme also need to be specified to rule out tuning to ground truth. These issues are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the multi-LLM penalty idea is real but modest, and the paper's own experiments undercut its headline claim. The framework is simple: query several LLMs for adjacency matrices, weight each by its BIC score on the observed data, then add a weighted l1/l2 penalty to any score-based causal discovery objective. That specific combination is new as far as I can tell; prior work used a single LLM or externally supplied expert priors. The proofs that the penalties are decomposable (for GES) and differentiable (for NOTEARS) are correct — but they are also immediate consequences of the definitions, so the theoretical contribution is thin.\n\nWhat the paper does well: it is clearly written, the framework is genuinely general, and the case study in Section 4.2 is honest about per-LLM performance. That honesty is what exposes the problem. GPT-4 alone recovers all 8 Asia edges with zero false positives, yet the multi-LLM-enhanced GES has SHD 4 and FDR 0.33 — worse than the original GES (SHD 3) and far worse than GPT-4 alone. There is no single-LLM baseline anywhere in Table 1. Without that control, you cannot attribute any gain to the collaboration mechanism rather than to the best LLM or to per-dataset tuning. And the weight computation is circular in a soft way: each LLM's weight comes from the same BIC score on the same dataset used for the final fit, so the penalty is influenced by the objective it is meant to regularize.\n\nThe empirical section has other problems: several Table 1 entries regress (GES on Child SHD 23→26, NOTEARS on Asia TPR 0.5→0.375), the abstract claims improvements across all datasets, which the table contradicts, and lambda is a tuned hyperparameter with no values or selection protocol reported. The Discussion admits tuning 'remains a complex task.' No code and no error bars either.\n\nWho this is for: someone working on LLM priors for causal discovery who wants to see one more knob on the penalty approach. The central claim is unsupported as stated, but the idea is plausible and the paper does not hide its limitations. If the authors add single-LLM baselines, fix a lambda protocol, and report proper uncertainty, it could become a solid workshop paper or a minor conference contribution.\n\nMy recommendation: send it to peer review, but expect major revision. The novelty is enough to deserve referee time; the current evidence is not enough to accept.","headline":"A modest multi-LLM penalty idea whose own experiments undercut the 'improvements across all datasets' claim.","tokens_in":7708,"tokens_out":2772,"would_cite":false,"duration_ms":23272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim: a BIC-weighted penalty from several LLM-proposed causal graphs improves score-based causal discovery on benchmark datasets.","keywords":["causal discovery","large language models","score-based methods","prior knowledge","GES","NOTEARS","KCRL","BIC penalty"],"falsifier":"On a dataset with known ground truth, compute each LLM's BIC-derived weight $\\mu_m$ and its edge-level accuracy; if the most accurate LLM is not the highest-weighted one, or if re-fitting $\\lambda$ by data-only validation removes the reported gains, the penalty is not carrying the causal knowledge.","tokens_in":6755,"feed_emoji":"🤖","tokens_out":9153,"duration_ms":73834,"temperature":0.7,"pith_summary":"The paper tries to establish that score-based causal discovery can be improved by asking several large language models (LLMs) to propose causal graphs and adding a weighted penalty to the score function that pulls the final graph toward those proposals. The motivation is economic: expert priors and randomized trials are expensive, while LLM queries are cheap. The authors argue that no single LLM is reliable enough, so they weight each LLM's graph by the Bayesian Information Criterion (BIC) score of that graph on the observed data and sum the penalties. If the claim holds, any score-based discovery algorithm can accept LLM advice without retraining, replacing costly prior-knowledge pipelines with a prompt and a penalty term.","feed_headline":"Weighted LLM advice improves causal discovery benchmarks","feed_subtitle":"A BIC-weighted penalty from GPT-3.5, GPT-4, and Gemini can lift accuracy for GES, NOTEARS, and KCRL.","key_machinery":"The load-bearing mechanism is the penalty term itself. Each LLM is prompted with a three-stage protocol—variable understanding, causal statement, self-revision—to produce an adjacency matrix $\\hat{M}_m$. A quality weight $\\mu_m$ is computed by evaluating the BIC score of $\\hat{M}_m$ on the same dataset and normalizing the scores so they sum to one. The penalty $\\lambda\\sum_m \\mu_m\\|\\hat{M}-\\hat{M}_m\\|_p$ is then added to the base score, with $\\lambda$ expressing confidence in the LLM advice; decomposability of the $l_1$ form and differentiability of the $l_2$ form are what let existing search and optimization routines run unchanged.","core_discovery":"The paper's central claim is that adding a weighted penalty built from several LLM-proposed causal graphs to an existing score function improves score-based causal discovery. The augmented objective is $S(G)=S_{\\mathrm{score}}(G)+\\lambda\\sum_m \\mu_m\\|\\hat{M}-\\hat{M}_m\\|_p$, where $\\hat{M}_m$ is the adjacency matrix proposed by LLM $m$, $\\mu_m$ is that proposal's normalized BIC score on the observed data, and $p=1$ or $2$. The paper proves the $l_1$ penalty is node-wise decomposable and the $l_2$ penalty is differentiable, so the penalties can be dropped into GES and NOTEARS respectively, and it reports experiments with GES, NOTEARS, and KCRL on LUCAS, Asia, Earthquake, SACHS, and Child showing improved accuracy metrics, with TPR reaching 1.0 for enhanced GES on Asia and FDR reaching 0.0 for enhanced KCRL on three datasets.","pith_inferences":["A testable extension is to replace BIC-derived weights with weights learned from a small labeled validation set; if accuracy gains persist, the score-proxy assumption is not load-bearing.","The disagreement among LLM proposals could be converted into an uncertainty signal, treating edges all models agree on as near-certain priors and disputed edges as open.","The same penalty construction could seed constraint-based discovery, using high-confidence LLM edges as conditional-independence constraints instead of score penalties.","Because the paper's own results table shows some metrics moving against the enhancement, practical deployment would need automatic $\\lambda$ selection rather than a fixed confidence weight."],"forward_implications":["Any score-based causal discovery method can accept LLM advice as a plug-in penalty without retraining the underlying search, since the penalty is additive to the score.","The node-wise decomposability proof means the $l_1$ penalty extends beyond GES to every additive score-based search, not just the three algorithms tested.","The differentiable $l_2$ penalty lets continuous optimizers such as NOTEARS carry LLM advice through the same augmented-Lagrangian procedure used without it.","Combining several LLMs can recover correct edges that no single LLM finds, which the paper's case study illustrates by showing distinct correct predictions from GPT-3.5, GPT-4, and Gemini."],"supporting_citations":[{"why":"supplies the three-stage prompt used to elicit causal graphs from each LLM.","marker":"[2]"},{"why":"defines GES, the score-based algorithm whose greedy search needs the decomposable penalty.","marker":"[3]"},{"why":"defines NOTEARS, the continuous optimizer extended with the differentiable l2 penalty.","marker":"[20]"},{"why":"provides KCRL, the prior-knowledge reinforcement-learning framework whose expert prior is replaced by LLM knowledge.","marker":"[6]"},{"why":"shows LLMs can construct causal graphs, supporting the premise that LLM proposals can serve as priors.","marker":"[11]"},{"why":"treats LLMs as imperfect experts, the line of work the multi-LLM weighting scheme extends.","marker":"[10]"},{"why":"demonstrates injecting expert knowledge into NOTEARS, the pattern the penalty generalizes.","marker":"[5]"},{"why":"defines the BIC score used both as the base score and as the basis for LLM quality weights.","marker":"[4]"}],"fun_headline_variants":["Multiple LLMs boost causal discovery accuracy","BIC-weighted LLM penalties refine causal graphs","LLM ensemble improves score-based causal discovery","GPT-3.5, GPT-4, Gemini help causal discovery","Multi-LLM advice lifts causal discovery performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM graph's BIC score on the observed data measures how much that LLM should be trusted, and that one penalty weight $\\lambda$ can be set without tuning on the ground-truth graph.","fun_headline_variants_meta":{"raw":{"variants":["Multiple LLMs boost causal discovery accuracy","BIC-weighted LLM penalties refine causal graphs","LLM ensemble improves score-based causal discovery","GPT-3.5, GPT-4, Gemini help causal discovery","Multi-LLM advice lifts causal discovery performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1340,"prompt_tokens":889,"completion_tokens":451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":505,"tokens_out":451,"duration_ms":4418,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:36:53.313668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset with known ground truth, compute each LLM's BIC-derived weight $\\mu_m$ and its edge-level accuracy; if the most accurate LLM is not the highest-weighted one, or if re-fitting $\\lambda$ by data-only validation removes the reported gains, the penalty is not carrying the causal knowledge.","supporting_citations":[{"cited_title":"CoRR (2023)","cited_arxiv_id":null,"evidence_quote":"supplies the three-stage prompt used to elicit causal graphs from each LLM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines GES, the score-based algorithm whose greedy search needs the decomposable penalty."},{"cited_title":"In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Sys- tems 2018 (2018)","cited_arxiv_id":null,"evidence_quote":"defines NOTEARS, the continuous optimizer extended with the differentiable l2 penalty."},{"cited_title":"In: Proceedings of the Machine Learning for Healthcare Conference, MLHC 2022","cited_arxiv_id":null,"evidence_quote":"provides KCRL, the prior-knowledge reinforcement-learning framework whose expert prior is replaced by LLM knowledge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"shows LLMs can construct causal graphs, supporting the premise that LLM proposals can serve as priors."},{"cited_title":"CoRR (2023)","cited_arxiv_id":null,"evidence_quote":"treats LLMs as imperfect experts, the line of work the multi-LLM weighting scheme extends."},{"cited_title":"In: Proceedings of the 12th Interna- tional Conference on Pattern Recognition Applications and Methods, ICPRAM","cited_arxiv_id":null,"evidence_quote":"demonstrates injecting expert knowledge into NOTEARS, the pattern the penalty generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the BIC score used both as the base score and as the basis for LLM quality weights."}],"review_version":1}