{"id":"b3c53656-40f2-4558-aa23-6b911d5a6160","arxiv_id":"2412.10039","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Causal discovery evaluations should report negative control baselines, because common metrics can look good under random guessing.","lead":"The paper proposes that causal discovery algorithms should be evaluated against a negative control baseline of random guessing, and it derives exact formulas for what common metrics like precision and recall look like when an algorithm is just guessing. It then presents a statistical test and a simulation pipeline for deciding whether an algorithm actually beats that baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The reader's weakest assumption names the uniform random edge placement. I agree this is the standing assumption, but the paper explicitly conditions on it and does not overclaim generality. The exact distributional results are mathematically sound; the simulation-based pipeline is clearly heuristic and properly labeled. The only potentially misleading aspect is that the Section 5 test is specific to a uniform-subset null and may reject algorithms with non-uniform structural biases, but this is a caveat the paper partially anticipates. No change to the accept verdict is warranted.","tokens_in":15184,"tokens_out":12400,"duration_ms":143344,"concrete_test":"Run a Monte Carlo simulation with 100,000 replicates for, e.g., (d, mtrue, mest) = (10, 30, 15) and (10, 30, 30): draw mest edges uniformly from the mmax possible edges conditional on a fixed true graph, compute TP, and compare the empirical mean and 2.5%/97.5% quantiles of precision and recall to the Table 3 formulas. If the empirical values agree within Monte Carlo error, the distributional core is verified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No significant objection identified. The central claim is a conditional statement: under the explicit null model of uniformly random edge placement, TP follows HyperGeom(mmax, mtrue, mest), and the Table 3 expectations follow by linearity. The derivation is correct and does not require the true graph to be random. The uniform-null assumption is the key limitation, but it is transparently stated and the paper offers a simulation-based negative control (Section 6) for other graph families. The exact test is conservative by design. I do not see an internal inconsistency or missing step that would threaten the main conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that evaluations of causal discovery algorithms should include negative controls (random guessing) as a common baseline, and it provides the mathematical and methodological tools to do so. For the task of graph skeleton estimation, it derives exact distributional results: conditional on the maximal possible number of edges mmax, the true number of edges mtrue, and the estimated number of edges mest, the number of true positives under random edge placement follows a hypergeometric distribution. From this it obtains expected values and quantiles for precision, recall, F1, negative predictive value, and specificity, showing that these metrics can take surprisingly favorable values under random guessing when the true and/or estimated graphs are dense. The paper proposes an exact test of overall skeleton fit and applies it to a temporal PC analysis of a real cohort dataset. It further proposes a simulation-based negative control pipeline for more general metrics (e.g., structural Hamming distance, orientation measures, SID), demonstrated on a PC simulation study and on the Sachs protein-signaling data.","tokens_in":15233,"tokens_out":20894,"duration_ms":226596,"significance":"If the claims are correct, this is a valuable and practically relevant contribution to the causal discovery evaluation literature. It gives a simple, principled baseline and quantifies the often unappreciated fact that raw precision/recall values are uninterpretable without a random-guessing reference. The hypergeometric derivation is correct, clearly presented, and exact under the stated model; the paper is transparent about the key assumptions (fixed mmax, mtrue, mest and uniform edge placement) and about the limitations of the Erdős–Rényi-type null. The emphasis on reporting distributional summaries and on comparing algorithms only after matching edge count is sound advice that is likely to influence evaluation practice. The main weakness is in the presentation of the simulation-based p-value in Section 6, which is internally inconsistent and needs to be fixed before the pipeline can be reproduced or interpreted reliably.","major_comments":[{"comment":"The definition of the p-value for the simulation-based negative control is internally inconsistent and cannot reproduce the values reported in Tables 5 and 6. In Section 6, the p-value is defined as the proportion p = (1/b) * sum_i 1{f(G_i_true, Ghat_i_algo) <= f(G_i_true, Ghat_i_NC)} for metrics where small values are favorable, while Supplementary C uses the opposite inequality for the same metric (SHD) and reports p = 0.202. The reported values such as p = 0.002 in the sparse case and p = 0.001 for PC in Table 6 are not proportions of pairs where the algorithm performs at least as well as the negative control; if the algorithm is better, that proportion would be near 1, not 0.001. These numbers are instead consistent with a binomial tail probability P(T >= t_obs) under a null of exchangeability, but that is not what the displayed formula gives. Moreover, the conclusion in Section 6.1 that PC is 'not significantly different' in the dense case does not follow from the formula as written, since a proportion of 0.202 would be far below 0.5 and would indicate the opposite direction. The authors should state explicitly whether the reported p is a test statistic or a p-value, give the correct formula for the p-value (e.g., the binomial tail probability under the null), and verify that the code and the Supplementary Materials use the same definition.","section":"Section 6 and Supplementary C"},{"comment":"The procedure says to draw 'a negative control random DAG' but does not specify the distribution over DAGs with a given edge count. To be consistent with the exact results of Sections 3–5, the negative control graph should place its mest edges uniformly at random among all possible edges (that is, a uniform random skeleton conditional on the edge count, with an acyclic orientation). If instead one draws uniformly from the set of all DAGs with mest edges, the skeleton is not uniformly distributed over all m-edge subsets, and the negative control distribution will differ from the hypergeometric null even when the algorithm has no information. The paper should state explicitly how the negative control graphs are generated and, if a different graph-generating process is intended, clarify that the exact distributional results of Sections 3–5 apply only to the uniform-edge-placement null.","section":"Section 6, Step 2"}],"minor_comments":[{"comment":"The intervals labelled '95% CI' are empirical quantile intervals for the distribution of each metric, not confidence intervals for a parameter; consider relabelling them as 'empirical 95% intervals' or similar to avoid ambiguity.","section":"Table 5, notes"},{"comment":"There is a typographical error in the phrase 'how often the discovery algortihm' — 'algortihm' should be 'algorithm'.","section":"Section 6.2"},{"comment":"The sentence 'the expected precision does not depend on the number of edges in the estimated graph' is correct under the conditional hypergeometric model but could be misread as an unconditional statement; a short reminder that this holds conditional on (mmax, mtrue, mest) would improve clarity.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The reader's take did not flag the p-value inconsistency in Section 6, but it is a genuine load-bearing issue for the proposed simulation-based pipeline. The central hypergeometric derivation is sound and the paper is likely to be a useful contribution once the statistical test in Section 6 is correctly described and the supplementary example is brought into line with the main text. If the GitHub code already implements a valid test, the revision is straightforward; otherwise the authors need to recompute the reported p-values."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a clean, useful paper. If you work on causal discovery evaluation, you should know it: the authors derive the exact distribution of true positives when an estimated skeleton is a uniform random draw of m_est edges among all possible, which is hypergeometric, and show that expected precision equals the true graph density and expected recall equals the estimated graph density. The warning is real: a random guess can score F1 above 0.8 on dense graphs, so reporting precision and recall without a baseline is close to meaningless.\n\nWhat is actually new is the systematic packaging: expectations and quantiles for five adjacency metrics, an exact test of overall skeleton fit, and a simulation-based pipeline that extends the negative control idea to oriented metrics and SID. The derivations in Section 4 and the supplement are correct and clearly presented. The examples—the dense 5-node DAG, the PC simulation, and the Sachs comparison—are well chosen and illustrate the main points. The authors are transparent about the key assumption: the hypergeometric null requires uniform random edge placement, and they note that real algorithms have structural biases. They offer the simulation pipeline for other graph families, which is the right fix.\n\nSoft spots, in proportion: the null is narrow. Rejecting it only tells you the algorithm's edge set is not an iid uniform draw; an algorithm that always outputs a star graph or a chain will be \"better than random\" even if it is useless for causal structure. That is not a flaw in the math, but it means the negative control is a very weak floor, and the paper could have said so a bit more sharply. The empirical \"confidence intervals\" in Table 5 are actually empirical quantile ranges, not confidence intervals—a minor wording issue. The exact test is conservative by design, which they acknowledge; for small mmax it may have little power. Conditioning on mest is debatable but they justify it well. Nothing there threatens the central claim.\n\nWho is this for? Anyone who reviews or designs simulation studies in causal discovery. It deserves a serious referee and likely publication with only light revision. I would want the authors to tighten the wording on the intervals and perhaps add a sentence on the limitations of the uniform null, but the core is solid.","headline":"A clean, correct, and practically useful argument that causal discovery evaluations should report negative control baselines; the hypergeometric derivations are right and the warning about interpreting precision and recall is well supported.","tokens_in":15720,"tokens_out":2653,"would_cite":true,"duration_ms":29861,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that standard causal discovery metrics like precision and recall can be matched by random guessing, and it proposes negative controls as an evaluation baseline.","keywords":["causal discovery","negative controls","random guessing baseline","hypergeometric distribution","skeleton estimation","evaluation metrics","precision and recall","exact test"],"falsifier":"Simulate many random true DAGs with a fixed number of edges and an estimator that always outputs the same fixed skeleton, such as a star graph, with the same edge count; the observed true-positive counts will not follow $\\mathrm{HyperGeom}(m_{\\max}, m_{\\mathrm{true}}, m_{\\mathrm{est}})$, and the proposed exact test will not hold its nominal level, showing the null applies only to genuinely uniform random placement.","tokens_in":14985,"feed_emoji":"🎲","tokens_out":12649,"duration_ms":122121,"temperature":0.7,"pith_summary":"The paper argues that standard causal discovery evaluation metrics—precision, recall, and F1—can be deceptively high under random guessing, so reported numbers mean little without a random baseline. It proves that if an estimated skeleton places its edges uniformly at random, the true-positive count follows an exact hypergeometric distribution, giving expected precision equal to the true graph's density and expected recall equal to the estimate's density. As a result, dense true graphs let a completely uninformative estimate earn high precision, and adding many edges inflates recall and F1. The paper proposes negative controls as a required benchmark, constructs an exact test of skeleton fit, and provides a simulation-based pipeline for metrics without exact null distributions.","feed_headline":"Chance can hit high causal discovery scores—use negative controls","feed_subtitle":"A hypergeometric baseline shows when precision and recall reflect chance; report it with every benchmark.","key_machinery":"The load-bearing object is the hypergeometric null for true positives, $\\mathrm{HyperGeom}(m_{\\max}, m_{\\mathrm{true}}, m_{\\mathrm{est}})$, obtained by conditioning on the fixed margins of the adjacency confusion matrix when edges are placed uniformly at random. This single distribution carries the argument because every adjacency metric considered is a linear function of the true-positive count, so expectations, quantiles, confidence intervals, and the one-sided skeleton-fit test all follow from it. For metrics outside this class—orientation precision and recall, structural Hamming distance, structural intervention distance—the paper substitutes a simulation-based negative-control pipeline: draw random graphs matched to the algorithm's edge-count distribution, evaluate the metric, and compare pairwise with a one-sided test.","core_discovery":"Conditional on the maximal number of possible edges $m_{\\max}$, the true edge count $m_{\\mathrm{true}}$, and the estimated edge count $m_{\\mathrm{est}}$, the paper shows that the number of true positives in an estimated skeleton placed uniformly at random is exactly distributed as $\\mathrm{HyperGeom}(m_{\\max}, m_{\\mathrm{true}}, m_{\\mathrm{est}})$. Because precision, recall, F1, negative predictive value, and specificity are linear functions of the true-positive count given these margins, the paper derives their exact expectations and quantiles under this null: expected precision is $m_{\\mathrm{true}}/m_{\\max}$, expected recall is $m_{\\mathrm{est}}/m_{\\max}$, and expected F1 is $2 m_{\\mathrm{est}} m_{\\mathrm{true}} / (m_{\\max}(m_{\\mathrm{est}}+m_{\\mathrm{true}}))$. This yields a warning that apparently strong metric values can arise from chance, an exact one-sided test for whether a skeleton is better than random, and, for metrics that cannot be expressed through the adjacency confusion matrix, a simulation-based negative-control pipeline matching the algorithm's edge-count distribution.","pith_inferences":["The negative-control principle transfers to any structured-output prediction task—such as community detection, network reconstruction, or feature-set selection—where precision and recall are reported; a random baseline should become part of the evaluation protocol.","For small graphs with small $m_{\\max}$, the exact test is conservative and borderline p-values should not be over-read; this follows from the discreteness the paper notes, but the practical warning is our emphasis.","A reader can use the paper's formulas to compute a chance ceiling for any previously reported precision; some existing benchmark claims may not outperform random guessing once graph density is accounted for."],"forward_implications":["Report a negative-control value whenever precision, recall, or F1 is used to evaluate a causal discovery algorithm; without it, high values may reflect graph density rather than skill.","When comparing two algorithms, the comparison is interpretable only if the outputs have similar edge counts or if each is judged against its own negative control, since dense outputs can look better without being more informative.","The exact hypergeometric test gives a calibrated p-value for 'better than random' skeleton recovery, and the paper demonstrates both rejection (TPC on the Metropolit cohort, p = 0.002) and failure to reject for several algorithms on the Sachs data.","For metrics beyond adjacency, the simulation-based pipeline provides an empirical chance distribution, and the paper's PC simulation shows that in dense settings even a sound algorithm may not separate from random guessing.","Ranking algorithms by negative-control p-values can differ from ranking by raw SHD, because the latter rewards sparsity regardless of correctness, as the Sachs example shows."],"supporting_citations":[{"why":"Supplies the expert-model versus TPC confusion matrix on the Metropolit cohort that the exact skeleton-fit test is applied to.","marker":"Petersen et al. [2023a]"},{"why":"Supplies the real proteomics dataset and 20-edge ground truth used to benchmark five causal discovery algorithms against negative controls.","marker":"[Sachs et al., 2005]"},{"why":"Defines the PC algorithm that is the main subject of the simulation study and one of the Sachs benchmarks.","marker":"[Spirtes and Glymour, 1991]"},{"why":"Defines NOTEARS and provides the Sachs evaluation whose reported SHD and edge count are reanalyzed against negative controls.","marker":"[Zheng et al., 2018]"},{"why":"Defines the structural intervention distance used as a metric in the simulation-based negative-control pipeline.","marker":"[Peters and Bühlmann, 2015]"},{"why":"Defines the structural Hamming distance used to evaluate skeleton fits in the simulation and Sachs applications.","marker":"[Tsamardinos et al., 2006]"},{"why":"Documents misleading use of evaluation metrics and the density-learning flaw that motivates conditioning on edge counts.","marker":"[Petersen et al., 2023b]"},{"why":"Defines GES, one of the algorithms benchmarked against negative controls on the Sachs data.","marker":"[Chickering, 2002]"}],"fun_headline_variants":["Random guessing can top causal discovery metrics","Is your causal discovery just random luck?","Negative controls: The cure for chance-score claims","Causal discovery scores can deceive—use negative controls","Chance can mimic good causal discovery—test it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a random guess places its $m_{\\mathrm{est}}$ edges uniformly among all $m_{\\max}$ possible edges, so conditioning on the edge counts fully describes guessing behavior; algorithms whose outputs have extra structure, such as preferred degrees, clustering, or acyclicity constraints, may not be faithfully benchmarked by that null.","fun_headline_variants_meta":{"raw":{"variants":["Random guessing can top causal discovery metrics","Is your causal discovery just random luck?","Negative controls: The cure for chance-score claims","Causal discovery scores can deceive—use negative controls","Chance can mimic good causal discovery—test it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000361,"raw_usage":{"total_tokens":1964,"prompt_tokens":970,"completion_tokens":994,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":924}},"tokens_in":586,"tokens_out":994,"duration_ms":9707,"temperature":1.0,"reasoning_tokens":924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:24:13.439067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate many random true DAGs with a fixed number of edges and an estimator that always outputs the same fixed skeleton, such as a star graph, with the same edge count; the observed true-positive counts will not follow $\\mathrm{HyperGeom}(m_{\\max}, m_{\\mathrm{true}}, m_{\\mathrm{est}})$, and the proposed exact test will not hold its nominal level, showing the null applies only to genuinely uniform random placement.","supporting_citations":[],"review_version":1}