{"id":"4d378164-7c8f-444d-8dbc-54d2cbc7c3ad","arxiv_id":"2507.20349","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A supervised graph neural network, fed with 114 statistical and information-theoretic edge features, predicts causal graphs and enforces acyclicity via post-hoc probabilistic inference.","lead":"Researchers trained a graph neural network to predict which variables cause which others, using 114 hand-crafted statistical and information-theoretic features per variable pair, then converted the predictions into a causal graph. The paper claims top accuracy on synthetic and real data, but its own tables show several baselines matching or beating the method, and no code is released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 in Section 4.3 contradicts the abstract's 'superior performance' claim: NOTEARS and DAGMA match or beat the proposed GNN methods on synthetic ER/SF graphs in both SHD and TPR.","rationale":"The reader's weakest_assumption concerns generalization from synthetic training data to CSuite and Sachs real-world datasets. That is a legitimate secondary concern, especially given the close-world training/test setup and the lack of error bars in Tables 2 and 3. However, the more load-bearing problem is earlier and more direct: the paper's own synthetic results, which are the primary evidence for the headline claim, already contradict the abstract's unqualified 'superior performance' statement. NOTEARS and DAGMA match or beat GNN-MLDAG and GNN-PDAG on SHD/d and TPR in the ER and SF conditions, so the central claim is not supported even before considering real-world transfer. The reader did mention this contradiction in the rationale, but did not identify it as the weakest assumption; I therefore mark disagreement on that specific field. My recommendation does not change the reader's REJECT verdict: the evidence as presented does not support the claimed contribution, and the contradiction in Table 1 is sufficient on its own to require major revision. I am not disputing that a supervised GNN approach to causal discovery is plausible, nor that the components reused from Rashid et al. (2022) are transparently acknowledged; the issue is that the empirical support does not match the strength of the claims.","tokens_in":14205,"tokens_out":6165,"duration_ms":63587,"concrete_test":"Obtain the per-graph SHD, TPR, and FPR values for the 80 ER and 80 SF test graphs used in Table 1 and run paired comparisons (e.g., Wilcoxon signed-rank or paired bootstrap) between GNN-MLDAG/GNN-PDAG and NOTEARS/DAGMA. If NOTEARS has significantly lower SHD on ER graphs, or DAGMA has significantly higher TPR on SF graphs, the abstract's unqualified 'superior performance' claim should be retracted or narrowed to specific regimes. If the paired tests show GNN methods dominate on all primary metrics, the concern would be resolved. Also verify that the reported means and standard errors in Table 1 are correctly computed from these per-graph values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is that the proposed framework 'outperforms both traditional and recent non-GNN-based methods, as well as a GNN-based approach, in terms of accuracy and scalability on synthetic and real-world datasets.' Table 1, the primary synthetic evaluation, does not support this. On Erdos-Renyi graphs, NOTEARS achieves SHD/d = 1.33 ± 0.10, while GNN-MLDAG achieves 1.66 ± 0.15 and GNN-PDAG achieves 1.75 ± 0.11; NOTEARS also has a higher TPR (0.58 ± 0.02 vs. 0.54 ± 0.03 for GNN-MLDAG). On Scale-Free graphs, NOTEARS (1.36 ± 0.11) and DAGMA (1.39 ± 0.09) have equal or lower SHD/d than GNN-MLDAG (1.40 ± 0.11), and DAGMA has higher TPR (0.54 ± 0.02 vs. 0.48 ± 0.03). Since the paper itself states that lower SHD and higher TPR are better, the synthetic results fail to establish the advertised superiority over per-dataset optimization methods. This is a direct, internal contradiction, not a question of generalization to real-world data: even under the closed-world synthetic benchmark, the proposed method is not consistently better than the baselines it claims to beat. The absence of per-graph raw results and error bars for Tables 2 and 3 further weakens the real-world claims, but the synthetic contradiction alone is sufficient to invalidate the unqualified 'superior performance' claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a supervised graph neural network framework for causal discovery. A fully connected graph is built over the observed variables; each node is given 13 statistical features and each edge 114 statistical, information-theoretic, and causal-pairs features. A GraphSAGE-style message-passing model with edge features is trained on synthetic graphs to classify every pair as forward edge, reverse edge, or no edge. Four inference strategies (PG, MLG, PDAG, MLDAG) from the authors' earlier work turn these edge probabilities into graphs or DAGs. The framework is evaluated without per-dataset retraining on synthetic ER and Scale-Free graphs, five CSuite benchmarks, and the Sachs protein network, and compared against PC, GES, LiNGAM, DAG-GNN, NOTEARS-MLP, DiBS, and DAGMA. The abstract claims superior accuracy and scalability over all these methods.","tokens_in":14537,"tokens_out":11619,"duration_ms":113495,"significance":"If the claims were supported, the contribution would be practically valuable: a single model that amortizes causal discovery, provides probabilistic outputs, and avoids per-dataset optimization. The paper is commendably concrete about its feature set, training distribution, and backbone, which makes the method reproducible in principle, and it evaluates on a broad set of standard baselines. The core empirical claim, however, is not supported by the paper's own Table 1, and the real-world evaluation is not yet rigorous enough to establish generalization. The significance is therefore conditional on a substantial revision that either demonstrates superiority in well-defined settings or reframes the contribution as a competitive amortized alternative.","major_comments":[{"comment":"The central claim of 'superior performance' is contradicted by the paper's own table. On Erdos-Renyi graphs, NOTEARS achieves SHD/d = 1.33 +/- 0.10 and TPR = 0.58 +/- 0.02, while GNN-MLDAG achieves SHD/d = 1.66 +/- 0.15 and TPR = 0.54 +/- 0.03; on Scale-Free graphs, DAGMA achieves SHD/d = 1.39 +/- 0.09 and TPR = 0.54 +/- 0.02 versus GNN-MLDAG's 1.40 +/- 0.11 and 0.48 +/- 0.03, and NOTEARS has lower SHD/d (1.36 +/- 0.11). Because the paper defines lower SHD and higher TPR as better, these numbers show that the proposed GNN variants are not consistently superior even under in-distribution synthetic data. The abstract, Section 1, and the first paragraph of Section 4.3 need to be rewritten, or the authors need paired per-graph statistical tests that identify the specific settings in which the GNN methods beat the baselines.","section":"Section 4.3, Table 1, and abstract"},{"comment":"The novelty claim is confounded with self-reference. Three of the 114 edge features are the causal-pairs direction probabilities from the authors' own Rashid et al. (2022), and the inference procedures PG, MLG, PDAG, and MLDAG (Eqs. 5-10) are explicitly taken from that same paper, with the reader referred there for 'detailed algorithmic derivations and proofs.' The evaluation then compares the GNN framework against CausalPairs as if it were an independent baseline. The comparison to CausalPairs is best understood as an ablation of the GNN aggregation layer, not as a comparison against an unrelated method; this should be stated explicitly, and the paper should delineate which components are new.","section":"Sections 3.1 and 3.3"},{"comment":"The real-world evidence is not sufficient to support the claimed generalization. Table 2 reports one SHD/d, TPR, and FPR value per dataset with no repeated runs or error bars; Table 3 switches to Predicted/Correct/Reversed edge counts, a different metric family, and for non-standardized data the DAG-GNN and NOTEARS-MLP results are quoted from their original manuscripts rather than produced under the same pipeline. The conclusion that the model 'effectively generalizes to real-world datasets' is therefore not supported by the tables as they stand. The authors should add repeated runs with matched preprocessing, report consistent metrics, and document the distribution shift between training and test.","section":"Section 4.3, Tables 2 and 3"},{"comment":"The maximum likelihood topological ordering used by PDAG and MLDAG is approximated as the topological sort of the maximum spanning DAG of the edge-probability graph, and no analysis or empirical check of this approximation is provided. Since PDAG/MLDAG are the variants highlighted as the best performers, this step is load-bearing; the paper should either supply a proof or a direct comparison with a search-based ordering, or explicitly report the sensitivity of the final DAG to this approximation.","section":"Section 3.3, Eq. 9"},{"comment":"The synthetic test data in Table 1 come from the same generative family as the training data (graph models ER/SF, d in {10,20,50,100}, edge ratios {1d,2d,4d}, sample sizes {500,1000,2000}, nonlinear SEMs 'similar to the NOTEARS-MLP implementation'), so Table 1 only demonstrates interpolation in distribution. The headline generalization to CSuite and Sachs is an out-of-distribution claim, but Tables 2 and 3 are the only evidence and, as noted above, are not yet adequate. Please add held-out synthetic families, larger-d or different-n test cases, or otherwise restrict the generalization claim.","section":"Section 4.1"}],"minor_comments":[{"comment":"The symbol e_ij is overloaded: it denotes both the categorical edge label (values -1, 0, 1) and the edge feature vector, making the conditioning in p(e_ij | h_i, h_j, e_ij) confusing; use different symbols (e.g., y_ij for the label and x_ij for the feature vector).","section":"Equation 1"},{"comment":"The list of CSuite datasets ends with 'symprod simpson (4 nodes, 4 edges);.'; remove the stray semicolon or period.","section":"Section 4.1"},{"comment":"The text says the full feature list is in 'Appendix 5,' but the appendix is unnumbered; reference it properly (e.g., Appendix A) and include a table that explicitly sums the 114 edge features, since the current grouped list does not make the count verifiable.","section":"Section 3.1 and Appendix"},{"comment":"The figure uses 'Normalized TPR,' 'Normalized FPR,' and 'Normalized SHD/d' without defining the normalization; define it in the caption so the reader can interpret the magnitudes.","section":"Figure 2"},{"comment":"The phrase 'trained on synthetic train data' is redundant; use 'trained on synthetic data.'","section":"Table 2 caption"},{"comment":"The paper does not include a data or code availability statement; providing the trained model and generation scripts would materially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This manuscript has a serious mismatch between its claims and its own Table 1, and the real-world evaluation needs rework. The self-citation pattern (Rashid et al. 2022 supplying both the inference engine and three edge features, with CausalPairs as a baseline) should be scrutinized during revision. I do not see a fatal flaw in the general idea, so I am recommending major revision rather than rejection, but the revision must be substantial: the authors need to either prove superiority in well-defined regimes with paired tests or honestly reposition the paper as an amortized, uncertainty-aware alternative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a serious empirical paper, but the headline claim is wrong. On their own synthetic benchmark (Table 1), NOTEARS beats GNN-MLDAG on ER graphs (SHD/d 1.33 vs 1.66, TPR 0.58 vs 0.54), and on SF graphs DAGMA and NOTEARS match or beat it. The abstract's unqualified \"superior performance\" claim is not supported by the primary evidence. The stress-test note is correct.\n\nWhat's actually new: they train a GraphSAGE edge classifier on a rich feature set (114 edge features, including their own CausalPairs probabilities) to predict edge direction probabilities, then apply the four inference procedures from Rashid et al. (2022). That's a legitimate engineering contribution: a one-pass, amortized causal discovery model with uncertainty estimates, trained once on synthetic data. The CSuite and Sachs results are often competitive, and the robustness to data scaling is a real advantage over NOTEARS/DAG-GNN.\n\nSoft spots: First, the internal contradiction above is load-bearing. You can't claim to outperform NOTEARS and DAGMA when Table 1 shows otherwise. Second, the inference stage and three of the 114 edge features come from their own 2022 paper; that's not a flaw per se, but it concentrates the novelty in the GNN edge classifier. The comparison against CausalPairs is partly a self-comparison, which is fine for an ablation but shouldn't be sold as a benchmark win. Third, no code or data is released, and Tables 2 and 3 have no error bars, so the real-world numbers are hard to verify. Fourth, the synthetic training and test sets come from the same NOTEARS-MLP-style generator, so the generalization to CSuite and Sachs is genuinely informative but weaker than the \"diverse synthetic datasets\" phrasing suggests.\n\nWho is this for? Researchers working on amortized or supervised causal discovery will find the feature engineering and GNN architecture useful. The probabilistic inference layer is a reasonable way to get uncertainty estimates. But the paper needs honest claims and more evidence before it can be trusted.\n\nRecommendation: It deserves peer review, but only with major revisions. I would ask the authors to recalibrate the claims, release code, and add per-graph error bars. As submitted, I would not accept.\n\nBest,\n\n[Your name]","headline":"A plausible amortized GNN causal discovery pipeline, but the paper's central 'superior performance' claim is contradicted by its own Table 1.","tokens_in":15138,"tokens_out":1892,"would_cite":false,"duration_ms":18106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph neural network trained once on synthetic data can predict the causal graph of a new observational dataset in a single forward pass, returning edge probabilities rather than a single deterministic structure.","keywords":["causal discovery","graph neural network","directed acyclic graph","probabilistic inference","supervised edge classification","structural equation models","GraphSAGE","causal pairs"],"falsifier":"Hold out a real dataset whose true causal graph is known but whose data were produced by a very different mechanism than the synthetic generator, such as discrete variables, heavy measurement noise, or latent confounders; run the trained model and per-dataset baselines on it. If the model's advantage in SHD and true-positive rate disappears or reverses, the paper's generalization claim is falsified.","tokens_in":13887,"feed_emoji":"🔗","tokens_out":8888,"duration_ms":81396,"temperature":0.7,"pith_summary":"The paper's central claim is that causal discovery can be recast as a supervised edge-classification problem. A graph neural network, trained once on synthetic graphs generated from nonlinear structural equation models, takes the observed variables of a new dataset and outputs a probability for every ordered pair of variables being cause, effect, or unrelated. Because the output is a distribution over graphs rather than a single graph, the model carries uncertainty information, and acyclicity can be imposed afterward via topological ordering. The paper reports that this trained-once model matches or beats classical and recent optimization-based causal discovery methods on synthetic data, on the CSuite benchmarks, and on the Sachs protein-signaling data without retraining. A reader would care because it replaces per-dataset combinatorial or continuous search with a single fast prediction.","feed_headline":"One trained GNN beats per-dataset causal search without retraining","feed_subtitle":"Edge-by-edge probabilities replace combinatorial search; wins on synthetic, CSuite, and protein-network data.","key_machinery":"Central object: a GraphSAGE-style message-passing model extended to consume edge features, acting as an edge classifier. Each node is initialized with 13 statistical features; each edge with 114 features that condense pairwise dependence, including mutual information, conditional entropy, polynomial-fit error, Pearson correlation, HSIC, and a prior edge-direction distribution from a causal-pairs model. Message passing concatenates source, target, and edge features, averages the messages over neighbors, and updates node embeddings; the final per-edge probability comes from a function f([h_i, h_j, e_ij]). A secondary mechanism is the four inference strategies (PG, MLG, PDAG, MLDAG) that convert edge probabilities into graphs, with the DAG variants using a maximum spanning DAG plus topological sorting to enforce acyclicity.","core_discovery":"The paper argues that the structural equations underlying a causal system leave measurable footprints in the joint distribution of the observed variables, and that those footprints are enough for a graph neural network to classify edge direction and existence directly. Concretely, it constructs a fully connected graph over variables, decorates each node with 13 distributional features and each edge with 114 statistical and information-theoretic features, then learns a function from concatenated node and edge embeddings to the probability of the edge being forward, reverse, or absent. Under the paper's four inference strategies, these edge probabilities form a full distribution over digraphs, or are refined into maximum-likelihood DAGs by conditioning on a topological order from a maximum spanning DAG. The reported results place the DAG-refined variants, GNN-PDAG and GNN-MLDAG, ahead of the baselines on most synthetic and CSuite comparisons and competitive on the Sachs network.","pith_inferences":["The edge-probability output could be calibrated and used as a prior for interventional or counterfactual reasoning, but the paper does not test this downstream use.","A natural stress test is to train the same architecture on a deliberately shifted synthetic distribution, such as different SEM families, discrete variables, or latent confounders, and measure the change in SHD; the paper only reports one training distribution.","Because the feature list is explicit and fixed, the framework could double as a probe for which statistical signatures actually drive causal direction, connecting to interpretability without any architectural change.","The reported insensitivity to data standardization suggests the model relies on scale-invariant features, a property that could be verified by ablating the moment- and variance-based edge features."],"forward_implications":["Once trained, the model returns edge probabilities for a new dataset in a single forward pass, removing the need for per-dataset optimization or conditional-independence testing.","Acyclicity is handled downstream rather than during optimization, which lets the same learned edge probabilities serve both general digraph queries and strict DAG estimates.","The probabilistic output makes uncertainty quantification available to downstream users, who can sample graphs or rank edges by confidence.","The method scales to at least 100 nodes in training, and its reported advantage grows on denser graphs, pointing to a scalable route for complex causal systems.","Because the same architecture produces all four inference variants, practitioners can trade off precision against recall without retraining."],"supporting_citations":[{"why":"Provides the NOTEARS-MLP baseline and the nonlinear SEM data-generation recipe used to create the training and synthetic test sets.","marker":"(Zheng et al., 2020)"},{"why":"Supplies the causal-pairs edge-direction prior and the four inference strategies (PG, MLG, PDAG, MLDAG) that convert edge probabilities into graphs.","marker":"(Rashid et al., 2022)"},{"why":"Provides the GraphSAGE message-passing backbone, extended in this paper to consume edge features.","marker":"(Hamilton et al., 2017)"},{"why":"Defines the five CSuite benchmark SEMs the paper uses to test generalization without retraining.","marker":"(Geffner et al., 2022)"},{"why":"Supplies the real-world protein-signaling dataset and its consensus ground-truth graph.","marker":"(Sachs et al., 2005)"},{"why":"The GNN-based DAG-GNN baseline, contrasted as a deterministic structure learner against the paper's probabilistic approach.","marker":"(Yu et al., 2019)"},{"why":"The constraint-based PC algorithm used as a classical baseline in all comparisons.","marker":"(Spirtes et al., 2001)"},{"why":"The score-based GES algorithm used as a classical baseline in all comparisons.","marker":"(Chickering, 2002)"}],"fun_headline_variants":["GNN turns causal discovery into edge-probability prediction","One trained GNN outperforms per-dataset causal search","Probabilistic GNN predicts causal graphs without retraining","Edge probabilities from GNN beat combinatorial causal search","Causal discovery via GNN: probabilistic, scalable, accurate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole transfer rests on the assumption that the synthetic data generator used for training, with its specific graph sizes, edge densities, and sample sizes, produces statistical footprints representative enough of real-world causal systems that a model trained on it keeps its accuracy on unseen real data.","fun_headline_variants_meta":{"raw":{"variants":["GNN turns causal discovery into edge-probability prediction","One trained GNN outperforms per-dataset causal search","Probabilistic GNN predicts causal graphs without retraining","Edge probabilities from GNN beat combinatorial causal search","Causal discovery via GNN: probabilistic, scalable, accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1427,"prompt_tokens":926,"completion_tokens":501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":421}},"tokens_in":542,"tokens_out":501,"duration_ms":4634,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:45:08.586885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a real dataset whose true causal graph is known but whose data were produced by a very different mechanism than the synthetic generator, such as discrete variables, heavy measurement noise, or latent confounders; run the trained model and per-dataset baselines on it. If the model's advantage in SHD and true-positive rate disappears or reverses, the paper's generalization claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NOTEARS-MLP baseline and the nonlinear SEM data-generation recipe used to create the training and synthetic test sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the causal-pairs edge-direction prior and the four inference strategies (PG, MLG, PDAG, MLDAG) that convert edge probabilities into graphs."}],"review_version":2}