{"id":"0ecdae2a-e482-415f-b3e5-908ad2c9ffc7","arxiv_id":"2608.00198","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An auditable consensus workflow around four causal-discovery methods gives higher-precision links on synthetic benchmarks, but the precision gain disappears on a real river network where the reference graph is incomplete.","lead":"AutoCause is an open-source Python workflow that records and automates expert choices in time-series causal discovery, wrapping four existing methods and grading links by how many methods agree. Its benchmark on 145 datasets finds that majority-supported links are more precise on synthetic data, but not against real river topology, showing consensus is a priority signal, not proof.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CausalRivers tier-precision inversion rests on 6 and 7 links with no uncertainty, so the headline's 'not against river topology' is statistically unsupported.","rationale":"The paper's primary contribution is a carefully documented, open-source framework that records and automates the six expert decisions around established causal-discovery methods. The software artifacts, reproducibility claims, and honest limitation sections are strong, and the synthetic-benchmark consensus-precision results (DGP-Atlas and TimeGraph) are plausible. However, the headline empirical finding that majority support fails on CausalRivers is load-bearing for the paper's broader message about reference-dependent consensus. The reader's weakest assumption correctly identifies the incomplete topology reference as a validity threat, but I find an even more immediate problem: the Tier-2 and Tier-3 precision estimates on CausalRivers are based on 6 and 7 links, respectively, and no uncertainty is reported. The difference between 0.425 and 0.667 could easily arise by chance, so the empirical basis for 'not against river topology' is fragile even before considering reference incompleteness. A concrete check—bootstrap confidence intervals for the tier-precision difference—would settle whether the inversion is real. If the intervals overlap, the paper should soften the abstract and conclusion. The verdict remains CONDITIONAL because the software contribution is valuable and the concerns are addressable with additional analysis or more cautious wording; no fundamental error warrants rejection.","tokens_in":34843,"tokens_out":9615,"duration_ms":117170,"concrete_test":"Using the released CausalRivers per-subgraph tier files (experiments/causalrivers_validation/), compute Tier-1, Tier-2, and Tier-3 precision for each of the 30 subgraphs. Construct 95% confidence intervals for the Tier-1 minus Tier-2 precision difference via nonparametric bootstrap over subgraphs (2000 resamples) and also compute exact Wilson intervals per tier. Test the null that Tier-1 precision ≥ Tier-2 precision using a paired bootstrap test. If the confidence interval includes zero or the p-value exceeds 0.05, the inversion is not statistically supported, and the abstract's 'not against river topology' should be rephrased as 'the expected ordering was not observed in this small-sample real-data benchmark.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.4 reports CausalRivers Tier-1 precision 0.425 (n=287), Tier-2 0.667 (n=6), Tier-3 0.571 (n=7). The claimed inversion—the paper's key counterpoint to the synthetic benchmarks—depends entirely on the Tier-2 and Tier-3 estimates. With only 6 and 7 links, a single true/false reclassification changes precision by 0.14–0.17. No confidence intervals, bootstrap, or hypothesis test is provided. For example, a Wilson 95% CI for 4/6 true positives spans roughly 0.30–0.90, overlapping substantially with the Tier-1 CI. The abstract's statement that majority-supported links are 'not [more precise] against river topology' overstates what the data establish. The paper honestly acknowledges (Section 2.4) that the topology reference omits shared rainfall, snowmelt, reservoir operations, and tributary inflows, which could bias Tier-1 precision downward and even produce the inversion. But even setting that validity threat aside, the point estimates alone do not support the strong comparative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AutoCause, an open-source Python workflow for time-series causal discovery that wraps VAR-Granger, VARLiNGAM, transfer entropy, and PCMCI+, adds non-causal correlation and random-forest references, records six expert configuration decisions, and assigns consensus-support tiers by method count. It evaluates on 145 datasets from DGP-Atlas, TimeGraph, and a topology-derived CausalRivers reference, reporting method F1, complementarity, and tier-level precision. The headline empirical claim is that majority-supported (Tier-1) links are more precise than single-method links on the synthetic benchmarks (DGP-Atlas 0.848 vs 0.126; TimeGraph 0.694 vs 0.300), but that the ordering reverses on CausalRivers (Tier-1 0.425 vs Tier-2 0.667 and Tier-3 0.571).","tokens_in":35145,"tokens_out":6131,"duration_ms":65118,"significance":"If the results hold, the contribution is a valuable reproducibility and auditing layer around established causal-discovery methods, not a new discovery algorithm. The strengths of the manuscript are substantial: a released, versioned Python package with pinned dependencies, a file manifest tracing every reported number, machine-readable configuration logs, bootstrap intervals for method-level F1, inclusion of non-causal reference models, and unusually candid discussion of the limits of consensus tiering and of the CausalRivers topology reference. The paper also avoids overclaiming algorithmic novelty. However, the central consensus-precision claims need stronger statistical support and a clearer separation between development and independent evaluation before they can serve as the paper's main empirical message.","major_comments":[{"comment":"The CausalRivers tier-precision inversion rests on Tier-2 and Tier-3 estimates computed from n=6 and n=7 links, with no confidence intervals or bootstrap. One true/false reclassification changes Tier-2 precision by 0.167 and Tier-3 by 0.143; Wilson 95% intervals for the observed 4/6 and 4/7 counts overlap substantially with the Tier-1 interval (0.425, n=287). The paper acknowledges in §2.4 that the topology reference omits rainfall, snowmelt, reservoir operations, and tributary inflows, so links outside the topology may be genuine couplings scored as false. The abstract's 'but not against river topology' therefore overstates the support. I request uncertainty quantification on all tier-precision estimates and a softened or properly conditioned claim (e.g., 'inconclusive on this reference').","section":"§4.4, Fig. 9"},{"comment":"DGP-Atlas is the development ground for the pre-discovery extensions and decision thresholds, yet its 97 datasets are used as primary evidence for the 'synthetic benchmarks' half of the headline claim. The paper states that the three extensions 'emerged during DGP-Atlas validation' (§3.4), that the thresholds are 'implementation defaults' informed by DGP-Atlas (§4.1), and that family-level agreement in Table A.2 is an 'internal consistency check rather than independent external validation.' Under these conditions, DGP-Atlas tier-precision numbers cannot serve as independent confirmation of the workflow. The only independent synthetic evidence is TimeGraph, with n=49, 31, 20 for the three tiers and no uncertainty intervals. I recommend separating development and evaluation sets, or explicitly recasting DGP-Atlas as an illustrative stress test and basing the synthetic claim on independentl","section":"§3.4, §4.1, §4.4"},{"comment":"Even for the supported orderings, precision differences between tiers are reported as point estimates only. For TimeGraph, Tier-1 precision 0.694 (n=49) vs Tier-3 0.300 (n=20) is suggestive but not quantified; for DGP-Atlas, Tier-1 0.848 (n=1291) vs Tier-2 0.344 (n=209) is a large difference, but bootstrap intervals across datasets (as already used for method F1 in Fig. 3) would make the claim rigorous. I request dataset-level bootstrap or Wilson intervals for the consensus-support precision table, and the same for the vote-threshold comparison.","section":"§4.4, Table 11"}],"minor_comments":[{"comment":"Stage 5 in Figure 1 is labeled 'regime-weighted ensemble scoring,' but the text and Table 6 describe an unweighted method-count tier rule. Please align the figure with the implemented rule or explain the weighting if it exists.","section":"Fig. 1 vs §3.1/Table 6"},{"comment":"The sentence 'PCMCI+ has mean F1=0.814 on the same skeleton endpoint' appears twice in consecutive sentences. One repetition should be removed.","section":"§A.2"},{"comment":"The dashed 'chance' line is said to represent a 'common reference prevalence,' but the definition is not given. Please state explicitly how the reference prevalence is computed and whether it is the same quantity across the three benchmarks.","section":"Fig. 9"},{"comment":"The operational sample-size thresholds (50/200/75/120) are described as 'informed by published guidance and observed estimator behavior'; citing the specific sources in the text or appendix would help readers judge the safety margins.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is unusually transparent about its limitations, and the software contribution is likely valuable. My main concern is that the abstract and conclusions lean on two fragile supports: the DGP-Atlas evaluation is partly circular because the pre-discovery module was developed on that benchmark, and the CausalRivers inversion is based on six or seven links with no uncertainty. Both can be addressed by reframing and adding proper uncertainty quantification, which is why I recommend major revision rather than rejection. If the authors can report independent held-out synthetic results and confidence intervals for the tier-precision comparisons, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a solid software contribution, not a methods paper. It wraps four established discovery methods with a decision-logging layer, pre-discovery diagnostics, adaptive CI-test selection, and consensus tiering, and it ships code, data, and per-claim file provenance. The authors are unusually explicit about what they don't do: no new discovery algorithm, no causal proof, and limitations listed throughout. The causal-audit extensions (VARLiNGAM routing, a real diagnostic-extraction bug fix, and the revised abstention rule) are concrete and clearly described. The TimeGraph comparison against Ferdous et al.'s published PCMCI+ values is a good diagnostic exercise.\n\nSecond thing: the main empirical claim in the abstract overreaches. The DGP-Atlas evaluation is circular in a way the paper acknowledges but doesn't resolve: the three pre-discovery changes were developed during DGP-Atlas validation, so the strong results there are internal consistency checks, not independent evidence. That's survivable for a workflow paper, but it means the headline \"majority-supported links are more precise\" is only supported on TimeGraph, where n=49 Tier-1 links.\n\nThe CausalRivers inversion is the real problem. Tier-2 and Tier-3 precision are based on six and seven links, with no interval. The Wilson 95% CI for 4/6 true positives is roughly 0.30–0.90, which overlaps the Tier-1 CI. The abstract's \"not against river topology\" is not supported by these point estimates. To the paper's credit, Section 2.4 admits the topology reference omits shared rainfall, snowmelt, reservoirs, and tributary inflows, so even a well-powered comparison would need careful interpretation. But as written, the headline finding is an artifact of tiny counts plus an incomplete reference.\n\nWhat's actually solid: the integration itself, the reproducible artifacts, the honest limitation statements, and the demonstration that a multi-method workflow can be audited. None of these are undermined by the CausalRivers issue. The paper just needs to soften the empirical claims, add uncertainty quantification for small-n tiers, and be clear that DGP-Atlas results are development-set evaluations.\n\nWho should read it: environmental scientists who want a template for reproducible causal-discovery pipelines, and anyone building benchmark evaluations for workflow tools. It doesn't belong in a methods-focused venue without a new algorithm, but as a software/evaluation paper it deserves serious refereeing. My recommendation: send it out, with a request to reframe the CausalRivers conclusion and to treat DGP-Atlas as a development-set check rather than validation.","headline":"Useful workflow integration, honestly documented; the headline CausalRivers claim rests on six and seven links and should be reframed.","tokens_in":35609,"tokens_out":2531,"would_cite":false,"duration_ms":29803,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoCause claims that recording expert decisions yields reproducible causal graphs, and that majority-supported links are more precise on synthetic benchmarks but not on real river topology.","keywords":["causal discovery","time series","environmental modeling","multi-method consensus","assumption diagnostics","open-source framework","reproducibility","reference graph"],"falsifier":"Add a measured common driver, such as catchment-wide precipitation, as an observable variable and rerun AutoCause on the same river subgraphs; if Tier-1 precision against a hydrologically expanded reference rises above the Tier-2 and Tier-3 estimates, the inversion is an artifact of the incomplete reference. Alternatively, on synthetic data with a hidden common cause, compare Tier-1 precision with and without the hidden variable; a similar drop would explain the river result as latent confounding.","tokens_in":1357,"feed_emoji":"🌊","tokens_out":3933,"duration_ms":76202,"temperature":0.7,"pith_summary":"AutoCause is an open-source workflow that records the six decisions environmental analysts make when applying causal-discovery methods to time series, and grades every discovered link by how many of four methods agree on it. The paper's central empirical claim is that majority-supported links are more precise than single-method links on two synthetic benchmarks, but that this ordering reverses on a real river network where the reference graph is the physical topology. That reversal shows that consensus support is a useful precision filter only when the reference captures all genuine dependencies. The framework is not a new causal algorithm; it is an auditable decision-and-evaluation layer around established methods, with pre-discovery diagnostics, adaptive test selection, non-causal baselines, and reproducibility logs.","feed_headline":"Consensus causal links beat single methods—except on real rivers","feed_subtitle":"New open-source workflow logs every expert choice; majority-support precision falls from 0.85 to 0.42 on river topology.","key_machinery":"The central mechanism is method-count consensus tiering combined with a pre-discovery causal-audit module. The audit converts six assumption-violation risks into method, preprocessing, and conditional-independence-test recommendations; the consensus step then counts how many of the four methods detect each link and assigns Tier-1 (≥3 methods), Tier-2 (exactly 2), or Tier-3 (exactly 1) support. The reference graph supplied for evaluation is the object that makes precision measurable and determines whether consensus support appears valuable.","core_discovery":"AutoCause formalizes six expert decisions—method choice, conditional-independence test, lag horizon, sample-size adequacy, multiple-testing control, and evidence interpretation—into a single Python workflow. On 145 datasets, links detected by at least three of four causal methods (Tier-1) have higher precision than single-method links on DGP-Atlas (0.848 vs 0.126) and TimeGraph (0.694 vs 0.300), but on CausalRivers Tier-1 precision is 0.425 while Tier-2 and Tier-3 are 0.667 and 0.571. The paper attributes this inversion to the reference graph: the river topology omits rainfall, snowmelt, reservoir operations, and tributary inflows that can create real statistical dependence. AutoCause theref","pith_inferences":["The CausalRivers inversion implies that consensus algorithms can share blind spots: if all methods detect a common forcing or an indirect path, agreement may reflect a shared error rather than a direct causal link. A synthetic test that adds a hidden common driver and measures Tier-1 precision collapse would directly probe this mechanism.","The paper leaves open whether cross-family agreement carries information beyond vote count; an equal-vote comparison of same-family versus cross-family pairs would settle this, and the paper itself flags it as untested.","The competitive performance of the lagged-correlation baseline on river data hints that for some environmental questions, association screening combined with strong domain constraints may be nearly as informative as causal discovery, especially when the reference graph is incomplete.","The auditable-decision protocol could transfer to other application domains by adding new method families (e.g., continuous optimization, neural causal discovery) and new reference types, without changing the recorded-decision structure."],"forward_implications":["A single logged workflow makes environmental causal-discovery analyses reproducible and comparable across datasets, since every configuration choice is recorded and can be overridden by domain knowledge.","Majority-support tiering is a reliable precision filter only when the reference graph is complete; on real-world data with unobserved drivers, Tier-1 links should be read as repeated detection for further inspection, not as validated causal edges.","Audit-triggered preprocessing can rescue methods: on irregular-sampling DGP-Atlas data, PCMCI+ rises from F1 0.00 to 0.78 after interpolation, and deseasonalization recovers all skeleton edges on a trend-seasonal TimeGraph case.","No single method dominates across all regimes; a per-dataset oracle gains +0.104, +0.153, and +0.048 F1 over PCMCI+ alone on the three benchmarks, supporting multi-method ensembles as a robustness check.","The support tiers are descriptive labels, not posterior probabilities or causal validity scores; they prioritize which links deserve further physical interpretation."],"fun_headline_variants":["Consensus causal links underperform on river data","Majority-supported causality: precise on synthetic, not rivers","AutoCause: river topology reverses consensus causal edge","Causal consensus loses to single methods on real rivers"],"cache_read_input_tokens":36992,"weakest_assumption_plain":"The CausalRivers evaluation counts any link not in the physical river topology as a false positive, even though the topology omits rainfall, snowmelt, reservoir operations, and tributary inflows that can create genuine statistical dependence between stations.","fun_headline_variants_meta":{"raw":{"variants":["Consensus causal links underperform on river data","Majority-supported causality: precise on synthetic, not rivers","AutoCause: river topology reverses consensus causal edge","Causal consensus loses to single methods on real rivers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000622,"raw_usage":{"total_tokens":2718,"prompt_tokens":744,"completion_tokens":1974,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":1921}},"tokens_in":488,"tokens_out":1974,"duration_ms":18495,"temperature":1.0,"reasoning_tokens":1921,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:01:04.884258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Add a measured common driver, such as catchment-wide precipitation, as an observable variable and rerun AutoCause on the same river subgraphs; if Tier-1 precision against a hydrologically expanded reference rises above the Tier-2 and Tier-3 estimates, the inversion is an artifact of the incomplete reference. Alternatively, on synthetic data with a hidden common cause, compare Tier-1 precision with and without the hidden variable; a similar drop would explain the river result as latent confounding.","supporting_citations":[],"review_version":1}