{"id":"8eabe1ca-bfa1-46ad-90fc-3902d29e61df","arxiv_id":"2506.22848","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An automatically learned portfolio of four fGES configurations, selected by BIC, consistently outperforms single-algorithm baselines in large-scale Bayesian network structure learning.","lead":"This paper tests whether an ensemble of Bayesian network structure learning algorithms, automatically selected from a candidate pool, can improve divide-and-conquer methods on large networks. On synthetic problems with up to 10,000 variables, the ensemble improved accuracy by roughly 30% to 225% over single-algorithm baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-set network families overlap the test networks (probability >95% per family), so the reported 30-225% gains at 10,000 variables may reflect in-distribution tuning; out-of-distribution WS/Yeast gains are far smaller, and a disjoint-family retraining is needed.","rationale":"The reader's weakest assumption was the oracle-vs-BIC selection mismatch. That concern matters for Theorem 1, but the empirical central claim can survive imperfect BIC selection if the portfolio still beats single-algorithm baselines. The network-family overlap is more load-bearing because it targets the empirical numbers themselves: if the SLE is trained on the same bnlearn topologies used for testing, the 30-225% gains at 10,000 variables are in-distribution results. The paper's own out-of-distribution experiments (Table VII) show much smaller or even negative improvements, casting doubt on 'consistently significantly higher accuracy in learning large BNs.' A leave-one-family-out retraining directly tests this. This is a methodological design issue, not an internal inconsistency, and it is fixable, so the CONDITIONAL verdict remains appropriate. I partially agree with the reader: we both flag evaluation weaknesses, but I elevate the overlap to the primary load-bearing concern, with the BIC/oracle mismatch as a secondary issue about the theoretical guarantee.","tokens_in":23626,"tokens_out":9391,"duration_ms":98674,"concrete_test":"Retrain Auto-SLE from scratch on a training set that excludes the ten test families (e.g., generate 100 problems from only the other 22 bnlearn networks, or use leave-one-family-out on each test network), keep all other settings identical (same Θ, SMAC budget, k=4, stop condition), and re-run the 10,000-variable experiments in Table VI. If the median F1 adjacent or F1 arrowhead improvement over the best baseline drops below 30% or becomes non-significant, the reported gains are an artifact of training/test network-family overlap and the generality claim in the abstract is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The training set (Section IV.A.3) is built by randomly drawing a base network from the 32 bnlearn networks, 100 times. The test suite (Section IV.A.1; Tables V-VI) uses 10 specific bnlearn networks. Each test network therefore appears in the training set with probability 1 - (31/32)^100 ≈ 96%, so the 'independent' training problems share network topology with the test problems even though the simulated datasets differ. The headline 30-225% improvements at 10,000 variables are all measured on these same 10 families. The only genuinely out-of-training tests (Table VII) show much weaker results: WS-1000 F1 arrowhead is -0.6%, WS-10000 F1 arrowhead improves 21.1%, and Yeast has absolute F1 around 0.1. This makes the central claim 'consistently achieves significantly higher accuracy in learning large BNs' unsupported for network families outside the training distribution. Auto-SLE may simply be tuning fGES parameters to the benchmark structures, and the ensemble's cross-family generality remains unestablished.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces structure learning ensembles (SLEs) for divide-and-conquer Bayesian network structure learning. It formalizes SLE learning as maximizing average F1 over a training problem set with an oracle that selects the best member output, proves that this objective is monotone submodular, and provides a (1-1/e)-type approximation guarantee for the proposed greedy algorithm Auto-SLE. The learned SLE (which, in the experiments, consists of four fGES parameter configurations) is embedded into the PEF divide-and-conquer pipeline, yielding P/SLE. Experiments on bnlearn-based synthetic benchmarks with 1,000 and 10,000 variables report consistent accuracy improvements over single-algorithm baselines, with gains of 30%-225% at 10,000 variables; additional experiments on 20,000-30,000 variable versions of two networks, a small-world network, and a yeast gene network are used to support generalization claims.","tokens_in":23801,"tokens_out":5527,"duration_ms":55622,"significance":"If the central claims are established, the paper would make a useful practical contribution: automatically configured algorithm portfolios for large-scale BN structure learning, with an elegant and correct theoretical analysis of the training objective (Fact 1 and Theorem 1 follow from standard submodularity), open-sourced code, and extensive experiments comparing against strong baselines. The idea of using an ensemble in the estimation step of divide-and-conquer methods is sensible and underexplored. However, the significance is substantially tempered by two issues: the training set appears to be drawn from the same network families as the main test suite, which makes the headline improvements partly an in-distribution tuning effect, and the theoretical guarantee applies to an oracle-selection objective that is not what is used at deployment time (BIC-based selection). The out-of-distribution results (WS and Yeast, Table VII) are much weaker and sometimes negative, so the paper's generalization claims are not yet supported. The core method is defensible, but the current evidence base and framing need rework.","major_comments":[{"comment":"The training set is generated by randomly drawing a base network from the same 32 bnlearn networks used to construct the 10 test networks, 100 times. Each of the 10 test networks therefore appears as a base topology in the training set with probability 1 - (31/32)^100 ≈ 96%. The simulated datasets differ, but the underlying DAG topology is shared, so the SLE's parameters (Table II) are effectively tuned to the exact structures on which the headline 30%-225% improvements (Table VI) are measured. The paper's statement that these training problems are 'independent of the testing problems' is therefore misleading. The only genuinely out-of-distribution tests (Table VII, WS and Yeast) show much smaller gains: for WS-1000 the F1 arrowhead improvement is -0.6%, and for Yeast the absolute F1 values are around 0.08-0.1. This does not support the abstract and introduction's claim of 'consistently achieving significantly higher accuracy' or generalization to 'different network characteristics'. A disjoint-family training/test split is needed to establish that the learned SLE provides more than benchmark-specific tuning.","section":"Section IV.A.3 and Tables V-VI"},{"comment":"The training objective Q in Eq. (3) uses an oracle that selects the member algorithm with the best F1 score (ground truth known), while at test time Section III.D states that the best output is selected using BIC because ground truth is unavailable. Theorem 1's guarantee applies only to the oracle-based Q; it does not cover the BIC-selected output of the deployed system. The paper offers no analysis or empirical evidence that BIC-based selection tracks oracle F1 selection on the training problems or on the test problems. Since P/SLE's advantage over a single fGES configuration depends on BIC picking the right member, the mismatch between the optimized objective and the deployed selection rule is a load-bearing gap. This should be addressed, for example by training with a surrogate score that mimics test-time selection, or by reporting the correlation between BIC and F1 on training and test subproblems.","section":"Section III.D and Theorem 1"},{"comment":"The results on all 10,000-variable problems are based on a single generated problem per network, with no standard deviation or significance test. The improvement ratios of 30%-225% are computed against the best of P/PC-Stable and P/fGES, and the paper acknowledges that P/SLE(D) and P/fGES are identical because BIC always selects the fGES member. Thus the effective comparison set is two points per network, and the reported gains could be unstable across seeds. Given that the central quantitative claim rests on these numbers, the paper should provide multiple independent runs per network (or at least report the spread), particularly because the lower-variance 1,000-variable results in Table V show much smaller and sometimes negative improvements (e.g., Healthcare F1).","section":"Table VI"}],"minor_comments":[{"comment":"There are several typos: 'conder' should be 'considered', 'he other' should be 'the other', and in Section IV.D 'structure leanring' should be 'structure learning'.","section":"Section IV.A.2"},{"comment":"The caption says 'P/SLE-B REPRESENTS THE BEST PERFORMANCE ACHIEVED AMONG P/SLE(R), P/SLE(D), P/FGES, AND P/PC-S TABLE', where 'P/PC-S TABLE' appears to be a typo for 'P/PC-Stable'.","section":"Table VII"},{"comment":"The paper says the SLE is learned from 'a diverse training problem set', but the learned SLE (Table II) contains only fGES with different parameter values; PC-Stable was never selected. This is a legitimate outcome, but the framing of an 'algorithm ensemble' combining multiple structure learning algorithms is somewhat stronger than what the experiments actually produce, and this should be stated more precisely.","section":"Section IV.A.3"},{"comment":"The proof has minor notational issues: in Eq. (8) the line 'for all positive integers i < l≤ k' should be 'i < l ≤ k', and the telescoping sum notation is slightly compressed. The argument itself is sound.","section":"Theorem 1 proof"}],"recommendation":"major_revision","confidential_remarks":"The training/test topology overlap is the most serious issue and may not be apparent to all readers because the paper explicitly calls the training problems 'independent' of the testing problems; the independence claim technically refers to the simulated datasets, not the underlying networks. I would encourage the editor to require a disjoint-family evaluation or a clear disclosure of the overlap before publication. The theoretical result is correct but has a narrower scope than the deployed system, which is a second core point that needs strengthening. If the authors can provide a disjoint-family training/test split and show that the gains persist (or are at least honestly reported with the overlap disclosed), the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful empirical contribution to large-scale BN structure learning, but the headline gains are mostly in-distribution. The training set draws base networks from the same 32 bnlearn repository that supplies all ten test networks, so each test family appears in training with probability ~96%. The 30-225% improvements on 10,000-variable problems all come from those families. On the out-of-distribution WS and Yeast problems, the gains shrink to single digits or go negative (WS-1000 F1 arrowhead: -0.6%), and Yeast absolute F1 is around 0.1. So \"consistent\" is not supported across network characteristics.\n\nWhat's new and good: embedding an automatically constructed portfolio into the estimation step of PEF is a sensible idea, and this is the first time it's done for D&C BN learning. The greedy submodular formulation is standard (AutoAttack, their own prior work), but the application is new and the experiments are extensive, including up to 30,000 variables. The code and data are open-sourced, and the theoretical bound is correct for the training objective. The paper also honestly flags that test-time selection uses BIC rather than oracle F1.\n\nSoft spots: (1) the train/test family overlap, which the text calls \"independent\" because the simulated datasets differ—the topologies are the same, so this is tuning, not generalization; (2) the BIC-vs-oracle mismatch means Theorem 1 does not cover deployment; a practical gap, not fatal, but it should be analyzed or at least discussed; (3) the learned SLE is four fGES configurations, so the ensemble may just be hyperparameter adaptation. Without a per-subproblem tuned fGES baseline, we can't tell whether complementarity buys anything beyond picking a good lambda per subproblem. A cheap experiment—tune fGES on each subproblem, or use a simple per-subproblem model selection—would settle this; (4) the generalization curves in Figure 2 are only for Alarm and Asia, again two families seen in training.\n\nBottom line: for readers working on scalable causal discovery, the paper is worth reading and the approach worth building on. It deserves a serious referee, but the central claim needs sharper evidence: retrain on a disjoint family set and compare against per-subproblem tuned fGES. I'd accept it into the review process with the expectation of major revision.","headline":"A useful portfolio-for-PEF result undermined by train/test family overlap and a theory-practice gap in selection; the 30-225% headline is in-distribution.","tokens_in":24395,"tokens_out":3440,"would_cite":true,"duration_ms":37256,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learning an ensemble of Bayesian-network structure learners and embedding it in a divide-and-conquer pipeline lifts accuracy on 10,000-variable networks by 30–225% over the best single-algorithm baseline.","keywords":["Bayesian network structure learning","divide-and-conquer","algorithm ensembles","submodular optimization","greedy selection","large-scale Gaussian networks","BIC score selection","partition-estimation-fusion"],"falsifier":"Run P/SLE on 10,000-variable Gaussian problems constructed so that BIC systematically favors a low-F1 member, for example by varying the fGES penalty in ways that overfit BIC; if accuracy drops to the level of P/fGES on a set of such problems, the claimed ensemble gain is an artifact of BIC selection rather than of ensemble complementarity.","tokens_in":23360,"feed_emoji":"📈","tokens_out":8310,"duration_ms":83986,"temperature":0.7,"pith_summary":"At the core of this paper is a single claim: the main obstacle to learning very large Bayesian networks by divide-and-conquer is not the partition or fusion step but the instability of a single structure-learning algorithm across heterogeneous subproblems, and an automatically learned ensemble of algorithms removes that instability. The authors define a structure learning ensemble, a set of parameterized learner configurations, whose quality on any problem is the best quality among its members, and show that this quality function is monotone and submodular. They then propose Auto-SLE, a greedy constructor that repeatedly adds the configuration giving the largest marginal improvement and inherits a (1 - 1/e)-approximation guarantee relative to the optimal ensemble on the training problems. Embedded in the partition-estimation-fusion method, the learned ensemble (P/SLE) improves F1 and SHD accuracy on 10,000-variable Gaussian networks by roughly 30% to 225% over the best single-algorithm divide-and-conquer baseline and generalizes to 30,000-variable and non-block-structured networks with no retuning. If the paper is right, structure learning for large Bayesian networks ceases to be a contest between individual algorithms and becomes an automated portfolio-selection problem.","feed_headline":"Algorithm ensembles lift huge Bayesian-net accuracy up to 225%","feed_subtitle":"Auto-SLE assembles complementary learners per subproblem, so accuracy holds where single algorithms collapse.","key_machinery":"The engine of the argument is the SLE learning problem and the greedy Auto-SLE algorithm. The quality measure is $Q(\\mathcal{A},\\mathcal{D}) = \\max_{\\theta \\in \\mathcal{A}} Q(\\theta,\\mathcal{D})$ averaged over training problems as $Q(\\mathcal{A},\\mathcal{T}) = \\frac{1}{|\\mathcal{T}|}\\sum_{\\mathcal{D}\\in\\mathcal{T}} Q(\\mathcal{A},\\mathcal{D})$, which is monotone and submodular by Fact 1 because the maximum operator has diminishing returns. Auto-SLE starts with an empty ensemble and, for $k$ iterations, adds the candidate configuration $\\theta^\\circ$ maximizing the marginal gain $\\Delta(\\theta|\\mathcal{A}) = Q(\\mathcal{A}\\cup\\{\\theta\\},\\mathcal{T}) - Q(\\mathcal{A},\\mathcal{T})$, terminating early if no gain is positive; Theorem 1 gives the $(1-1/e)$ approximation with an explicit $k\\epsilon$ term under imperfect parameter optimization. This learned ensemble is then dropped into the estimation step of PEF (partition, estimation, fusion), with the member whose output has the best BIC score selected for each subproblem.","core_discovery":"The paper's central discovery is that a structure learning ensemble, learned once from diverse small training problems, can be transplanted into the estimation step of a divide-and-conquer method and keeps its accuracy advantage as the network scales to 10,000 and 30,000 variables. The learned ensemble contains only variants of a single score-based algorithm (fGES) with different penalty and maximum-parents settings; the constraint-based algorithm also in the design space never wins a slot. At training time the ensemble is scored by the best member's F1 on each problem, and at deployment the member output with the best BIC score is returned. The empirical core is a set of comparisons on ten benchmark-derived networks: P/SLE's F1-adjacent accuracy at 10,000 variables exceeds every single-algorithm divide-and-conquer baseline, typically by 30% or more and in one case by 133%, with corresponding structural-Hamming-distance reductions, and it is the only method that keeps accuracy roughly flat from 1,000 to 30,000 variables.","pith_inferences":["A concrete test of the paper's logic would be to use a learned selection model, as the authors mention in future work, to pick the member algorithm from cheap problem features instead of BIC; if that matches or beats BIC selection, the ensemble method would become applicable to discrete and non-Gaussian Bayesian networks where BIC is a less reliable accuracy proxy.","The submodular portfolio view could be applied to the estimation step of other divide-and-conquer pipelines in causal discovery, such as subsampled or distributed skeleton finding, where per-subproblem algorithm instability is also the bottleneck.","Because the training and deployment sample size is fixed at 1000, the learned ensemble may be miscalibrated when deployment data are much denser or sparser; re-learning on training problems with matched sample size is an obvious extension the paper does not test.","The reported gains are relative to baselines that use default parameters; a more demanding comparison would pit P/SLE against single-algorithm PEF whose hyperparameters are tuned per network class by the same Bayesian optimizer, isolating the contribution of ensembling from that of parameter tuning."],"forward_implications":["At 10,000 variables, P/SLE reduces wall-clock time from no solution in 24 hours for fGES and PC-Stable to tens of minutes on most tested networks, while keeping F1-adjacent accuracy above 0.8 on most.","Accuracy improvements grow with network size: on 10,000-variable problems the F1-arrowhead improvement over the best baseline reaches 225%, and the margin continues to widen through 20,000 and 30,000 variables.","The learned ensemble transfers to networks not represented in training: small-world networks and a 4,441-node gene regulatory network, where P/SLE still beats every single-algorithm PEF baseline.","Auto-SLE's guarantee is not limited to Gaussian Bayesian networks; the monotone-submodular formulation and the greedy approximation hold for any quality measure, so the same pipeline could learn ensembles for other causal models.","The success of an all-fGES ensemble implies that the practical diversity that matters is in hyperparameter configurations, such as penalty and maximum parents, rather than in algorithm family differences."],"supporting_citations":[{"why":"Supplies the partition-estimation-fusion framework that P/SLE embeds its ensemble into.","marker":"[13]"},{"why":"Supplies fGES, the main member algorithm and the strongest single-algorithm baseline.","marker":"[6]"},{"why":"Supplies PC-Stable, the constraint-based candidate added to the design space and a baseline.","marker":"[5]"},{"why":"Supplies the motivating example of an attack ensemble whose reliability transfers to structure learning.","marker":"[15]"},{"why":"Supplies the Bayesian optimization subroutine Auto-SLE uses to maximize each marginal gain.","marker":"[31]"},{"why":"Provides the submodular maximization approximation theory that Theorem 1 extends.","marker":"[32]"},{"why":"Supplies the technique that would recover train-to-test guarantees for the learned ensemble.","marker":"[36]"},{"why":"Supplies the small base networks from which all large training and testing problems are generated.","marker":"[8]"}],"fun_headline_variants":["Auto-SLE ensemble lifts Bayesian-net accuracy 30-225% on huge nets","Learned algorithm ensemble boosts BN learning at 10k and 30k variables","Ensemble of fGES variants beats single learners in BN structure discovery","Auto-SLE: automatic ensemble design for scalable Bayesian network learning","One ensemble, many scales: Bayesian nets get 30-225% accuracy boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the BIC score, used to select a member's output on new problems, picks the member with the highest true accuracy just as reliably as the oracle used when training the ensemble.","fun_headline_variants_meta":{"raw":{"variants":["Auto-SLE ensemble lifts Bayesian-net accuracy 30-225% on huge nets","Learned algorithm ensemble boosts BN learning at 10k and 30k variables","Ensemble of fGES variants beats single learners in BN structure discovery","Auto-SLE: automatic ensemble design for scalable Bayesian network learning","One ensemble, many scales: Bayesian nets get 30-225% accuracy boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000509,"raw_usage":{"total_tokens":2489,"prompt_tokens":969,"completion_tokens":1520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":1420}},"tokens_in":585,"tokens_out":1520,"duration_ms":13330,"temperature":1.0,"reasoning_tokens":1420,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:57:10.350192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run P/SLE on 10,000-variable Gaussian problems constructed so that BIC systematically favors a low-F1 member, for example by varying the fGES penalty in ways that overfit BIC; if accuracy drops to the level of P/fGES on a set of such problems, the claimed ensemble gain is an artifact of BIC selection rather than of ensemble complementarity.","supporting_citations":[{"cited_title":"Learning big gaussian bayesian networks: Partition, estimation and fusion,","cited_arxiv_id":null,"evidence_quote":"Supplies the partition-estimation-fusion framework that P/SLE embeds its ensemble into."},{"cited_title":"Ramsey, M","cited_arxiv_id":null,"evidence_quote":"Supplies fGES, the main member algorithm and the strongest single-algorithm baseline."},{"cited_title":"Order-independent constraint- based causal structure learning","cited_arxiv_id":null,"evidence_quote":"Supplies PC-Stable, the constraint-based candidate added to the design space and a baseline."},{"cited_title":"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,","cited_arxiv_id":null,"evidence_quote":"Supplies the motivating example of an attack ensemble whose reliability transfers to structure learning."},{"cited_title":"Smac3: A versatile bayesian optimization package for hyperparameter optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian optimization subroutine Auto-SLE uses to maximize each marginal gain."},{"cited_title":"An analysis of ap- proximations for maximizing submodular set functions - I,","cited_arxiv_id":null,"evidence_quote":"Provides the submodular maximization approximation theory that Theorem 1 extends."},{"cited_title":"On performance estimation in automatic algorithm configuration,","cited_arxiv_id":null,"evidence_quote":"Supplies the technique that would recover train-to-test guarantees for the learned ensemble."},{"cited_title":"Learning bayesian networks with the bnlearn r package,","cited_arxiv_id":null,"evidence_quote":"Supplies the small base networks from which all large training and testing problems are generated."}],"review_version":1}