{"id":"450eb5d7-a3a1-4229-bf0d-21a7436a8234","arxiv_id":"2502.00838","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new Bayesian optimization framework with hierarchical kernels and hierarchical sampling solves a jet engine architecture problem in about 300 evaluations, matching NSGA-II results that previously required 3250.","lead":"This paper develops Bayesian optimization strategies for system architecture design problems with hierarchical, mixed-discrete design spaces: new kernels, sampling, and correction algorithms. On a jet engine architecture benchmark, the resulting algorithm reaches the same optimum as NSGA-II using about one tenth of the function evaluations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 92% claim relies on a single NSGA-II baseline from prior work that is not re-run here, so the comparison is uncontrolled.","rationale":"The paper makes a real contribution: a hierarchical categorical kernel, a hierarchical sampling method, new hierarchy metrics, and an open-source implementation with extensive experiments. The empirical case that BO needs an order of magnitude fewer evaluations than NSGA-II is plausible and directionally supported by the within-paper 300-evaluation comparison. However, the headline number depends on comparing the BO runs to an NSGA-II baseline that was not re-run in this paper. The reader's weakest assumption identifies exactly this uncontrolled comparison, and I find that concern load-bearing. I did not find a more serious internal inconsistency: the kernel PSD gap is real but secondary, since BO without the new hierarchical kernel also reaches the target TSFC within 300 evaluations. The test problems are reasonable, and the open-source code is a meaningful independent support. The appropriate verdict remains CONDITIONAL, so no change to the reader's verdict is needed.","tokens_in":37245,"tokens_out":4152,"duration_ms":43040,"concrete_test":"Add a controlled replication to the released experiment repository: run NSGA-II using SBArchOpt's ArchOptNSGA2 on SimpleTurbofanArch with the repair operator and hierarchical xact sampling, exactly as described in Section 7, at budgets of 300, 600, 1200, 2000, and 3250 evaluations, with at least 10 independent seeds per budget. Record median and minimum feasible TSFC for each budget. If NSGA-II reaches TSFC <= 6.65 at a budget substantially below 3250, or if the 3250-evaluation result from [46] is not reproduced under these settings, the '92% less' claim must be replaced by a budget-dependent comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim in Section 7 and Table 19 is a cross-paper comparison: the BO result at 300 evaluations (TSFC 6.633 g/kNs) is measured in this paper, but the NSGA-II result at 3250 evaluations (TSFC 6.640 g/kNs) is taken from the authors' earlier work [46]. The paper does not re-run NSGA-II at 3250 on the same problem instance with the same correction, hierarchical sampling, constraints, and termination logic. Function-evaluation counts are only comparable if the evaluated search space and evaluation function are identical. If [46] used a different problem variant or NSGA-II reached TSFC <= 6.65 well before 3250 evaluations, the claimed '92% fewer' figure is unsupported. The paper's own NSGA-II run at 300 evaluations (TSFC 7.455) shows NSGA-II is slow on this benchmark, but it does not locate the crossover budget between 300 and 3250. The table also reports no variance for the 3250 baseline, so even the numerical comparison is not statistically grounded. This is the load-bearing weakness of the paper's headline claim; the other technical gaps, such as the missing PSD proof for the new kernel, do not affect the central BO-vs-NSGA-II conclusion because even the Repair and Hierarchical-sampling BO variants reach the target TSFC within 300 evaluations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes methods for system architecture optimization (SAO) with expensive, hierarchical, mixed-discrete, constrained, multi-objective problems. It introduces metrics (imputation/correction ratios, correction fraction, max rate diversity), a new Gaussian process kernel for hierarchical categorical variables, hierarchical sampling and correction algorithms, and a comparison of hierarchy integration strategies in NSGA-II and Bayesian optimization (BO). The central empirical claim is that the proposed BO algorithm reaches a TSFC of about 6.63-6.65 g/kNs in 300 function evaluations, which is 92% fewer than the 3250 evaluations reportedly needed by NSGA-II for a comparable or slightly worse result (Section 7, Table 19).","tokens_in":37523,"tokens_out":5902,"duration_ms":56605,"significance":"If substantiated, the paper provides a compelling demonstration that BO can reduce evaluation counts by an order of magnitude on realistic SAO problems, and it contributes reusable open-source components (SBArchOpt) and a dataset with experiment code. The new hierarchy metrics and the kernel for categorical hierarchical variables address a real gap in surrogate-based architecture optimization. The strengths include the reproducible experimental pipeline, the explicit statistical ranking procedure, and the careful discussion of when hierarchical GP models help or hurt.","major_comments":[{"comment":"The central 92% claim rests on an uncontrolled cross-paper comparison. The NSGA-II result at 3250 evaluations is taken from prior work [46] and is not re-run under the same conditions as the BO runs in this paper (problem formulation, correction mechanism, hierarchical sampling, constraint handling, and termination logic). Table 19 reports no variance for this baseline and explicitly states that '∆HV regret was not available for NSGA-II with 3250 evaluations'. To support the headline claim, the authors should re-run NSGA-II at both 300 and 3250 evaluations under the present settings, reporting distributions, or else clearly qualify the claim as a comparison to a previously published result rather than a controlled experimental finding.","section":"Section 7, Table 19"},{"comment":"The proposed hierarchical categorical kernel is not proven to be positive semi-definite. The text states that '[Φ(Θi)] is a chosen symmetric positive definite parameterization', but this does not by itself guarantee that the correlation matrices defined by Eqs. (12)-(14) are PSD for arbitrary collections of points. Since a valid GP kernel requires PSD, the paper should provide a proof or a precise citation for the PSD property of this hierarchical kernel, or present it as an empirical/heuristic construction with the associated caveat. This is load-bearing for the paper's kernel contribution.","section":"Section 4, Eqs. (12)-(14)"}],"minor_comments":[{"comment":"The sentence 'The developed BO algorithm is implemented as as ArchSBO in SBArchOpt' contains a duplicated 'as' and should be corrected.","section":"Section 8"},{"comment":"The caption reads 'running NGSA-II', which should be 'NSGA-II'.","section":"Table 13 caption"},{"comment":"The label 'Hier.' for the non-grouping hierarchical sampler is ambiguous: it could be confused with 'Hierarchical sampling' generally. The table should clarify that this row is the hierarchical sampler without grouping or weighting.","section":"Section 5.2, Table 10"},{"comment":"Equation (12) writes the exponent as `exp(-√2[Φ(Θi)]_ℓ_i^r,ℓ_i^r - √2[Φ(Θi)]_ℓ_i^s,ℓ_i^s)`, which is not a standard distance-based correlation form; please verify the intended expression and correct any typographical error.","section":"Section 4, Eq. (12)"},{"comment":"The ranking procedure uses a 10% significance level for the two-sample t-test without justification; please state why this threshold was chosen or cite a reference for this practice.","section":"Appendix A"},{"comment":"The abstract's statement that 'integrating more hierarchy information in the optimization algorithms yields better optimization results for BO algorithms' is too strong in view of Table 18, where the Activeness configuration (hierarchical GP) has a 20% penalty relative to Hierarchical sampling, and Table 19, where Activeness has higher ∆HV regret than Hierarchical sampling. The abstract should include the exception noted in Section 6.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful empirical contribution with open artifacts, but the headline 92% claim needs a matched re-run of the NSGA-II baseline, and the kernel PSD proof is required for the theoretical novelty. The remaining issues are presentation-level and should be addressed in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives you a genuinely useful toolkit: the imputation/correction/rate-diversity metrics are well-defined and will be handy for characterizing hierarchical design spaces, and the activeness-based hierarchical sampler is simple, effective, and reproducible. The integration-strategy comparison is thorough—lots of repetitions, statistical ranking, open code and data—and the categorical hierarchical kernel is a real extension of the authors' prior work, even if it lacks a positive-definiteness proof. Credit where due: this is careful engineering, not a toy study.\n\nThe soft spot is exactly where your reader put it. The headline '92% fewer evaluations than NSGA-II' compares a fresh BO run at 300 evaluations against an NSGA-II result at 3250 taken from earlier work, and that earlier baseline is not re-run under matched conditions. The paper's own NSGA-II run at 300 evaluations shows it is slow on this benchmark, but that does not locate the crossover budget. Table 19 gives no variance for the 3250 baseline, so the numerical comparison is not statistically grounded. This is a load-bearing flaw in the claim as stated, but it is fixable: rerun NSGA-II at 3250 with the same problem formulation, correction, and termination logic, or at least make the comparability argument explicit.\n\nA second, minor concern: the test-problem results (mostly synthetic or random-forest substitutes) do not fully predict the real jet-engine outcome, and the authors acknowledge that the hierarchical GP actually hurt BO performance on the test problems while helping slightly on the real application. That inconsistency is reported honestly, which is a point in the paper's favor, but it means the integration-strategy conclusions are less crisp than the abstract suggests.\n\nThe missing PSD proof for the kernel is a technical gap but does not affect the central BO-versus-NSGA-II direction—even the Repair variant reaches the target TSFC within 300 evaluations. So the main claim is plausible, just not proven to the stated precision.\n\nWho is this for? Researchers working on surrogate-based optimization for expensive, hierarchical, mixed-discrete problems, especially in aerospace. It deserves a serious referee; the empirical work is extensive and the code is a real asset. I would send it to review with a request to tighten the NSGA-II comparison.","headline":"Solid toolkit for hierarchical architecture optimization with a plausible but uncontrolled 10x speedup claim that needs a matched NSGA-II rerun or explicit comparability argument.","tokens_in":38037,"tokens_out":2094,"would_cite":true,"duration_ms":23715,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C26","90C29"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that hierarchy-aware Bayesian optimization finds the same jet engine architecture optimum as NSGA-II using 92% fewer function evaluations.","keywords":["system architecture optimization","Bayesian optimization","hierarchical design spaces","Gaussian process kernels","hierarchical sampling","NSGA-II","jet engine architecture","hidden constraints"],"falsifier":"Re-run the simple jet engine problem in the same SBArchOpt framework: run NSGA-II for 3250 evaluations and the proposed BO algorithm for 300 evaluations under identical initial design-of-experiments and termination settings. If NSGA-II's final TSFC is more than 0.2% below BO's, or if BO cannot reach the reference TSFC value within 300 evaluations, the 92% claim is refuted.","tokens_in":37040,"feed_emoji":"✈️","tokens_out":3780,"duration_ms":32717,"temperature":0.7,"pith_summary":"This paper claims that Bayesian Optimization, when given information about the hierarchical structure of a design space, can solve realistic system architecture optimization problems with one order of magnitude fewer function evaluations than evolutionary algorithms. The demonstration case is a jet engine architecting problem where the BO algorithm reaches essentially the same optimal thrust-specific fuel consumption in 300 evaluations that NSGA-II reaches in 3250, a 92% reduction. For this to work, the paper develops three things: metrics that quantify how much of a design space is missing or redundant because of hierarchy, a Gaussian process kernel that handles hierarchical categorical variables, and a sampling scheme that corrects for uneven representation of architecture groups. It also shows experimentally that for BO, the more hierarchy information integrated, the better the optimization results.","feed_headline":"BO finds jet engine optimum with 92% fewer evaluations","feed_subtitle":"Hierarchy-aware Bayesian optimization reaches the reference design in 300 evaluations instead of 3250.","key_machinery":"The argument rests on four devices. First, four new metrics: imputation ratio (declared vs valid design space size), correction ratio (declared vs correct size), correction fraction (share of hierarchy due to value constraints), and max rate diversity (imbalance in how often discrete values occur). Second, a hierarchical Gaussian process kernel for categorical conditionally-active variables, built by one-hot encoding with a modified Gower distance that treats inactive levels as distance zero between inactive pairs and as a penalized distance between active and inactive pairs. Third, a hierarchical sampling algorithm that groups all valid discrete design vectors by their active variable sets and weights groups to mitigate rate diversity, then fills continuous variables with Sobol' sampling. Fourth, a set of eager and lazy correction algorithms that repair invalid design vectors.","core_discovery":"The central claim is that hierarchy-aware Bayesian Optimization is a practical solver for expensive, black-box, mixed-discrete, constrained, multi-objective architecture optimization problems. On the jet engine benchmark, the BO algorithm with activeness information reaches a median optimal TSFC of 6.633 g/kNs within 300 evaluations, compared with 6.640 g/kNs for NSGA-II at 3250 evaluations, which the authors describe as 92% fewer function evaluations for the same result. The paper further establishes that integrating hierarchy information through hierarchical sampling and, to a lesser extent, hierarchical Gaussian process kernels improves BO performance, while the level of hierarchy integration has little influence on NSGA-II.","pith_inferences":["If the 92% savings transfer to other expensive MDAO problems, the practical bottleneck shifts from evaluation count to GP training and infill search time, which become the dominant costs.","The hierarchical kernel's active-to-inactive distance rule suggests a natural way to extend BO to graph-based architectures: define similarity between architecture instances directly instead of through one-hot relaxed variables.","The paper's own comparison suggests a sharper test: run NSGA-II and BO on the same problem with identical initial populations and termination criteria; the 10x claim would then be directly quantified rather than inherited from a prior run.","On problems with rougher, more discontinuous objectives than the rocket and GNC tests, hierarchical GP kernels should show larger gains; that is a testable prediction of the paper's reasoning."],"forward_implications":["Bayesian Optimization can replace evolutionary algorithms on expensive architecture problems, cutting evaluation budgets by roughly an order of magnitude.","Problem-agnostic eager correction performs close to problem-specific correction, so hierarchy-aware optimizers need not hand-code repair logic.","For BO, hierarchical sampling consistently beats non-hierarchical sampling, while hierarchical GP kernels did not help on the tested smooth problems; the paper keeps both for harder cases.","The metrics IR, CR, CRF, and MRD give a standard way to characterize how much hierarchy matters before choosing an optimization strategy.","All algorithms and test problems are released open-source in SBArchOpt, enabling direct replication."],"supporting_citations":[{"why":"Supplies the jet engine benchmark problem and the NSGA-II 3250-evaluation optimum that the 92% reduction is measured against.","marker":"[46]"},{"why":"Provides the hierarchical Gaussian process and kernel implementation in SMT that this work extends to categorical variables.","marker":"[83]"},{"why":"Underpins the Gower-distance categorical kernel that the new hierarchical kernel generalizes.","marker":"[28]"},{"why":"Defines NSGA-II, the evolutionary algorithm used as the comparison baseline.","marker":"[60]"},{"why":"Provides the probability-of-viability handling used for the jet engine's approximately 50% hidden-constraint failure rate.","marker":"[23]"},{"why":"Introduces the infill ensemble and MEPoI strategy that the BO algorithm builds on.","marker":"[6]"},{"why":"Supplies the KPLS dimension-reduction method for high-dimensional GP models used by the BO algorithm.","marker":"[94]"}],"fun_headline_variants":["Hierarchy-aware BO solves jet engine design in 300 evals","BO beats NSGA-II on jet architecture with 92% fewer calls","Bayesian optimization cuts jet evaluations by 92%","300 evals to optimum: hierarchy-aware BO impresses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the NSGA-II result at 3250 evaluations from the authors' earlier work is directly comparable to the BO runs here, with the same problem formulation, evaluation function, termination logic, and cost model.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchy-aware BO solves jet engine design in 300 evals","BO beats NSGA-II on jet architecture with 92% fewer calls","Bayesian optimization cuts jet evaluations by 92%","300 evals to optimum: hierarchy-aware BO impresses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000467,"raw_usage":{"total_tokens":2326,"prompt_tokens":939,"completion_tokens":1387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1316}},"tokens_in":555,"tokens_out":1387,"duration_ms":10693,"temperature":1.0,"reasoning_tokens":1316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:32:04.778544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the simple jet engine problem in the same SBArchOpt framework: run NSGA-II for 3250 evaluations and the proposed BO algorithm for 300 evaluations under identical initial design-of-experiments and termination settings. If NSGA-II's final TSFC is more than 0.2% below BO's, or if BO cannot reach the reference TSFC value within 300 evaluations, the 92% claim is refuted.","supporting_citations":[{"cited_title":"In: AIAA A VIATION 2021 FORUM, Virtual Event (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the jet engine benchmark problem and the NSGA-II 3250-evaluation optimum that the 92% reduction is measured against."},{"cited_title":"In: AIAA A VIATION 2024 FORUM, Las Vegas, NV, USA (2024)","cited_arxiv_id":null,"evidence_quote":"Provides the probability-of-viability handling used for the jet engine's approximately 50% hidden-constraint failure rate."},{"cited_title":"In: AIAA A VIATION 2021 FORUM, Virtual Event (2021)","cited_arxiv_id":null,"evidence_quote":"Introduces the infill ensemble and MEPoI strategy that the BO algorithm builds on."}],"review_version":1}