{"id":"98beeed9-1069-4c99-b15e-99584fca691a","arxiv_id":"2608.03045","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Bilevel Bayesian optimization that optimizes only black-box variables and solves the known white-box subproblem exactly reports 11x to 10^8x lower regret than full-space BO on 13 separable grey-box benchmarks.","lead":"Researchers reformulate grey-box optimization problems as a bilevel program: a Bayesian optimizer searches only the black-box variables while a deterministic global solver handles the known white-box equations. The paper reports 11x to 10^8x lower regret than monolithic Bayesian optimization on 13 synthetic benchmarks, which could cut expensive simulations in materials and process design.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own data contradict the headline claims: Bi-BO (BH) loses to black-box BO on SFR-1 and is 11.5x slower on Williams-Otto, and four constrained problems show 59-91% feasibility, so 'robust to inner-solver choice' and 'exact constraint satisfaction' are overstated.","rationale":"Proposition 1 is correct as a conditional equivalence: if the inner subproblem is solved to global optimality at every outer iterate, the bilevel reformulation preserves the global optimum and reduces the surrogate to R^{n_BB}. The empirical study is unusually careful in scale (8,450 runs) and the code is released, which strengthens confidence in the reproducibility of the stated numbers. The most load-bearing weakness is not the math but the translation of a conditional guarantee into unconditional headline claims. The paper's own Table 3, Figure 7, and the Limitations section document that the inner solver can fail to find feasible points (feasibility 59-91%), can lose to the black-box baseline (SFR-1 with BH), and can dominate wall time (Williams-Otto). These are not hypothetical failures of an unstated assumption; they are observed outcomes in the reported experiments. Since the abstract and Section 1 present 'exact constraint satisfaction' and 'robust to inner-solver choice' as signature advantages, the central claim is overstated as written. The correct framing is: the method is highly effective when a reliable inner solver is available for the white-box subproblem; solver choice is problem-dependent. This warrants a CONDITIONAL verdict requiring the authors to qualify the abstract, add the planned baselines (COBALT, BOCF), and provide a decision rule or sensitivity analysis for inner-solver selection. I therefore agree with the reader's assessment and recommend no change to the verdict.","tokens_in":37899,"tokens_out":12043,"duration_ms":113831,"concrete_test":"Worth running: reproduce Table 3 on the public code repository with two fixed inner solvers (SLSQP-50 and BH-100) and a single fixed xi (e.g., 0.01) across all 13 problems, 10 seeds, and record per-run regret, wall time, and maximum constraint violation over the 250 black-box evaluations. If Bi-BO (BH) remains worse than BB-BO on SFR-1 in most seeds and >10x slower on Williams-Otto, and if any problem has a maximum constraint violation greater than 1e-3, then the robustness/exactness claims in the abstract require explicit qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that the bilevel strategy is robust to inner-solver choice and satisfies white-box constraints exactly—is undercut by the paper's own results. Proposition 1 and the abstract's 'exact' constraint-satisfaction framing are conditional on Assumption 1(ii): the inner white-box subproblem must be solved to global optimality, cheaply, at every outer query. Table 3 shows this condition fails in practice: Bi-BO (BH) is worse than black-box BO on SFR-1 (regret 0.0334 vs 0.0048) and 11.5x slower on Williams-Otto (3,130 s vs 273 s), and Figure 7 reports feasibility of only 59-91% on Batch-Reactor, Evaporator, SFR-1, and PSA rather than exact satisfaction. The paper's response in Section 7—that this is a limitation of the inner solver, not the architecture—does not rescue the headline because the user is given no principled way to choose the inner solver; the aggregate sign test (SLSQP wins 6, BH wins 7, p=1.0) masks per-problem variance of 5+ orders of magnitude (Rastrigin: BH achieves 7.2e7 improvement, SLSQP 698; SFR-1: BH 0.1x, SLSQP 11x). Thus the practical claim 'robust to inner-solver choice' fails, and the 'cheap inner solve' overhead assumption (Remark 4) is falsified on the paper's own benchmark. Additionally, the abstract's 'fewer iterations and wall clock time' is contradicted by Williams-Otto, where even Bi-BO (SLSQP) is 1.19x slower than BB-BO (326 s vs 273 s).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a bilevel reformulation for grey-box optimization problems whose decision variables split into black-box variables x_BB and white-box variables x_WB. An outer Bayesian optimization loop searches over x_BB alone, while an inner global optimizer solves the white-box NLP over x_WB for each outer query. The authors argue that this reduces the GP surrogate domain from R^{n_WB+n_BB} to R^{n_BB}, preserves the global optimum under Assumption 1, and enforces white-box constraints exactly without penalties. They introduce a 13-problem benchmark suite and report 8,450 optimization runs comparing bilevel BO with black-box BO and black-box BH baselines, claiming 11x to 10^8x lower regret with equal or better wall time on most problems.","tokens_in":38287,"tokens_out":5131,"duration_ms":54857,"significance":"The structural decomposition idea is attractive and timely: separating a known white-box subproblem from an expensive black-box component is a principled way to reduce surrogate dimension and avoid surrogate-based constraint approximation. The manuscript ships a reproducible benchmark suite, a large number of statistically analyzed runs, and a clearly stated Proposition 1 whose proof is correct under Assumption 1. These are concrete strengths. The empirical evidence is extensive, with multiple hypothesis tests, hyperparameter sweeps, and per-problem convergence plots. However, the significance is reduced by the mismatch between the unconditional claims in the abstract and conclusions and the conditional nature of the underlying assumption, and by the absence of any empirical comparison with the closest grey-box BO baselines, which limits what the benchmark comparison can establish.","major_comments":[{"comment":"The claims that white-box constraints are satisfied exactly and that performance is robust to inner-solver choice are contradicted by the paper's own results. Table 3 shows Bi-BO (BH) with regret 0.0334 on SFR-1 versus 0.0048 for black-box BO, and wall time 3,130 s versus 273 s on Williams-Otto. Figure 7 reports 59-91% feasibility on Batch-Reactor, Evaporator, SFR-1, and PSA, rather than 100%. Section 7 acknowledges these failures as limitations of the inner solver, but the abstract and conclusion repeat 'exact constraint satisfaction' and 'robust to inner-solver choice' without these caveats. These claims should be reworded as conditional on Assumption 1(ii) holding for the chosen inner solver, or the empirical sections should separately report the failure modes and their frequency.","section":"Abstract, §4.3, Table 3, Figure 7"},{"comment":"The empirical comparison does not include any state-of-the-art grey-box BO baseline. The baselines are a full-space BO with a fixed penalty of 10^6 and a full-space Basin-Hopping solver, but the paper's contributions in Section 1 and Table 1 position the method against COBALT, BOCF, Kieffer et al., and other grey-box approaches. Section 7 states that a direct comparison with COBALT and BOCF is planned for future work. Without such a comparison, the headline '11x-10^8x lower regret' only demonstrates an advantage over a deliberately simple penalty-based baseline; it does not establish the claimed advantage over the closest existing grey-box methods, especially because those methods also reduce input dimensionality in some cases.","section":"§6.1.1, §6.3, §7"},{"comment":"The paper relies on Assumption 1(ii) for its theoretical guarantee, but the implemented inner solvers (multi-start SLSQP and Basin-Hopping) are heuristics that do not certify global optimality. Moreover, the assumption that the inner solve is cheap relative to the black-box evaluation is not supported by the benchmark data: on Williams-Otto, even Bi-BO (SLSQP) is 1.19x slower than black-box BO, and Bi-BO (BH) is 11.5x slower. These are exactly the cases where the method's practical viability depends on the inner solver being reliable and cheap. The paper should either restrict its claims to settings where the inner solver's global optimality is verifiable, or provide a more careful analysis of when the wall-clock overhead can be tolerated.","section":"§4.2, Remark 4, Table 3"},{"comment":"The headline regret ratios are computed using per-problem, per-method best values of the exploration parameter xi, as stated in Table 3 and Appendix J. This is a favorable selection for both methods, but the specific numbers 11x-10^8x and the geometric mean 3,192x are upper-tail estimates rather than typical performance. Figure 6 shows that Bi-BO (SLSQP) beats black-box BO for all 273 (problem, n_init, xi) configurations, which supports robustness, but the paper should report median or quartile regret ratios across the configuration grid in addition to the best-xi values, so that the headline reflects the distribution of performance rather than a cherry-picked configuration.","section":"§6.3, Table 3, Appendix J, Figure 6"}],"minor_comments":[{"comment":"The sentence 'The Gaussian process surrogate used in BO is therefore defined rather than and white-box constraints are satisfied exactly' appears to be missing words; it should read something like 'defined on R^{n_BB} rather than on R^{n_WB+n_BB}'.","section":"Abstract"},{"comment":"The naming is inconsistent: 'Small-Feasible-Region 1' and 'Small-Feasible-Region-2' are used interchangeably, and the same inconsistency appears in Table 3 and Figure 7. Please standardize the problem names.","section":"Table 2 and throughout"},{"comment":"The verification section says all optima agree to within 2x10^-4 relative error, but for problems with J* = 0 (e.g., Rastrigin and Membrane in Table 18), a relative error is undefined; an absolute tolerance should be specified as well.","section":"Appendix F"},{"comment":"Table 3 reports several Bi-BO regret values as exactly 0.0000, but the paper later states a geometric mean improvement of 3,192x; ratios involving zero regret are undefined, so the paper should explain how zero regrets were treated in the aggregate statistics.","section":"§6.3"},{"comment":"The black-box BH baseline is described as using 1,000 iterations in Section 6.1.1 and Appendix D, but the failure-mode discussion says 'the BH baseline fails on Rastrigin' with the implication that more iterations might help. The number of iterations used in the benchmark should be stated consistently and motivated.","section":"§6.3 and Appendix D"},{"comment":"The statement that a direct comparison with COBALT and BOCF is 'planned for future work' is repeated in the limitations section; given that these methods are close competitors, the paragraph would be stronger if it stated which specific experiments are needed to settle the comparison.","section":"§7, 'Complementarity with existing grey-box BO'"}],"recommendation":"major_revision","confidential_remarks":"The paper's central structural idea is sound, and the empirical infrastructure is a real asset. The main risk is overclaiming: the abstract and conclusions assert exactness and robustness that the paper's own tables and figures only partially support. A revision that tones down these claims, adds a baseline from the grey-box BO literature, and reports aggregate rather than best-xi results would make the contribution solid. The self-citation of Baldea is contextual and not a concern. The benchmark suite is entirely closed-form, so validation on a genuinely expensive black-box simulation would substantially strengthen the practical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is sound and demonstrated, but the packaging oversells it. The bilevel reformulation—BO only over the black-box variables with an exact solve of the white-box NLP inside—is a clean, useful way to cut surrogate dimension, and Proposition 1 is correct under Assumption 1. The 13-problem suite with verified optima is a genuine asset for the community, and the 8,450-run comparison is careful: Wilcoxon, Friedman/Nemenyi, sensitivity sweeps, and code on GitHub. That is real work.\n\nThe soft spots are in the claims, not the architecture. The abstract says 'white-box constraints are satisfied exactly' and the advantage is 'robust to inner-solver choice.' The paper's own data say otherwise: on four constrained problems feasibility is 59–91%; with Basin-Hopping inside, Bi-BO loses to black-box BO on SFR-1 and is 11.5× slower on Williams-Otto. The paper acknowledges all of this in Section 7, which is honest, but then the abstract and conclusion repeat the blanket claims. 'Robust' needs a conditional: robust when the inner solver actually solves the subproblem, which is exactly Assumption 1(ii) and not guaranteed by the architecture.\n\nThe experimental design is workmanlike but leaves the strongest case unmade. The black-box baseline is vanilla BO with a fixed penalty; no comparison to Kieffer et al., COBALT, or DOMINO, so we don't learn how the method compares to existing grey-box alternatives. And the headline regret ratios use per-problem best ξ selected on the test problems, which optimistically biases the numbers. The problem class is also narrow—2–5 variables, and every 'black box' is a closed-form surrogate. The claimed 80–100% evaluation savings is plausible for exactly this separable structure, but it is not yet supported for real DFT/MD-scale costs.\n\nWho should read it: anyone building surrogate methods for multi-scale design, and anyone constructing grey-box benchmarks. It deserves a serious referee. I would send it out, but ask the authors to temper the abstract and conclusion, add at least one existing grey-box baseline, and report results without oracle ξ. The method is worth engaging with; the claims just need to match the evidence.","headline":"A sound separable-grey-box method whose headline claims outrun its own data; referee-worthy, but the abstract and conclusions need an honest rewrite.","tokens_in":38841,"tokens_out":2286,"would_cite":true,"duration_ms":23581,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C26","90C30","90C56"],"pacs":[],"model":"deepseek-v4-flash","headline":"Separating known from unknown variables cuts optimization regret by up to 10^8.","keywords":["Bayesian optimization","grey-box optimization","bilevel optimization","surrogate modeling","dimensionality reduction","exact constraint satisfaction","multi-scale design","benchmark suite"],"falsifier":"Take a separable problem with a cheap black box (e.g., $y = x_{BB}$) and a white-box subproblem with many local minima, run the bilevel method with a fixed inner-solver budget, and compare final regret against the verified global optimum: if the inner solver systematically returns local optima, the value-function surrogate learns the wrong scalar landscape and Proposition 1's optimality preservation fails, showing up as regret no better than black-box BO.","tokens_in":37688,"feed_emoji":"🧩","tokens_out":8585,"duration_ms":77723,"temperature":0.7,"pith_summary":"The paper claims that many expensive optimization problems contain a separability that standard Bayesian optimization ignores: some variables enter a cheap, explicitly known set of equations (white-box variables), while others enter only an expensive black-box function whose outputs feed those equations. It proposes reformulating such problems as a bilevel program, with an outer Bayesian loop over the black-box variables alone and an inner global optimizer that solves the white-box subproblem exactly for each candidate. Because the Gaussian-process surrogate now lives in the low-dimensional black-box space instead of the joint space, dimensionality is reduced and white-box constraints are enforced by the inner solver rather than by penalties. On a suite of 13 benchmark problems with verified optima, the bilevel method achieves 11-fold to $10^{8}$-fold lower regret than full-space black-box BO, with equal or better wall time on most problems.","feed_headline":"Split known and unknown variables, cut optimization regret 10^8-fold","feed_subtitle":"Bilevel approach shrinks the surrogate to black-box variables and enforces white-box constraints exactly.","key_machinery":"The central object is the bilevel reformulation of the joint grey-box problem: an outer Bayesian optimization loop over black-box variables $x_{BB}$, and an inner nonlinear program over white-box variables $x_{WB}$ that is solved to (near) global optimality by Basin-Hopping or multi-start SLSQP for each candidate. The carrier of the argument is the scalar value function $x_{BB} \\mapsto J(x_{WB}^*(x_{BB}), f_{BB}(x_{BB}), x_{BB})$, which replaces the joint objective and lets a single GP with an ARD RBF kernel and Expected Improvement acquisition operate in $\\mathbb{R}^{n_{BB}}$ rather than $\\mathbb{R}^{n_{WB}+n_{BB}}$. Proposition 1 uses Assumption 1 to show the two formulations share the same global optimum, so dimensionality reduction costs nothing in solution quality; the inner optimizer is what makes constraint satisfaction exact rather than approximate.","core_discovery":"Under a separability assumption (the black-box function depends only on $x_{BB}$, and for every fixed $x_{BB}$ the inner white-box subproblem is solvable to global optimality), the paper establishes Proposition 1: the bilevel reformulation preserves the global optimum of the original joint problem and reduces the surrogate domain from $\\mathbb{R}^{n_{WB}+n_{BB}}$ to $\\mathbb{R}^{n_{BB}}$. The reformulation replaces the joint objective $J(x_{WB}, y, x_{BB})$ with the scalar value function $x_{BB} \\mapsto J(x_{WB}^*(x_{BB}), f_{BB}(x_{BB}), x_{BB})$, where the inner solve returns the best feasible $x_{WB}$. This makes white-box constraints exact whenever the inner optimizer converges to a feasible point, with no penalty functions, chance constraints, or moment propagation. Empirically, on 8,450 runs over 13 problems, the SLSQP-inner variant beats black-box BO on all 13 problems with 11x to $10^{8}$x lower regret, and the improvement is robust to initialization size and exploration parameter; the Basin-Hopping inner variant wins on 12 of 13, failing only where its random perturbations overshoot narrow feasible regions.","pith_inferences":["If the exact-inner-solve premise holds only approximately—inner solves are local, not global—the equality in Proposition 1 degrades; the observed 59-91 percent feasibility on four constrained problems suggests that a deterministic global inner solver would be needed to restore the exactness guarantee in practice.","The same decomposition could be applied to other surrogate families: any regression model over $x_{BB}$ plus an exact inner solver would inherit the dimensionality reduction, so the result is not specific to GPs or EI.","The value-function viewpoint suggests a testable extension: when the inner problem is solved parametrically offline as a function of $y$, the outer loop could switch from nested solves to direct evaluation of a precomputed value function, eliminating inner cost entirely.","For problems with pure black-box constraints not filtered through the white box, the paper's exactness argument stops at the white-box constraints; combining the outer loop with constrained acquisition would be needed, which the paper lists as an open problem."],"forward_implications":["When the inner subproblem can be solved globally and cheaply, practitioners can replace a full-space GP with a lower-dimensional GP and obtain the same global optimum at a fraction of the black-box evaluations.","On tightly constrained problems such as Heat-Exchanger and Distillation, exact inner constraint handling removes the penalty method's failure mode, yielding regret reductions of 10^6 to 10^8.","The choice of inner solver matters: multi-start SLSQP is faster and handles narrow feasible regions, while Basin-Hopping handles multimodal inner landscapes; neither wins everywhere.","The method's gains come from problem structure, not from novel surrogate or acquisition choices, since the paper uses the same GP kernel and EI as the black-box baseline.","Sample-efficiency crossover shows bilevel BO can match black-box BO's final performance within 0-47 evaluations on 13 problems, meaning multi-hour black-box evaluations can be saved."],"supporting_citations":[{"why":"supplies the Efficient Global Optimization baseline and Expected Improvement acquisition against which bilevel BO is measured.","marker":"[10]"},{"why":"defines composite-function Bayesian optimization, the main uncertainty-propagation alternative the paper contrasts with scalar value-function surrogates.","marker":"[5]"},{"why":"COBALT is the closest grey-box BO competitor that also reduces the surrogate input space, but uses chance constraints instead of exact inner solves.","marker":"[6]"},{"why":"DOMINO is the closest non-BO predecessor, solving the inner subproblem globally at each sample while using deterministic surrogates for the outer search.","marker":"[8]"},{"why":"first applied BO to bilevel problems with EI outer and SLSQP inner, but did not exploit separability to shrink the surrogate domain.","marker":"[9]"},{"why":"introduced the multi-scale bilevel BO framework that this paper extends with a 13-problem suite and hyperparameter sweeps.","marker":"[25]"},{"why":"provides the small-feasible-region constrained test problems used to visualize the search-mechanism difference.","marker":"[26]"},{"why":"supplies the second family of small-feasible-region test problems used in the benchmark suite.","marker":"[39]"}],"fun_headline_variants":["Bilevel split shrinks surrogate, cuts regret up to 10^8x","Exact constraints, lower regret: bilevel BO separates variables","Separable grey-box optimization: bilevel reformulation wins big","Cut regret 100 million-fold by splitting known and unknown variables","Bilevel BO: smaller surrogate, exact constraints, lower regret"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that for every black-box variable choice the inner white-box subproblem can actually be solved to global optimality by a practical optimizer, and that this solve is cheap compared with evaluating the black-box function; the paper's own data show this premise fails on narrow-feasible-region problems, where the Basin-Hopping inner solver regresses or becomes much slower.","fun_headline_variants_meta":{"raw":{"variants":["Bilevel split shrinks surrogate, cuts regret up to 10^8x","Exact constraints, lower regret: bilevel BO separates variables","Separable grey-box optimization: bilevel reformulation wins big","Cut regret 100 million-fold by splitting known and unknown variables","Bilevel BO: smaller surrogate, exact constraints, lower regret"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000815,"raw_usage":{"total_tokens":3580,"prompt_tokens":960,"completion_tokens":2620,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2530}},"tokens_in":576,"tokens_out":2620,"duration_ms":17947,"temperature":1.0,"reasoning_tokens":2530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:59:13.710599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a separable problem with a cheap black box (e.g., $y = x_{BB}$) and a white-box subproblem with many local minima, run the bilevel method with a fixed inner-solver budget, and compare final regret against the verified global optimum: if the inner solver systematically returns local optima, the value-function surrogate learns the wrong scalar landscape and Proposition 1's optimality preservation fails, showing up as regret no better than black-box BO.","supporting_citations":[{"cited_title":"A multiscale Bayesian optimization framework for process and material codesign.AIChE Journal, 2026","cited_arxiv_id":null,"evidence_quote":"introduced the multi-scale bilevel BO framework that this paper extends with a 13-problem suite and hyperparameter sweeps."},{"cited_title":"Bayesian optimization with inequality constraints","cited_arxiv_id":null,"evidence_quote":"provides the small-feasible-region constrained test problems used to visualize the search-mechanism difference."},{"cited_title":"Admmbo: Bayesian optimization with un- known constraints using admm.Journal of Machine Learning Research, 20(123):1–26, 2019","cited_arxiv_id":null,"evidence_quote":"supplies the second family of small-feasible-region test problems used in the benchmark suite."}],"review_version":1}