{"id":"36c308b3-cb31-436b-8b82-cc084adb4e87","arxiv_id":"2412.13948","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A tutorial-and-benchmark chapter that ranks ten surrogate-based derivative-free optimization algorithms on four synthetic functions and two process engineering case studies, with code released on GitHub.","lead":"This book chapter reviews surrogate-based (model-based) derivative-free optimization and benchmarks ten algorithms on four test functions plus two chemical engineering case studies. It reports RBF-based DYCORS and SRBF as the strongest on synthetic problems, and CUATRO-pls as best on a 32-dimensional PID controller tuning task.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark rankings rest on unreconciled run counts (5 vs 10) and no variance reporting; close p_a differences in Tables 2 and 4 are not statistically supported.","rationale":"The reader's weakest assumption is the same load-bearing concern I identify: the rankings depend on a small, undocumented number of runs with no variance or significance information. I agree fully. I would sharpen the point by noting the explicit conflict between Section 3.1 ('five times') and the figure/table captions ('10 repetitions'), which means the reported convergence intervals may correspond to a different experimental setup than the p_a scores themselves. This is not a fundamental mathematical error in the algorithms or the tutorial content, and the released code is a genuine asset; it is a reproducibility and statistical-support gap in the empirical claims. The appropriate disposition is therefore the reader's CONDITIONAL verdict: accept if the authors reconcile the run counts and supply variance-aware comparisons, otherwise the rankings should be treated as provisional. I do not see grounds for REJECT or for a stronger ACCEPT, and I found no indication of fabrication or cherry-picking; the in-house algorithms perform poorly in several blocks, which is consistent with honest reporting.","tokens_in":32292,"tokens_out":3999,"duration_ms":37364,"concrete_test":"Re-run the released GitHub benchmark for the unconstrained suite with the authors' code, using 50 independent repetitions, and compute bootstrap 95% confidence intervals for each p_a in Table 2 under the authors' scoring rule. If the interval for BO's 0.82 overlaps COBYLA's 0.81, or if the intervals for DYCORS and SRBF overlap at the reported 0.01–0.02 scale, the claimed ranking is not established; if the intervals are disjoint at that scale, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims—DYCORS best on unconstrained synthetics (§3.2.5), CUATRO-pls best on the PID case (§5.1.2, Table 4), and COBYQA/COBYLA tied on Williams-Otto (Table 5)—are point estimates from a scoring rule that assumes each algorithm's mean trajectory is representative. Section 3.1 says each algorithm is run 'five times,' and Eq. 35 says the mean is 'derived from the averaging of five optimization runs.' Every figure caption (Figs. 2, 3, 4, 8, 10) and Table 3 instead state '10 repetitions from 10 different starting points.' The paper never resolves whether p_a in Tables 2, 4, and 5 was computed from 5 or 10 runs, nor does it report per-algorithm variance, confidence intervals, or significance tests. The problem is amplified by the scoring definition: Eq. 36 averages r values that are normalized by the contemporaneous best and worst trajectories; near convergence, tiny absolute differences in y can produce large swings in r. Thus the 0.01 gap separating BO (0.82) from COBYLA (0.81) in Table 2, and the 0.02–0.11 gaps among ranks 2–6 in Table 4, are within any reasonable noise band. Because the chapter's practical deliverable is directional algorithm selection, these unresolved statistical issues make the headline rankings vulnerable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This book chapter surveys surrogate-based (model-based) derivative-free optimization methods for process systems engineering. It presents the mathematical foundations of ten algorithms (Bayesian optimization, ENTMOOT, COBYLA, COBYQA, LSQM, CUATRO/CUATRO-pls, SNOBFIT, DYCORS, SOP, SRBF), then benchmarks them on unconstrained synthetic test functions (Ackley, Levy, Rosenbrock, quadratic) in 2, 5, and 7 dimensions, on constrained synthetic problems, and on two chemical engineering case studies: a 32-dimensional CSTR-PID controller tuning problem and a low-dimensional Williams-Otto reactor problem. The headline empirical claims are that DYCORS is the best unconstrained synthetic solver, CUATRO-pls is the best on the PID case, and COBYQA and COBYLA tie for best on the Williams-Otto case. The chapter also provides a GitHub repository with the benchmarking code and emphasizes that its conclusions are relative to the algorithm set and problem set considered.","tokens_in":32567,"tokens_out":5199,"duration_ms":43928,"significance":"If the benchmark results are reliable, the chapter offers a useful, reproducible reference for practitioners choosing surrogate-based optimization algorithms in process systems engineering. Its strengths include the public code repository, the inclusion of both off-the-shelf and in-house implementations (some of which perform poorly on several problems, which reduces concern about selective reporting), and the explicit caveat that the rankings are relative to the chosen algorithm and function sets. The tutorial material on algorithm families is generally accurate and well organized. However, the quantitative rankings are the main contribution, and they currently rest on an unreconciled repetition-count discrepancy, an undefined protocol for 7D runs, and the absence of variance or significance reporting. These issues are load-bearing because the practical deliverable is directional algorithm selection.","major_comments":[{"comment":"The repetition count is inconsistent: Section 3.1 states that each algorithm is run 'five times' and Eq. (35) says ymean is 'derived from the averaging of five optimization runs,' while the captions of Figures 2, 3, 4, 8, and 10 and Table 3 state '10 repetitions from 10 different starting points.' Since every quantitative score p_a in Tables 2, 4, and 5 is computed from the averaged trajectory ymean_{k,a}, this discrepancy directly affects all ranking claims. The authors must state which number of repetitions was actually used, and should report per-algorithm variance, confidence intervals, or significance tests. Without this, close scores such as BO 0.82 versus COBYLA 0.81 overall in Table 2, or the 0.02-0.11 gaps among ranks 2-6 in Table 4, cannot be distinguished from noise.","section":"Section 3.1 and Eq. (35)"},{"comment":"The evaluation budget and burn-in length are defined only for nx = 2, 5, and 10, but the unconstrained benchmark uses nx = 2, 5, and 7 (Table 1, Figure 2). The budget of '100 function evaluations for nx = 10' and the burn-in of 'nc = 15 for nx = 10' have no stated analogue for the 7-dimensional runs that produce the D7 rows in Table 2. The D7 scores are therefore not reproducible from the described protocol. Please specify the 7D evaluation budget and burn-in, or remove the 7D results.","section":"Section 3.1 and Table 1"},{"comment":"The scoring metric normalizes each trajectory point by the contemporaneous best and worst trajectories among the algorithm set. Because these extrema are themselves noisy estimates, the resulting r and p_a values are not effect sizes with a meaningful absolute scale; near convergence, small absolute differences in objective value can produce large swings in the normalized score. The rankings in Sections 3.2.5, 5.1.2, and 5.2.1 are reported as point estimates with no significance testing. Please add uncertainty quantification (e.g., bootstrap confidence intervals or paired tests) or explicitly temper the ranking claims to directional observations, especially where score gaps are small.","section":"Eqs. (35)-(36) and Section 3.2.5"},{"comment":"The constraint-violation threshold of 0.001 is selected after observing that 'when the algorithms commence evaluating alongside the constraint, there are very small constraint violations e.g. 0.0008.' This ex post choice directly affects the constraint-satisfaction percentages in Table 3 and the 'Feasible Samples' column in Table 5. Please report how sensitive those columns are to the threshold, or justify the threshold on an a priori basis independent of the observed violations.","section":"Section 4.4.1"}],"minor_comments":[{"comment":"Section 5 states that the in-house BO implementation is replaced by GPyOpt and TuRBO for the PID case, but Table 4 lists both 'BO' and 'TURBO' as separate rows. Please clarify which implementation the 'BO' row refers to and whether 'TURBO' is the TuRBO algorithm.","section":"Section 5.1.2 and Table 4"},{"comment":"The subcaptions in Figure 3 do not identify which algorithms' trajectories are shown, and the phrase 'exemplary optimization trajectories' does not indicate the specific run or starting point. Please add a legend or state the algorithm names in each subcaption.","section":"Figure 3"},{"comment":"The text refers to an 'Ill-Constrained Quadratic' function; this appears to be a typo for 'Ill-Conditioned Quadratic' and should be corrected for consistency with Section 3.2.1.","section":"Section 4.4.1"},{"comment":"The statement that the RBF coefficient matrix is invertible 'if and only if rank(P) = nx + 1' is incomplete; standard RBF interpolation also requires distinct data points and suitable conditions on the kernel (e.g., conditional positive definiteness). Please rephrase to state the conditions accurately.","section":"Section 2.10.1, Eq. (34)"},{"comment":"COBYQA and COBYLA both receive p_a = 1.00, yet they have different feasible-sample rates (81.75% and 84.28%) and mean violations. Please clarify whether p_a is computed over all trajectories or only over feasible ones, and how the constraint-satisfaction metric interacts with the objective-based score.","section":"Section 5.2.1, Table 5"},{"comment":"The concluding text says SNOBFIT 'performed much better in this case study,' but Table 4 places SNOBFIT sixth with a score of 0.61, which is mid-pack rather than a clear improvement. Please rephrase to reflect the actual ranking.","section":"Section 7.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has the character of an educational book chapter combined with a benchmarking study. The tutorial portions are serviceable, but the empirical claims need to meet the standards of a journal publication. The repetition-count inconsistency and missing 7D protocol are objective reproducibility problems, and the absence of variance reporting makes the rankings difficult to evaluate. These are fixable within the scope of the manuscript by rerunning or clarifying the protocol and adding uncertainty quantification; hence I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on Neufang et al. It is a tutorial-and-benchmark book chapter, not a research paper, and it should be judged on that footing. The tutorial content is genuinely useful: clear, equation-level descriptions of BO, TuRBO, ENTMOOT, COBYLA, COBYQA, LSQM, CUATRO/CUATRO-pls, SNOBFIT, and the Shoemaker RBF family, with the standard workflow laid out. The released GitHub code is a real asset; readers can rerun every benchmark. The two case studies (CSTR PID tuning, Williams-Otto) are established PSE problems and add credibility beyond the synthetic test functions. I also note the in-house algorithms (LSQM, CUATRO, CBO) score worst in several blocks, which argues against cherry-picking.\n\nThe soft spots are concentrated in the benchmarking methodology. Section 3.1 says each algorithm is run five times, while every figure caption and Table 3 say '10 repetitions from 10 different starting points.' The paper never reconciles the numbers, and Equation 35 explicitly averages five runs. No variance, confidence intervals, or significance tests accompany the p_a scores. Because the scoring rule normalizes by the contemporaneous best and worst trajectories, tiny absolute differences near convergence can swing r substantially. The 0.01 gap between BO (0.82) and COBYLA (0.81) in Table 2, and the 0.02–0.11 gaps among ranks 2–6 in Table 4, sit inside any plausible noise band. The 0.001 constraint-violation threshold is chosen after observing violations around 0.0008, which is post hoc. And Section 7.2 attributes synthetic-benchmark behavior to CUATRO-pls, which only ran in the PID case study; that is an error the authors should correct.\n\nNone of these are fatal. The chapter is explicitly framed as directional guidance, and the authors say as much. But for a benchmark that ships code, the run-count inconsistency and missing variance reporting are avoidable. A referee should ask for reconciliation, per-algorithm variance or significance measures, and a corrected conclusion. The rankings might survive those changes, but they need the statistical support to be credible.\n\nThe intended audience is newcomers to surrogate-based optimization in process systems engineering who want a readable overview and a starting point for algorithm selection. It deserves a serious referee: the tutorial material is useful, the code is reproducible, and the empirical flaws are fixable. I would not cite it in my own work, but I would point students to the repository.","headline":"Useful tutorial and reproducible code, but the benchmark rankings rest on an unreconciled run-count inconsistency and no variance reporting.","tokens_in":33149,"tokens_out":4018,"would_cite":false,"duration_ms":31900,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C56","90C30","65K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This chapter benchmarks ten surrogate-based optimization algorithms on unconstrained and constrained test problems plus two chemical engineering case studies, finding that DYCORS leads on synthetic unconstrained problems while CUATRO-pls…","keywords":["surrogate-based optimization","derivative-free optimization","Bayesian optimization","radial basis functions","trust region methods","process systems engineering","black-box optimization","algorithm benchmarking"],"falsifier":"Rerunning the benchmarking code supplied by the authors with 50 or more independent restarts per algorithm and computing confidence intervals on the normalized scores $p_a$ would settle whether the reported orderings are stable; if DYCORS versus SRBF, or COBYQA/COBYLA versus CBO, swap places under this wider sampling, the chapter's directional conclusions would not be supported.","tokens_in":32007,"feed_emoji":"🧪","tokens_out":3734,"duration_ms":32367,"temperature":0.7,"pith_summary":"The paper is a book chapter aiming to give process systems engineers a practical guide to surrogate-based optimization, also called model-based derivative-free optimization, with both theoretical background and a comparative performance assessment. It claims to provide reproducible, directional evidence on which algorithms to choose: DYCORS is the best on unconstrained synthetic benchmarks, CUATRO-pls is best on a 32-dimensional CSTR PID controller tuning case, and COBYQA and COBYLA are the best of the tested methods on the constrained Williams-Otto process. The chapter also supplies code so that practitioners can rerun the comparisons on their own problems.","feed_headline":"Surrogate solver benchmark: DYCORS wins synthetic, CUATRO-pls PID","feed_subtitle":"Directional rankings across test functions and two reactor case studies guide algorithm choice for chemical engineering.","key_machinery":"The central machinery is the normalized trajectory scoring metric $r_{k,a}$ and its aggregate $p_a$ (Eqs. 35-36): at each function-evaluation count $k$, the mean best-so-far value of algorithm $a$ is compared to the best and worst values among all algorithms, and this relative score is averaged over the trajectory. This yields a number between 0 and 1 that ranks algorithms on a common normalized scale, enabling cross-algorithm comparison across test functions and case studies with different budgets.","core_discovery":"On its own terms, the chapter's central discovery is an empirical ranking of ten surrogate-based optimization algorithms under a common evaluation protocol using a normalized trajectory score $p_a$ (Eqs. 35-36) that measures how close each algorithm's best-so-far trajectory stays to the best trajectory among the competitors. DYCORS tops the unconstrained synthetic benchmarks across the Ackley, Levy, Rosenbrock, and ill-conditioned Quadratic functions in 2, 5, and 7 dimensions; on the 32-dimensional PID tuning case, the dimensionality-reducing CUATRO-pls is the best performer; and on the constrained Williams-Otto case, COBYQA and COBYLA achieve perfect scores. The chapter also documents that algorithm rankings differ between synthetic functions and engineering case studies, so performance on test functions alone is not a reliable guide.","pith_inferences":["The reported score differences, such as BO at 0.82 versus COBYLA at 0.81 overall, are not backed by statistical tests; a reader rerunning the supplied code with many more restarts could determine whether these orderings are stable.","The success of CUATRO-pls hints that many process systems engineering problems have low intrinsic dimensionality, so other subspace-based or dimensionality-reducing surrogate methods could be expected to perform well beyond the PID case studied.","The discrepancy between the stated five runs per algorithm in Section 3.1 and the ten repetitions referenced in figure captions and Table 3 should be resolved before treating any of the numerical scores as precise.","The comparison framework itself, based on best-so-far trajectory normalization, could be transferred to constrained, multi-objective, or noisy settings as a standardized way to report solver comparisons."],"forward_implications":["Practitioners can use DYCORS or SRBF as a first choice for unconstrained, low-to-moderate dimensional expensive black-box problems.","For high-dimensional problems with latent low-dimensional structure, such as controller tuning, CUATRO-pls offers a meaningful advantage over standard Bayesian optimization.","For constrained problems where constraint satisfaction matters, COBYQA and COBYLA are the safer choices among the tested methods.","Synthetic test-function rankings should be supplemented with engineering case studies, since relative performance shifted substantially between the two settings.","The normalized score methodology gives a reusable template for comparing derivative-free solvers on a user's own objective.","The open-source code released with the chapter allows the benchmarks to be extended to new functions and case studies."],"supporting_citations":[{"why":"Supplies the original CUATRO algorithm and the general data-driven optimization perspective that the chapter builds on.","marker":"[2]"},{"why":"Provides the COBYQA quadratic model-based trust-region method and its software.","marker":"[46]"},{"why":"Defines ENTMOOT, the decision-tree-based surrogate optimization framework benchmarked in the chapter.","marker":"[50]"},{"why":"Defines SNOBFIT, the branch-and-fit surrogate algorithm used in the benchmarking.","marker":"[51]"},{"why":"Introduces TuRBO, the trust-region Bayesian optimization variant included in the high-dimensional PID case study.","marker":"[69]"},{"why":"Introduces CUATRO-pls, the dimensionality-reducing extension that won the 32-dimensional controller tuning benchmark.","marker":"[75]"},{"why":"Defines DYCORS, the radial basis function dynamic coordinate search algorithm that ranked first on unconstrained synthetic benchmarks.","marker":"[76]"},{"why":"Supplies the Williams-Otto process model and real-time optimization benchmark used for the constrained engineering case study.","marker":"[84]"}],"fun_headline_variants":["DYCORS tops synthetic, CUATRO-pls wins PID tuning","Surrogate solver picks: DYCORS for tests, CUATRO-pls for PID","For reactor PID, CUATRO-pls beats DYCORS; synthetic reverses","Surrogate benchmarks: DYCORS wins synthetic, CUATRO-pls PID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings assume that a small number of independent runs per algorithm (five in the procedure text, ten in figure captions) is enough to represent each algorithm's typical performance, and the reported score differences are not tested for statistical significance.","fun_headline_variants_meta":{"raw":{"variants":["DYCORS tops synthetic, CUATRO-pls wins PID tuning","Surrogate solver picks: DYCORS for tests, CUATRO-pls for PID","For reactor PID, CUATRO-pls beats DYCORS; synthetic reverses","Surrogate benchmarks: DYCORS wins synthetic, CUATRO-pls PID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001635,"raw_usage":{"total_tokens":6515,"prompt_tokens":977,"completion_tokens":5538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":5451}},"tokens_in":593,"tokens_out":5538,"duration_ms":34019,"temperature":1.0,"reasoning_tokens":5451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:37:31.406394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerunning the benchmarking code supplied by the authors with 50 or more independent restarts per algorithm and computing confidence intervals on the normalized scores $p_a$ would settle whether the reported orderings are stable; if DYCORS versus SRBF, or COBYQA/COBYLA versus CBO, swap places under this wider sampling, the chapter's directional conclusions would not be supported.","supporting_citations":[{"cited_title":"Data-Driven Optimization for Process Systems Engineering Applications","cited_arxiv_id":null,"evidence_quote":"Supplies the original CUATRO algorithm and the general data-driven optimization perspective that the chapter builds on."},{"cited_title":"SNOBFIT – Stable Noisy Optimization by Branch and Fit","cited_arxiv_id":null,"evidence_quote":"Defines SNOBFIT, the branch-and-fit surrogate algorithm used in the benchmarking."},{"cited_title":"Scalable Global Optimization via Local Bayesian Optimization","cited_arxiv_id":null,"evidence_quote":"Introduces TuRBO, the trust-region Bayesian optimization variant included in the high-dimensional PID case study."},{"cited_title":"High-Dimensional Derivative-Free Opti- mization via Trust Region Surrogates in Linear Subspaces","cited_arxiv_id":null,"evidence_quote":"Introduces CUATRO-pls, the dimensionality-reducing extension that won the 32-dimensional controller tuning benchmark."},{"cited_title":"Assessing the Reliability of Different Real-time Optimization Methodologies","cited_arxiv_id":null,"evidence_quote":"Supplies the Williams-Otto process model and real-time optimization benchmark used for the constrained engineering case study."}],"review_version":1}