{"id":"732ed844-c6dc-480e-893d-b9807de35c3e","arxiv_id":"2504.13441","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Active learning criteria for prediction, EI-MC and EI-SC, carry over to computer experiments with mixed quantitative and qualitative inputs and outperform one-shot designs in the tested examples.","lead":"This chapter reviews and tests strategies for choosing the next runs of an expensive computer simulator when the inputs include both numbers and categories. It finds that adaptive, feedback-driven designs beat one-shot designs for optimization, contour finding, and prediction in the numerical examples tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptive gains rest on an unexamined candidate-set approximation that the paper itself flags as open in Section 5.","rationale":"The reader's weakest assumption is precisely the candidate-set discretization. The paper itself acknowledges this as an open and critical challenge in Section 5, so this is not a hidden flaw but an admitted limitation that nonetheless underlies the central empirical claim. The numerical evidence strongly suggests adaptive designs beat one-shot designs across most settings, and the self-acknowledgment plus the provided GitHub code are real positives. However, because no sensitivity analysis is given, a reader cannot tell whether the magnitudes of the reported gains—some of which are striking—would persist under a denser candidate grid or a true continuous optimization of the acquisition function. This is the single most load-bearing concern: it directly affects the credibility of the 'outperform' assertion. The proposed concrete test would settle it. Other issues (missing error bars, a likely typo in the stated min/max for Examples 2 and 3, and a dangling 'Although' in Section 5) are secondary and do not change the verdict. Since the reader already made the same point and returned CONDITIONAL with moderate confidence, my stress-test does not move the verdict.","tokens_in":21839,"tokens_out":8253,"duration_ms":74055,"concrete_test":"Rerun the Section 4 experiments (at least Example 3 for optimization and contour estimation, and the prediction studies for all three examples) with candidate sets of 250, 500, and 1000 LHS points per qualitative level combination, plus a continuous optimizer (e.g., L-BFGS-B with multiple random starts) for the quantitative variables within each level. Record the averaged best-minimum (Tables 1–3), MC0 (Tables 4, 6, 8), and RMSE (Figures 6–8) for each candidate-set size. If any reported relative-efficiency figure changes by more than ~10%, or if an adaptive method loses its advantage in any setting, the conclusion should be re-scoped as conditional on a sufficiently dense candidate set; otherwise the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that adaptive designs (including EI-MC and EI-SC) outperform one-shot space-filling designs. In every numerical study, the next input is selected by maximizing the acquisition criterion over a finite candidate set: all qualitative level combinations crossed with 100 random Latin hypercube points per level, generated once per simulation (Section 4, first paragraph). The reported superiority therefore assumes this discrete set contains points close enough to the continuous maximizer of the criterion. In Example 3 (p=3 quantitative inputs, 3^3=27 level combinations, 100 points per cube), the candidate grid is sparse, so the true maximizer of EI, LCB, ECL, or EI-MC can easily fall between candidates. If the approximation is inadequate, adaptive designs may select suboptimal points, and the magnitude of the reported advantage—up to 823-fold in Table 4—could change. The paper itself concedes this in Section 5: 'we addressed this limitation by evaluating a large set of candidate points to approximate the optimal solution. However, this approach is computationally intensive and represents an open and critical challenge for further research.' The abstract and concluding remarks nonetheless state the superiority claim without this qualification, and no sensitivity analysis for the candidate-set size is given. This is load-bearing because the chapter's contribution is precisely the empirical demonstration of superiority, and the discretization is an unexamined knob that could alter the comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reviews active learning and adaptive design methods for computer experiments with mixed quantitative and qualitative inputs. It describes the EzGP surrogate model, surveys adaptive design criteria for optimization (EI, LCB, ARSD, Hybrid), contour estimation (RCC, EI-C, ECL, ARSD-C, LCB-C), and prediction (EI-MC, EI-SC), and reports numerical studies on three test functions with 50 simulations each, comparing adaptive designs against one-shot space-filling designs. The central empirical claim is that adaptive designs outperform one-shot designs for optimization, contour estimation, and prediction. Code and examples are made available in a GitHub repository.","tokens_in":22086,"tokens_out":3381,"duration_ms":33280,"significance":"If the empirical claims hold, the paper provides useful practical guidance for practitioners who need to allocate limited simulation budgets in experiments with mixed inputs: the adaptive criteria, used with the EzGP emulator, can yield substantially better optimization, contour, and prediction accuracy per run than one-shot designs. The manuscript is also valuable as a review of recent developments in this area. Strengths include the breadth of methods compared, the use of 50 replications on several test functions, and the public code repository. The reported computational times provide useful practical information. However, the main comparisons currently lack uncertainty quantification, and the acquisition-optimization step is based on a fixed finite candidate set whose adequacy is not examined; these issues affect the strength of the central claim.","major_comments":[{"comment":"The numerical studies select the next input by maximizing each acquisition criterion over a finite candidate set consisting of all qualitative level combinations crossed with 100 random Latin hypercube points per level. The paper itself concedes in Section 5 that this is only an approximation to the true maximizer and that the problem of finding optimal inputs for adaptive designs with mixed inputs is 'an open and critical challenge'. Because the reported superiority of adaptive designs over one-shot designs is entirely based on this approximation, the central empirical claim is not fully supported unless the approximation is shown to be adequate. Please provide a sensitivity analysis with respect to the number of candidate points per level (e.g., 100, 200, 500, 1000) or compare against a continuous optimizer for the quantitative dimensions, and report how the relative efficiencies in Tables 4, 6, and 8 change. Without this, the magnitude of the reported advantages, including the 823-fold improvement in Table 4, is difficult to interpret.","section":"Section 4, first paragraph; Section 5"},{"comment":"All performance summaries are reported as averages over 50 simulations without standard errors, confidence intervals, or any measure of variability. Many of the claimed advantages rely on very small MC0 values, such as the values in Table 4 at N=17 and N=19, where relative efficiencies of 412, 823, or 364 are computed from averages that may be dominated by a few simulations or by numerical noise. Please report standard errors or confidence intervals, and ideally paired comparisons between methods, so that the reader can assess whether the differences are statistically meaningful. This is also needed for the prediction comparisons in Figures 6-8, where the boxplots are not accompanied by numeric summaries.","section":"Tables 1-4, 6, 8; Figures 6-8"},{"comment":"The contour-estimation comparisons depend on tuning parameters epsilon and delta, which are chosen separately for each example, and the text states that epsilon should be 'carefully chosen based on the complexity of the response function'. The RCC method in particular relies on delta, and the recommended value comes from the authors' prior work. No sensitivity analysis is provided for these choices, so it is unclear whether the relative performance of RCC is robust to the 'neither too large nor too small' recommendation in Section 3.2. Please add a small sensitivity study for epsilon and delta, or provide a principled data-driven rule for setting them, so that the comparisons are not tied to example-specific tuning.","section":"Section 4.2, especially Tables 4, 6, 8; Section 3.2"}],"minor_comments":[{"comment":"There is an inconsistency between the budget sizes stated in the text and those given in the figure captions: Example 1 is described with N={15,21} but Figure 6 says N=30; Example 2 is described with N={30,40} but Figure 7 says N=40; Example 3 is described with N={80,100} but Figure 8 says N=100. Please make these consistent and explain which values are being plotted.","section":"Section 4.3, Figures 6-8"},{"comment":"The sentence beginning 'The values of ymin and ymax are unknown contours in general but it can be estimated by a large input from the design domain. They suggest, 1000p design points.' is grammatically awkward and unclear; please rewrite and give the precise reference for the 1000p recommendation.","section":"Section 2.2.3"},{"comment":"The sentence 'It is important to note that the same dataset is utilized for approximating multiple contours in the EI-MC method described in (21)' appears in the contour-estimation section, but EI-MC is described in Section 2.2.3 as a prediction method. This cross-reference is confusing and should be clarified.","section":"Section 4.2"},{"comment":"Several equations and algorithms contain formatting artifacts that make them hard to read, for example 'Dn+1 =Dn S(wn+1,yn+1)' in Algorithm 2 and the union symbol appearing as the letter S. Please check the compiled version of the manuscript.","section":"Throughout"},{"comment":"The abstract says the paper 'propose[s] active learning approaches', but for prediction the manuscript applies existing EI-MC and EI-SC criteria to mixed inputs rather than proposing new criteria. The wording could more accurately distinguish the proposed/reviewed contributions from the transferred criteria.","section":"Section 1, Abstract"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be a handbook chapter or review with substantial numerical comparisons. The main empirical claim is plausible but rests on an unexamined candidate-set approximation that the authors themselves flag as open, and the summaries lack uncertainty quantification. Both issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I did not find evidence of misrepresentation of prior work; the reliance on the authors' own RCC method is accompanied by a review of competing criteria and is not in itself problematic, though sensitivity to its tuning parameter should be shown."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on computer experiments with mixed inputs, but don't expect a breakthrough. The new bit is carrying EI-MC and EI-SC—two prediction-oriented acquisition functions originally built for quantitative-only inputs—over to mixed inputs with an EzGP surrogate. The chapter also pulls together recent optimization and contour-estimation adaptive designs into one place, which is genuinely useful as a survey.\n\nThe numerical work is solid in setup: three test problems, 50 replications, comparisons against one-shot space-filling designs, and code on GitHub. The tables consistently show adaptive designs beating one-shot designs, and the computation-time breakdown is a nice touch. The review of the literature is competent.\n\nThe weaknesses are clear. The comparisons are averages without standard errors or intervals, so you can't tell if a 2x improvement is meaningful. Tuning parameters—delta, epsilon, rho—are set per example, often borrowed from the authors' own earlier papers, with no sensitivity analysis. More importantly, the acquisition criteria are maximized over a finite candidate set: 100 random LHD points per qualitative level combination. The paper itself flags this as 'an open and critical challenge' in Section 5, but the abstract and concluding remarks state the superiority claim without that caveat. In Example 3 (3 quantitative inputs, 27 level combinations), 100 points per cube is pretty sparse, so the true maximizer of EI or ECL can easily fall between candidates. That could change the magnitude of the reported gains (some are enormous, like 823x in Table 4). A sensitivity analysis with different candidate-set sizes would settle this. Also, Example 3's g(z2) has identical formulas for z2=1 and z2=2—likely a typo that should be fixed.\n\nOn the novelty question: the extension is fairly routine because EI-MC and EI-SC only depend on predictive mean and variance, so mixing input types just changes the surrogate. The empirical first application is still useful, but it's not a new idea.\n\nOverall: if you're a practitioner or a researcher building on mixed-input emulation, this chapter gives you a convenient map and a working starting point. It deserves a serious referee—the review parts are strong enough and the empirical claims, while underevidenced, are plausible. I'd send it out with the expectation that the authors add uncertainty quantification, check candidate-set sensitivity, and fix the typo. I wouldn't desk-reject it.","headline":"A useful survey plus a modest, plausible first application of prediction-oriented acquisition to mixed inputs, but the empirical case needs error bars and a sensitivity check on the discrete candidate set.","tokens_in":22625,"tokens_out":3317,"would_cite":false,"duration_ms":29306,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptive designs beat one-shot designs for mixed-input computer experiments.","keywords":["active learning","adaptive design","computer experiments","mixed quantitative and qualitative inputs","Gaussian process emulator","expected improvement","contour estimation","sequential design"],"falsifier":"Rerun the three examples with candidate sets of size 10, 100, and 1000 random Latin hypercube points per qualitative level, and also with a continuous optimizer for the quantitative coordinates; if one-shot designs match or beat the adaptive criteria as the candidate set grows, then the paper's superiority claim is an artifact of the coarse search grid rather than a property of the criteria.","tokens_in":21628,"feed_emoji":"🎯","tokens_out":6869,"duration_ms":58645,"temperature":0.7,"pith_summary":"This paper argues that adaptive, sequential designs outperform one-shot space-filling designs for computer experiments whose inputs include both quantitative and qualitative variables, across the three goals of optimization, contour estimation, and prediction. Using the easy-to-interpret Gaussian process (EzGP) emulator as the surrogate, it compares criteria originally developed for quantitative-only inputs, including expected improvement, lower confidence bound, adaptive-region, and entropy-based criteria, and shows numerically that spending the same total simulation budget adaptively produces better minima, contour locations, and predictions than running a fixed design in advance. The practical payoff is that scientists and engineers running expensive simulators can get more accuracy per simulation run by choosing where to sample next based on the current emulator. The paper also applies the EI-MC and EI-SC criteria, designed for prediction with quantitative inputs, to mixed inputs and reports that they outperform one-shot designs.","feed_headline":"Adaptive designs beat one-shot designs for mixed-input simulators","feed_subtitle":"Expected-improvement criteria on an EzGP surrogate deliver better optimization, contour, and prediction accuracy per run.","key_machinery":"The central object is the EzGP (easy-to-interpret Gaussian process) model, which writes the response as a constant mean plus an overall Gaussian process on the quantitative inputs plus a separate Gaussian process for each level of each qualitative factor, with multiplicative correlation between the qualitative and quantitative parts. This surrogate supplies the predictive mean $\\hat{\\mu}(w)$ and predictive variance $\\hat{\\sigma}^2(w)$ that every acquisition criterion consumes. The carrying mechanisms are the acquisition criteria: expected improvement for optimization and contours, lower and upper confidence bounds, the adaptive-region ARSD rule, the region-based cooperative RCC rule that splits the space into confident and uncertain zones, and the EI-MC and EI-SC criteria for prediction, each of which selects the next simulator run by maximizing a goal-specific score. The numerical comparisons then quantify how many simulation runs each criterion saves relative to a one-shot design.","core_discovery":"The paper's central claim is that active learning is both feasible and beneficial when the input space mixes continuous quantitative variables with discrete qualitative factors, provided the surrogate is an EzGP model that decomposes the response into an overall Gaussian process plus level-specific Gaussian processes for each qualitative factor. For optimization, the EI, LCB, ARSD, and Hybrid criteria all approach the true minimum more closely than one-shot designs in three test functions, with the best method varying by example. For contour estimation, the RCC, EI-C, ECL, ARSD-C, and LCB-C criteria improve contour accuracy, measured by MC0, the mean absolute prediction error on points near the contour, over one-shot designs in almost all settings, with RCC and ARSD-C being substantially faster. For prediction, the EI-MC and EI-SC criteria, originally proposed for quantitative-only inputs, yield lower log RMSE than one-shot designs. The paper's concluding statement is that adaptive designs outperform one-shot designs for prediction in mixed-input experiments, and it leaves the development of dedicated prediction criteria for mixed inputs as an open problem.","pith_inferences":["Editorial inference: because each acquisition step searches a fixed set of 100 random Latin hypercube candidates per qualitative level, the reported gains are demonstrated against that grid, not against a true continuous maximizer; denser or differently generated grids could change the ranking.","Editorial inference: replacing the random candidate grid with a continuous optimizer on the quantitative coordinates inside each qualitative level could reduce the computational cost the paper identifies as an open challenge, and would directly test whether the candidate-grid approximation is the bottleneck.","Editorial inference: the same EI-MC and EI-SC prediction machinery could be pointed at related mixed-input goals such as probability-of-failure estimation or simultaneous estimation of several contours, which the paper does not examine."],"forward_implications":["Practitioners with expensive mixed-input simulators can spend the same budget adaptively and expect better optimization, contour, and prediction accuracy than a one-shot space-filling design.","Quantitative-only criteria such as EI-MC and EI-SC transfer to mixed inputs when paired with an EzGP surrogate, giving lower RMSE than one-shot designs in all three examples.","For contour estimation, RCC and ARSD-C match or beat entropy-based and expected-improvement alternatives while requiring much less computation time.","No single optimization criterion dominates: EI converges fastest in one example, ARSD and LCB are best at the final budget in another, and Hybrid is best in the highest-dimensional example.","The main practical cost shifts to model fitting and criterion evaluation, so faster parameter estimation for mixed-input Gaussian process models would widen the applicability of these adaptive designs."],"supporting_citations":[{"why":"Proposes the EzGP model used as the surrogate in all numerical comparisons.","marker":"Xiao et al. (2021)"},{"why":"Introduces the expected improvement criterion used for optimization and as the basis for contour criteria.","marker":"Jones et al. (1998)"},{"why":"Proposes the EI-MC and EI-SC prediction criteria that the paper transfers to mixed-input experiments.","marker":"Yang et al. (2020)"},{"why":"Proposes the RCC adaptive contour estimation criterion and the MC0 accuracy measure used in the contour studies.","marker":"Shahrokhian et al. (2024)"},{"why":"Provides the ARSD adaptive-region optimization method and the beta tuning constant reused by RCC.","marker":"Cai et al. (2024)"},{"why":"Extends expected improvement to contour estimation, forming the basis of EI-C and the multiple-contour EI-MC improvement.","marker":"Ranjan et al. (2008)"},{"why":"Supplies the entropy-based contour locator (ECL) criterion that is compared in the contour estimation studies.","marker":"Cole et al. (2023)"},{"why":"Supplies the Hybrid Monte Carlo tree search and Gaussian process optimization method compared in the optimization studies.","marker":"Luo et al. (2024)"}],"fun_headline_variants":["Active learning beats static designs for mixed-type simulators","EzGP active learning improves optimization and contours","Adaptive design wins for mixed quantitative-qualitative inputs","Active learning excels with qualitative and quantitative inputs","Mixed-input simulation: adaptive beats one-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage of adaptive designs over one-shot designs assumes that maximizing each acquisition criterion over 100 random Latin hypercube candidates per qualitative level combination finds essentially the same input as maximizing over the whole design space.","fun_headline_variants_meta":{"raw":{"variants":["Active learning beats static designs for mixed-type simulators","EzGP active learning improves optimization and contours","Adaptive design wins for mixed quantitative-qualitative inputs","Active learning excels with qualitative and quantitative inputs","Mixed-input simulation: adaptive beats one-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2076,"prompt_tokens":873,"completion_tokens":1203,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1132}},"tokens_in":489,"tokens_out":1203,"duration_ms":8337,"temperature":1.0,"reasoning_tokens":1132,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:07:48.883324+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the three examples with candidate sets of size 10, 100, and 1000 random Latin hypercube points per qualitative level, and also with a continuous optimizer for the quantitative coordinates; if one-shot designs match or beat the adaptive criteria as the candidate set grows, then the paper's superiority claim is an artifact of the coarse search grid rather than a property of the criteria.","supporting_citations":[{"cited_title":"EzGP : Easy-to-interpret Gaussian process models for computer experiments with both quantitative and qualitative factors","cited_arxiv_id":null,"evidence_quote":"Proposes the EzGP model used as the surrogate in all numerical comparisons."},{"cited_title":"Efficient global optimization of expensive black-box functions","cited_arxiv_id":null,"evidence_quote":"Introduces the expected improvement criterion used for optimization and as the basis for contour criteria."},{"cited_title":"Global fitting of the response surface via estimating multiple contours of a simulator","cited_arxiv_id":null,"evidence_quote":"Proposes the EI-MC and EI-SC prediction criteria that the paper transfers to mixed-input experiments."},{"cited_title":"Adaptive Design for Contour Estimation from Computer Experiments with Quantitative and Qualitative Inputs","cited_arxiv_id":"2504.05498","evidence_quote":"Proposes the RCC adaptive contour estimation criterion and the MC0 accuracy measure used in the contour studies."},{"cited_title":"Adaptive-region sequential design with quantitative and qualitative factors in application to HPC configuration","cited_arxiv_id":null,"evidence_quote":"Provides the ARSD adaptive-region optimization method and the beta tuning constant reused by RCC."},{"cited_title":"Sequential experiment design for contour estimation from complex computer codes","cited_arxiv_id":null,"evidence_quote":"Extends expected improvement to contour estimation, forming the basis of EI-C and the multiple-contour EI-MC improvement."},{"cited_title":"Entropy-based adaptive design for contour finding and estimating reliability","cited_arxiv_id":null,"evidence_quote":"Supplies the entropy-based contour locator (ECL) criterion that is compared in the contour estimation studies."},{"cited_title":"Hybrid parameter search and dynamic model selection for mixed-variable Bayesian optimization","cited_arxiv_id":null,"evidence_quote":"Supplies the Hybrid Monte Carlo tree search and Gaussian process optimization method compared in the optimization studies."}],"review_version":1}