{"id":"7ef050ed-db78-4a7f-ad04-2dea09c5c864","arxiv_id":"2507.21995","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sampling optimal solutions from a Gaussian process posterior and feeding that sample into Sobol sensitivity analysis gives a distribution and importance ranking for optimal decisions.","lead":"This paper proposes a way to measure how uncertain an optimal manufacturing setting is when the objective is known only through a surrogate model. It applies the method to a composite cure process, where time and temperature change points are chosen to minimize deformation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sensitivity indices in §3 are computed with sobolSalt under an independence assumption while the decision-uncertainty sample (7) is dependent; the plotted indices may not equal the defined quantities.","rationale":"The paper's central claim has two components: (i) the decision-uncertainty sample is a valid empirical distribution for the optimal decision, and (ii) Sobol indices computed from that sample quantify how each decision variable contributes to output uncertainty. The reader's weakest assumption targets exactly the transition from (i) to (ii), and my reading confirms that this is the most load-bearing point. The finite-LHS grid approximation is also unquantified, but the independence violation in the sensitivity analysis is more fundamental because it means the reported indices do not necessarily estimate the quantities defined in Eq. (9), regardless of how large N and M are. The paper is otherwise coherent: the GP posterior sampling procedure for the argmin is a standard approach, the code is shared, and the illustrative examples are reproducible in principle. However, the sensitivity claims are the main new practical output, and they rest on an estimator that is not justified for the dependent empirical input distribution. A direct comparison with dependence-aware estimators would settle whether the numerical findings survive. Since the reader already recommended a conditional verdict, no change in verdict is needed, but the condition should explicitly require this validation.","tokens_in":7279,"tokens_out":5422,"duration_ms":72378,"concrete_test":"Recompute the right panels of Figure 2 (and, if feasible, Figure 5) without sobolSalt. Use the same M=500 decision-uncertainty sample (7) and the known test function (8), and estimate the two ratios in (9) directly under the empirical joint distribution: approximate Var(E[y|xi]) and E[Var(y|x_-i)] by kernel or nearest-neighbor conditional-expectation regression on the sample (or by numerical quadrature over the empirical distribution, since y is cheap in (8)). Repeat with an independent uniform sample as a control. If the sobolSalt outputs and the direct dependent-input estimates disagree by more than Monte Carlo error (e.g., >0.05 in either index), the paper's sensitivity interpretation fails; if they agree closely, the independence critique is not empirically damaging.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the claim in §3 that feeding the decision-uncertainty sample (7) into the R package sobolSalt yields Sobol indices quantifying how each optimal-decision coordinate contributes to output uncertainty. Equation (9) defines Sobol indices as variance ratios under a joint input distribution p(x1,...,xd). For dependent inputs these quantities are still well-defined, but the Saltelli-type estimators implemented in sobolSalt are derived under the assumption that inputs are independent, using permuted copies of independent samples. The sample (7) consists of argmins of M independent GP realizations; its coordinates are dependent, and in the cure example the argmins lie near a low-dimensional manifold traced by the optima. Permuting columns of such a sample creates combinations that are not draws from the joint distribution, so the sobolSalt estimates are not consistent for Var(E[y|xi])/Var(y) and E[Var(y|x_-i)]/Var(y) in (9). The sensitivity findings in Figures 2 and 5 are therefore not established. A secondary unquantified bias is the finite-LHS argmin in (7): with N=500 in four dimensions, the grid approximation error in the optimal decision sample is not assessed, and it propagates into both the uncertainty contours and the indices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Gaussian-process-based framework for estimating the uncertainty of the optimal decision in black-box optimization problems. The method fits a GP surrogate to simulation data, draws M independent realizations from the conditional GP, finds the argmin of each realization over a random Latin hypercube grid, and uses the resulting collection of argmins as an empirical distribution of the optimal decision. This distribution is then used as an input distribution in a Sobol sensitivity analysis, implemented with the R package sobolSalt, to rank the contributions of each decision variable to the output uncertainty near the optimum. The approach is illustrated on a two-dimensional toy example and applied to a composite cure process optimization problem with a black-box degree-of-cure constraint.","tokens_in":7496,"tokens_out":5273,"duration_ms":55735,"significance":"The proposed Monte Carlo approach to posterior decision uncertainty is coherent and computationally practical, and the application to a realistic cure process simulation with n=50 runs and a constraint is a useful demonstration. The paper also provides open-source code and data, which supports reproducibility. However, the sensitivity-analysis component, which is a central contribution of the paper, is implemented under an independence assumption that the decision-uncertainty sample does not satisfy; this issue, together with the unquantified finite-grid approximation bias, limits the validity of the reported sensitivity rankings.","major_comments":[{"comment":"The paper states that the decision-uncertainty sample in (7) can be used as the input distribution for Sobol indices computed with the R package sobolSalt. However, sobolSalt implements Saltelli-type estimators that are valid for independent inputs; the sample (7) has dependent coordinates, since each x*_(i),N is the argmin of a single GP realization and the coordinates tend to lie on a low-dimensional manifold (as seen in Figure 4). Permuting columns of this dependent sample, as done by the Saltelli scheme, generates combinations that are not draws from the joint empirical distribution, so the estimates are not consistent for the quantities defined in (9). Consequently, the sensitivity ratios reported in Figures 2 and 5 are not established. The authors should either employ a sensitivity-analysis method that explicitly handles dependent inputs (e.g., Shapley effects or a GP-based index computed under the posterior distribution) or clearly redefine the reported quantities as a heuristic permutation-based measure.","section":"Section 3, Eq. (9), Figures 2 and 5"},{"comment":"The optimal decisions in the sample (7) are obtained by minimizing each GP realization over a per-realization Latin hypercube of size N, with N=1000 in the two-dimensional illustration and N=500 in the four-dimensional application. The empirical distribution of x*_(i),N approximates the distribution of the continuous argmin only up to a discretization bias that is not quantified. With N=500 in four dimensions, the grid is sparse, and the bias propagates into the uncertainty contours (Figures 1 and 4) and into the sensitivity indices (Figures 2 and 5). The paper should provide a convergence check (e.g., recomputing the sample and the indices for a range of N values) or an error bound to demonstrate that the reported results are insensitive to N.","section":"Section 2, Eq. (7) and Appendix"},{"comment":"The Sobol' first-order and total indices in (9) are customarily defined for independent input variables, and their variance-decomposition interpretation depends on that assumption. When p(x1,...,xd) is taken to be the empirical distribution of (7), which has dependent coordinates, the conditional-variance ratios in (9) are still mathematically well-defined but no longer correspond to a unique ANOVA decomposition, and the interpretation that Si and STi measure the contribution of xi's uncertainty to the output uncertainty is not automatically valid. The paper should either derive the interpretation under dependence or restrict the claims accordingly.","section":"Section 3, Eq. (9)"}],"minor_comments":[{"comment":"The phrase 'To the best of out knowledge' should read 'To the best of our knowledge.'","section":"Section 1, paragraph 3"},{"comment":"The sentence 'This figure shows that, the second change point has higher uncertainty than the second change point' contains a typo and should presumably read '...the second change point has higher uncertainty than the first change point.'","section":"Section 4, Figure 4"},{"comment":"The phrase 'Those sensitivity indices are usually computed based on the independent uniform distributions in each dimension of the decision variable' would be clearer as '...based on independent uniform distributions for each decision variable.'","section":"Section 3, paragraph 1"},{"comment":"The notation X(i),N for the set of grid points and x*_(i),N for the sampled optimum are visually similar; consider using different symbols (e.g., G_i for the grid) to avoid confusion.","section":"Section 2, implementation paragraph"},{"comment":"The paper does not state explicitly whether the sensitivity indices in Figure 5 are computed with the true simulation output or with the GP surrogate; please clarify in the caption or text.","section":"Section 4, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is appropriate for Stat.AP and the reproducibility effort (code/data) is commendable. The main methodological idea — sampling posterior argmins to quantify decision uncertainty — is sound and likely useful. My main reservation is the sensitivity-analysis component, which currently relies on an independence assumption that the sample does not satisfy; this is fixable but requires substantial revision. The finite-grid bias also needs attention."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The decision-uncertainty part is a sensible Monte Carlo workflow: fit a GP to simulation data, sample posterior realizations, optimize each realization, and summarize the argmin distribution. That gives practitioners exactly the kind of statement they want about the reliability of recommended settings. The cure-process application is real, the authors ship code and data on GitHub, and the constrained version with two GPs is a reasonable extension. Credit where due: the implementation is careful and the exposition is clear.\n\nThe soft spot is the sensitivity analysis. Section 3 says it computes Sobol indices with expectations taken with respect to the empirical distribution of (7), and then uses sobolSalt. But the decision sample has dependent coordinates — the argmins cluster along a low-dimensional manifold in the cure example. Standard Saltelli estimators are derived for independent inputs; permuting columns of a dependent sample produces points that are not draws from the joint distribution, so the sobolSalt output is not a consistent estimate of the Si and STi defined in (9). That means Figures 2 and 5 are not established. This is a load-bearing issue for the sensitivity claims, though not for the uncertainty-estimation part. A second, smaller issue is the finite-LHS argmin: the grid approximation bias is not quantified and it propagates into both the contours and the indices. The abstract's phrase 'sensitivity with respect to the objective function' also overstates things; what is actually done is sensitivity of output variance to decision-variable uncertainty.\n\nThe paper would benefit from acknowledging these limitations and either changing the sensitivity method (e.g., dependent-input indices or an interpretability measure that accounts for the manifold) or framing the sobolSalt results as heuristic. The central decision-uncertainty idea remains useful for practitioners.\n\nFor peer review, I'd send it out — the application is real and the methodological flaw is fixable. It deserves a serious referee, but the sensitivity claims need major revision.","headline":"A practical recipe for propagating GP posterior uncertainty into optimal decisions, with a sensitivity-analysis extension that rests on an independence assumption the decision sample violates.","tokens_in":8045,"tokens_out":2490,"would_cite":false,"duration_ms":32021,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By minimizing random draws of a fitted Gaussian process, the framework turns a single recommended setting into an empirical distribution of optimal decisions and uses it in Sobol sensitivity analysis to rank parameter contributions.","keywords":["Gaussian process surrogate","decision uncertainty estimation","optimal decision distribution","Sobol sensitivity analysis","black-box constrained optimization","cure process optimization","computer experiments","Latin hypercube sampling"],"falsifier":"Run the procedure on a test function with a known optimum and compare the empirical distribution of minimizers and the resulting Sobol indices at grid sizes 100, 1000, and 10000 with M fixed; if the distribution shifts substantially with the grid size instead of stabilizing, the finite-grid argmin, not the fitted Gaussian process, is producing the reported decision uncertainty.","tokens_in":7041,"feed_emoji":"⚙️","tokens_out":12344,"duration_ms":130275,"temperature":0.7,"pith_summary":"Manufacturers often optimize process parameters using a Gaussian process surrogate trained on a small number of expensive simulation runs. The paper proposes that the uncertainty of the resulting optimal decision can be estimated by drawing many independent realizations of the fitted conditional Gaussian process, minimizing each realization, and collecting the minimizers into an empirical distribution. That distribution is then used as the input distribution for a variance-based sensitivity analysis (Sobol indices), so the reported numbers say how each decision variable contributes to output variability near the optimum rather than across the whole input range. If the approach holds, a practitioner can attach an uncertainty statement to a recommended process setting and a ranking of which parameters deserve follow-up experiments.","feed_headline":"Random GP draws map the uncertainty of the optimal process setting","feed_subtitle":"Minimize each posterior draw, then rank which cure-cycle parameters drive output uncertainty near the optimum.","key_machinery":"The load-bearing mechanism is the conditional Gaussian process (5), with posterior mean $\\hat{y}(x)$ and covariance $\\Sigma(x,x')$, combined with the argmin operation: minimizing each independent posterior draw produces the decision uncertainty sample (7). The second stage uses this sample as the input distribution for the Sobol first-order and total sensitivity indices in (9), approximated with the sobolSalt package. For black-box constrained problems, an independent Gaussian process for the constraint function (14) defines feasibility, and the same argmin step is applied only to realizations that satisfy the constraint.","core_discovery":"Using the fitted conditional Gaussian process (5) with posterior mean $\\hat{y}(x)$ and covariance $\\Sigma(x,x')$, the paper obtains $M$ realizations, evaluates each on a Latin hypercube grid, and takes the minimizer of each realization, producing the decision uncertainty sample $x^*_{(1),N},\\ldots,x^*_{(M),N}$. The central claim is that this sample is a valid empirical distribution for the optimal decision $x^*$, and that plugging it into the Sobol first-order and total indices (9) quantifies how each decision variable contributes to output uncertainty near the optimum. In the two-dimensional illustration, the decision-uncertainty view splits the uncertainty contribution roughly one-third to $x_1$ and two-thirds to $x_2$, a different message from the uniform-input analysis. In the composite cure process, the framework yields densities around the two cure-cycle change points and identifies $T_1$ as the most influential parameter for the first change point and $t_2$ for the second.","pith_inferences":["The authors do not pursue it, but the same decision-uncertainty sample could be summarized as a credible region around the recommended setting, giving a direct uncertainty budget without computing sensitivity indices.","A natural extension is to replace the Sobol formulas with a variance decomposition that allows dependent inputs, since the coordinates of the optimized decision points are not guaranteed to be independent.","The mechanism transfers to any surrogate from which posterior draws can be sampled cheaply after fitting, provided the finite-grid argmin approximation is controlled."],"forward_implications":["A single recommended process setting can be replaced by an empirical distribution of near-optimal settings, so decision makers can see which parameters are tightly pinned down by the data and which are interchangeable.","Sobol indices computed under decision uncertainty rank the parameters by contribution to output uncertainty at the optimum, identifying where extra simulation effort or experimental follow-up is most valuable.","The same machinery extends to black-box constrained optimization: independent Gaussian process realizations of the constraint define feasibility, and only feasible realizations of the objective are minimized.","In the cure process application, the second change point has visibly higher uncertainty than the first, and the sensitivity analysis singles out $T_1$ as a priority for follow-up investigation."],"supporting_citations":[{"why":"Establishes Gaussian process surrogates for computer experiments, the modeling foundation of the proposed framework.","marker":"Sacks et al. (1989)"},{"why":"Supplies the Gaussian process model and the black-box optimization context the decision procedure builds on.","marker":"Jones et al. (1998)"},{"why":"Provides the conditional Gaussian process formulas used to draw realization functions.","marker":"Williams and Rasmussen (2006)"},{"why":"Defines the Sobol first-order and total sensitivity indices used in the analysis.","marker":"Gramacy (2020)"},{"why":"Provides the sobolSalt implementation used to approximate the sensitivity indices.","marker":"Gilquin (2016)"},{"why":"Defines the cure process simulation and the constrained optimization problem reformulated in the application.","marker":"Limaye et al. (2025)"},{"why":"Motivates the Latin hypercube grids used to evaluate Gaussian process realizations and form the decision uncertainty sample.","marker":"McKay (1992)"}],"fun_headline_variants":["GP draws map uncertainty of optimal cure process setting","Which cure parameter drives output uncertainty? GP draws answer","Random GP samples pinpoint sensitive cure decision variables","Optimal decision uncertainty quantified via GP posterior draws","GP-based method ranks cure parameters by uncertainty impact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the finite collection of near-optimal points found by minimizing random model draws can be treated as a genuine sample from the decision distribution and plugged into sensitivity calculations that assume independent inputs; if the grid is too coarse or the decision coordinates are correlated, the uncertainty estimates and parameter rankings may be misleading.","fun_headline_variants_meta":{"raw":{"variants":["GP draws map uncertainty of optimal cure process setting","Which cure parameter drives output uncertainty? GP draws answer","Random GP samples pinpoint sensitive cure decision variables","Optimal decision uncertainty quantified via GP posterior draws","GP-based method ranks cure parameters by uncertainty impact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1116,"prompt_tokens":811,"completion_tokens":305,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":234}},"tokens_in":427,"tokens_out":305,"duration_ms":3928,"temperature":1.0,"reasoning_tokens":234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:08:33.507754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the procedure on a test function with a known optimum and compare the empirical distribution of minimizers and the resulting Sobol indices at grid sizes 100, 1000, and 10000 with M fixed; if the distribution shifts substantially with the grid size instead of stabilizing, the finite-grid argmin, not the fitted Gaussian process, is producing the reported decision uncertainty.","supporting_citations":[{"cited_title":"Statistical science,","cited_arxiv_id":null,"evidence_quote":"Establishes Gaussian process surrogates for computer experiments, the modeling foundation of the proposed framework."},{"cited_title":"Efficient global optimization of expensive black-box functions,","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian process model and the black-box optimization context the decision procedure builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the conditional Gaussian process formulas used to draw realization functions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Sobol first-order and total sensitivity indices used in the analysis."},{"cited_title":"(2016), sobolSalt: Monte Carlo Estimation of Sobol’ Indices Based on Saltelli’s Schemes, r package version 1.30.1","cited_arxiv_id":null,"evidence_quote":"Provides the sobolSalt implementation used to approximate the sensitivity indices."},{"cited_title":"Numerical Simulation Informed Rapid Cure Process Optimization of Composite Structures using Constrained Bayesian Optimization,","cited_arxiv_id":null,"evidence_quote":"Defines the cure process simulation and the constrained optimization problem reformulated in the application."},{"cited_title":"Latin hypercube sampling as a tool in uncertainty analysis of computer models,","cited_arxiv_id":null,"evidence_quote":"Motivates the Latin hypercube grids used to evaluate Gaussian process realizations and form the decision uncertainty sample."}],"review_version":1}