{"id":"5f0ae92d-a283-4295-bd4d-146ab2d5b974","arxiv_id":"1908.05357","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"For estimating extreme tail probabilities or quantiles with Gaussian process emulators, the discrepancy criterion converges faster than expected improvement, and uniform tail-over-sampling initial designs improve accuracy.","lead":"Computer models that simulate engineering systems are often too expensive to run thousands of times, which makes extreme failure probabilities hard to estimate directly. This paper compares two rules for adding runs one at a time to a statistical surrogate and recommends the discrepancy criterion plus tail-heavy initial sampling for faster convergence.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The EI-vs-discrepancy comparison never reports the EI tuning parameter alpha, so the headline preference could reflect an unrepresentative EI implementation.","rationale":"The reader's conditional verdict identifies the unreported EI tuning parameter alpha as the weakest assumption, and I agree. The paper is otherwise internally consistent: the discrepancy criterion is derived without a tuning constant, the short-column experiments show a large and consistent advantage, the floor-system application demonstrates the methodology, and the diagnostic plots provide a practical stopping aid. The missing alpha is a concrete, fixable gap that directly affects the fairness of the central comparison. If the authors disclose alpha and show the comparison is robust to it, the recommendation stands. If not, the claimed superiority of the discrepancy criterion over EI is not fully supported. Since the reader already made the verdict conditional on these missing details, my stress-test does not move the verdict; it reinforces the need for the requested revision.","tokens_in":18472,"tokens_out":4862,"duration_ms":50849,"concrete_test":"Rerun the Section 5 probability and quantile comparisons for the short column function using the same initial designs, MC sets, candidate sets, and repeat structure, but with a grid of EI values alpha in {1, 2, 3, 5} (and any value the authors actually used). Report the RMSE after 40 runs for each alpha and both criteria. If the discrepancy criterion maintains its large RMSE advantage for every reasonable alpha, the concern is resolved; if some alpha makes EI competitive or better, the paper's recommendation needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central recommendation (Section 8) is to prefer the discrepancy criterion because it converges faster than EI, based on the empirical comparisons in Section 5. The EI criterion defined in Section 4.2 depends on a user-set confidence parameter alpha in the improvement function (12), and equation (13) shows that alpha directly controls the trade-off between local search near the contour and global search driven by predictive variance. Section 5 describes the initial designs, MC set, candidate set, and number of repeats, but never states the value of alpha used in Figures 3-7 or Tables 1-2. The discrepancy criterion in Section 4.3 has no equivalent tuning constant, so the comparison is not symmetric unless the EI implementation uses a representative alpha. If the authors chose an unusually small or large alpha, the observed slow convergence of EI could be an artifact of that choice rather than a property of the criterion. Because the paper ships no code and does not report alpha, the central claim is conditional on an undocumented implementation decision. This is not an internal inconsistency, but it is the weakest point in the evidence for the headline recommendation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes and evaluates sequential experimental design strategies for estimating extreme tail probabilities and quantiles of a computer model output using a Gaussian process (GP) surrogate. It compares two existing search criteria, expected improvement (EI) and the hypothesis-testing-based discrepancy criterion, and uses two applications: a short-column limit-state function and an eight-dimensional floor-system computer model. The paper also addresses practical questions: choice of initial design, modeling of the input distribution, stratified construction of Monte Carlo and candidate sets, and a stopping diagnostic. The main empirical findings are that the discrepancy criterion converges much faster than EI, that uniform initial designs that over-sample input-distribution tails are preferable, and that a stratified weighting scheme based on a mixture of empirical lower-tail and upper-tail distributions works well for the floor system.","tokens_in":18650,"tokens_out":7328,"duration_ms":74285,"significance":"If the empirical conclusions hold, the paper provides a useful practical recommendation: the discrepancy criterion in Eq. (16) has no tuning constant, is simpler to implement than EI, and appears to reach accurate estimates with about 10 added runs where EI has not converged after 20. The study is strengthened by external verification against 10 million Monte Carlo draws for the short column and direct computer-model runs for the floor system, by the use of Bayesian uncertainty quantification for GP parameters, and by clear algorithmic descriptions. The main limitations are that the central EI-versus-discrepancy comparison rests on a single test function with only 10 repeats, and that the EI tuning parameter alpha is not reported, making the headline comparison difficult to reproduce or assess.","major_comments":[{"comment":"The value of the tuning constant alpha used in the EI implementation is never stated anywhere in the paper, even though Eq. (12)-(13) show that alpha directly controls the width of the improvement region and hence the balance between local search near the contour and global search driven by predictive variance. Because the headline recommendation that the discrepancy criterion converges faster than EI is based entirely on the experiments in Section 5, the comparison is not reproducible and could be an artifact of an unrepresentative alpha. The authors should report the alpha used for Figures 3-7 and Tables 1-2, and ideally include a sensitivity analysis over a range of alpha values to show that the conclusion is robust.","section":"Section 4.2, Eq. (12)-(13); Section 5.1"},{"comment":"The central claim that the discrepancy criterion is preferable to EI is supported by only one test function, the short-column function, with 10 repeats; the floor-system application in Section 6 applies only the discrepancy criterion, so it provides no independent replication of the comparison. Section 8 states the recommendation in general terms ('Use the discrepancy criterion'), and although the text adds 'in our study', the evidence base is narrow for a general methodological recommendation. Adding a second test problem that runs both criteria, or substantially tempering the scope of the recommendation, would make the central claim load-bearing rather than illustrative.","section":"Section 5 and Section 8"},{"comment":"The proposed stopping diagnostic plots, for all MC points, the standardized discrepancy -|m_hat(x)-y_f|/sqrt(v_hat(x)), which is exactly the quantity that the discrepancy search criterion optimizes when choosing the next point. A declining trend in these boxplots may therefore be a self-confirming property of the search rule rather than independent evidence that the probability or quantile estimate has converged. The paper should validate the diagnostic against the actual estimation error in a setting with a known truth, or at least explicitly acknowledge this possible circularity.","section":"Section 7"}],"minor_comments":[{"comment":"The RMSE values in Tables 1 and 2 are based on only 10 repeats but are reported without standard errors or any other measure of variability; adding per-repeat errors or bootstrap intervals would help the reader assess whether the large reported differences are statistically stable.","section":"Section 5, Tables 1 and 2"},{"comment":"The sentence 'Further calculations, not reported here for brevity, show that the EI method requires about 30 extra points after the 20-point initial design, to converge' is not verifiable; it should either be removed or supported by a figure or table.","section":"Section 5.1"},{"comment":"The statement that the fitted 2-parameter and 3-parameter Weibull distributions are 'virtually identical' is qualitative; a numerical comparison of fitted parameters or a distance measure would be more informative.","section":"Section 6.2"},{"comment":"The 'true' 0.999 quantile for the floor system is itself an estimate with a reported standard error of 0.00701; treating 3.88 as the exact truth in Figure 10 is acceptable for plotting, but the uncertainty in this reference value should be acknowledged in the text.","section":"Section 6.4"},{"comment":"The definition of the discrepancy criterion states epsilon > 0, but the text then says 'as we work with epsilon = 0'; this should be clarified, for example by noting that the formula is extended to the boundary case epsilon = 0.","section":"Section 4.3"},{"comment":"The captions describe the dotted line as 'the true quantile, 0', but the context is the lower 0.0025 quantile of a limit-state function; the caption could be more explicit about this being the lower quantile.","section":"Figures 6 and 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable applied methodology contribution, and the discrepancy criterion is indeed simpler than EI. My main concern is not the internal logic but the reproducibility and breadth of the central comparison: the EI tuning parameter alpha is missing, and the comparison is limited to one test function. These issues are fixable within the scope of a revision, so I would not reject the paper, but the current version should not be accepted without addressing them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is practical: it takes two existing sequential search criteria, EI and the discrepancy criterion, and asks which one actually works when the target is an extreme tail probability or quantile. The answer it gives—discrepancy converges much faster, roughly 10 added runs versus 20-plus for EI on the short column function—is clearly stated and supported by the experiments shown. The paper also addresses two questions practitioners care about: how to choose initial designs and how to build the MC/candidate sets. The recommendation to over-sample the tails in the initial design is sensible, and the stratified sampling scheme for the floor system is a nice piece of applied thinking. The comparison against external Monte Carlo benchmarks (10 million draws for the short column, direct simulation for the floor) is the right way to evaluate these methods, and the paper does not force the estimates by using its own search criterion as truth.\n\nThe soft spots are real but not fatal. The biggest one is the one your stress-test note flagged: the EI criterion in equation (13) depends on the confidence parameter alpha, and the paper never says what alpha was used in Figures 3–7 or Tables 1–2. That matters because alpha controls the local-global trade-off, and an unrepresentative choice could make EI look worse than it is. This is an omission, not a sign of dishonesty, but it is load-bearing for the central claim. Second, the direct EI-versus-discrepancy comparison rests on a single test function; the floor system application only runs the discrepancy criterion, so it does not corroborate the comparison. Third, the RMSE tables report point estimates without standard errors or intervals, which is thin for 10 repeats. Fourth, no code or data are shipped, so the empirical claims cannot be reproduced as-is. The stopping diagnostic in Section 7 is informal—it is a boxplot of standardized distances, not a rule—but the paper mostly presents it as a diagnostic aid, so I would treat that as minor.\n\nOn balance, the paper is honest, clearly written, and the recommendations are plausible. The citation pattern is fine: the key prior work by Roy and Notz, Ranjan et al., and Jones et al. is acknowledged. The missing alpha and the thin evidence base make the headline claim conditional, but conditional is not invalid. If the authors report alpha, rerun the comparison on at least one more function, and add error bars or code, this could be a solid practical reference for reliability analysis with expensive simulators.\n\nI would send this to peer review. It is a serious empirical study with a clear question, not a desk reject. The referees should push for the missing parameter values and more examples, but the work deserves that effort.","headline":"A useful empirical comparison of two search criteria for extreme tail estimation that deserves refereeing, but the headline EI-vs-discrepancy claim is only as strong as the unreported EI tuning constant and a one-function comparison.","tokens_in":810,"tokens_out":849,"would_cite":true,"duration_ms":20937,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that for estimating extreme tail probabilities and quantiles with a Gaussian process surrogate, the discrepancy criterion converges much faster than expected improvement and should be the preferred sequential search rule.","keywords":["sequential experimental design","Gaussian process surrogate","failure probability estimation","extreme quantile estimation","expected improvement","discrepancy criterion","computer experiments","tail probability"],"falsifier":"Run the short column experiment with the expected-improvement criterion across a grid of alpha values covering the range suggested by Ranjan et al. (2008), report the convergence curves and final RMSE for each alpha, and check whether any alpha lets expected improvement match the discrepancy criterion's convergence within ten added runs; if so, the paper's blanket recommendation would fail.","tokens_in":18246,"feed_emoji":"🎯","tokens_out":3370,"duration_ms":34489,"temperature":0.7,"pith_summary":"The paper establishes practical guidance for estimating a rare failure probability or quantile when the computer model is too expensive for plain Monte Carlo. Its central recommendation is that a Gaussian process surrogate should be trained with an initial design, then extended sequentially using the discrepancy criterion rather than expected improvement. In the short column illustration, the discrepancy criterion reaches accurate probability and quantile estimates after roughly ten added runs, while expected improvement has not converged after twenty and needs about thirty extra points. The paper also recommends a uniform initial design that oversamples distribution tails, stratified sampling for the Monte Carlo set, and Bayesian uncertainty quantification for the surrogate. These recommendations matter because they turn sequential design from a theoretical idea into a workable recipe for engineers estimating tail risk.","feed_headline":"Discrepancy search converges in ~10 added runs where EI needs 30","feed_subtitle":"For extreme tail probabilities from expensive simulations, a parameter-free discrepancy rule beats expected improvement.","key_machinery":"The load-bearing mechanism is the discrepancy criterion, which selects the next evaluation by minimizing $(\\hat m_\\psi(x)-y_f)^2/v_{\\psi,\\sigma^2}(x)$, equivalently maximizing $\\Pr(Z<-|\\hat m_\\psi(x)-y_f|/\\sqrt{v_{\\psi,\\sigma^2}(x)})$ with $Z\\sim N(0,1)$. Unlike the expected improvement criterion, this form requires no tuning constant $\\alpha$ and balances local search near the target contour with global search in regions of high predictive variance. The sequential algorithm wraps this criterion around a Bayesian Gaussian process emulator, whose predictive mean and variance are averaged over Markov chain Monte Carlo samples of the correlation parameters; quantile estimation is handled by treating the current estimated quantile as the target contour at each step.","core_discovery":"On the paper's own terms, the central discovery is that the hypothesis-testing-based discrepancy criterion is the more reliable search rule for sequential estimation of extreme tail probabilities and quantiles. For the short column function, estimates built from the discrepancy criterion effectively converge to the true failure probability of 0.0025 within roughly 11 added runs and stay there across all 10 repeated experiments, whereas expected improvement estimates still wander and require about 30 added runs to converge. The same pattern holds for quantile estimation and for a realistic eight-input floor deflection model. The paper also shows that a uniform initial design, which oversamples the tails of the input distributions, improves accuracy over a random initial design, and that stratifying the Monte Carlo and candidate sets makes the computation more efficient while preserving tail accuracy.","pith_inferences":["Going beyond the paper's comparisons, the reported advantage of the discrepancy criterion could depend on the value of the tuning constant $\\alpha$ used in the expected improvement implementation, and the paper does not report that value; sweeping $\\alpha$ over the range suggested by the contour-estimation literature would test whether the conclusion is robust.","The discrepancy criterion could be compared against other contour-adaptive rules, such as entropy-based or integrated-risk criteria, on higher-dimensional test functions to see whether its fast convergence generalizes beyond the two examples studied here.","The diagnostic boxplots the paper proposes could be turned into an automatic stopping rule: stop adding runs when the quantiles of the standardized discrepancy distribution stop decreasing by more than a small threshold.","A natural extension would be to apply the same sequential recipe to estimate simultaneous tail probabilities for multiple failure modes or to optimize a design parameter while estimating the tail probability, since the discrepancy criterion naturally targets the relevant contour at each step."],"forward_implications":["Practitioners estimating extreme tail probabilities or quantiles from expensive simulations should prefer the discrepancy criterion over expected improvement, since it converged in roughly ten added runs where expected improvement needed about thirty.","A uniform initial design that oversamples the tails of the input distribution is preferable to a purely random initial design, especially for expected improvement, because it places training points closer to the failure region.","Using a stratified Monte Carlo set with weighted stratum estimates can reduce the number of simulation points needed to resolve a tail probability or quantile, as demonstrated in the eight-input floor system example.","Diagnostic plots of the standardized discrepancy across the Monte Carlo set can indicate whether the sequential search has converged, giving practitioners a stopping rule when the true probability or quantile is unknown.","Bayesian estimation of Gaussian process parameters is recommended for these searches because the criteria rely on realistic predictive variances that fully account for parameter uncertainty."],"supporting_citations":[{"why":"Supplies the expected improvement criterion and the sequential global optimization framework that the paper adapts to contour and tail estimation.","marker":"Jones et al. (1998)"},{"why":"Defines the contour-improvement function whose expectation is the expected improvement criterion used here for probability and quantile estimation.","marker":"Ranjan et al. (2008)"},{"why":"Introduces the hypothesis testing-based discrepancy criterion and the sequential-versus-fixed design comparison that this paper extends to extreme probabilities and quantiles.","marker":"Roy and Notz (2014)"},{"why":"Provides the fully Bayesian Gaussian process with MCMC over correlation parameters used to compute the predictive means and variances in both search criteria.","marker":"Chen et al. (2017)"},{"why":"Supplies the short column reliability function with a known true failure probability that anchors the empirical comparison of the two criteria.","marker":"Kuschel and Rackwitz (1997)"},{"why":"Supplies the floor system computer model that motivates the methodology and provides the second test case.","marker":"McCutcheon (1984)"},{"why":"Establishes the Gaussian process model for computer experiments that underlies the surrogate and its predictive uncertainty.","marker":"Sacks et al. (1989)"},{"why":"Provides the Bayesian kriging predictive t-distribution used when deriving predictive mean and variance conditional on correlation parameters.","marker":"Handcock and Stein (1993)"}],"fun_headline_variants":["Discrepancy rule: 11 runs for tail estimate, EI needs 30","Extreme quantiles faster: discrepancy beats EI by 3x","Parameter-free search: 11 runs vs 30 for tail probability","Discrepancy criterion wins for extreme tails in ~11 runs","Fewer runs for extreme probabilities: discrepancy over EI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison between the two search criteria assumes that the expected-improvement implementation used a representative value of its tuning constant alpha; the paper never reports the alpha used in the Section 5 experiments, so if that value was atypical the discrepancy criterion's advantage could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Discrepancy rule: 11 runs for tail estimate, EI needs 30","Extreme quantiles faster: discrepancy beats EI by 3x","Parameter-free search: 11 runs vs 30 for tail probability","Discrepancy criterion wins for extreme tails in ~11 runs","Fewer runs for extreme probabilities: discrepancy over EI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1261,"prompt_tokens":790,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":406,"tokens_out":471,"duration_ms":4776,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:16:31.107805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the short column experiment with the expected-improvement criterion across a grid of alpha values covering the range suggested by Ranjan et al. (2008), report the convergence curves and final RMSE for each alpha, and check whether any alpha lets expected improvement match the discrepancy criterion's convergence within ten added runs; if so, the paper's blanket recommendation would fail.","supporting_citations":[{"cited_title":"Efficient global optimization of expensive black-box functions,","cited_arxiv_id":null,"evidence_quote":"Supplies the expected improvement criterion and the sequential global optimization framework that the paper adapts to contour and tail estimation."},{"cited_title":"Sequential Experimental Design for Contour Estimation from Complex Computer Codes,","cited_arxiv_id":null,"evidence_quote":"Defines the contour-improvement function whose expectation is the expected improvement criterion used here for probability and quantile estimation."},{"cited_title":"Estimating percentiles in computer experiments: A comparison of sequential-adaptive designs and fixed designs,","cited_arxiv_id":null,"evidence_quote":"Introduces the hypothesis testing-based discrepancy criterion and the sequential-versus-fixed design comparison that this paper extends to extreme probabilities and quantiles."},{"cited_title":"Flexible Correlation Structure for Accu- rate Prediction and Uncertainty Quantification in Bayesian Gaussian Process Emulation of a Computer Model,","cited_arxiv_id":null,"evidence_quote":"Provides the fully Bayesian Gaussian process with MCMC over correlation parameters used to compute the predictive means and variances in both search criteria."},{"cited_title":"Two Basic Problems in Reliability-Based Structural Opti- mization,","cited_arxiv_id":null,"evidence_quote":"Supplies the short column reliability function with a known true failure probability that anchors the empirical comparison of the two criteria."},{"cited_title":"Deflections of Uniformly Loaded Floors: A Beam-Spring Analog,","cited_arxiv_id":null,"evidence_quote":"Supplies the floor system computer model that motivates the methodology and provides the second test case."},{"cited_title":"Design and Analysis of Computer Experiments,","cited_arxiv_id":null,"evidence_quote":"Establishes the Gaussian process model for computer experiments that underlies the surrogate and its predictive uncertainty."},{"cited_title":"A Bayesian Analysis of Kriging,","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian kriging predictive t-distribution used when deriving predictive mean and variance conditional on correlation parameters."}],"review_version":1}