{"id":"7bc1536f-7c11-4c68-928a-8587e3298189","arxiv_id":"1908.07769","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A simulation-based sample size determination framework uses Gaussian process regression and efficient global optimization to find Pareto-optimal trial designs with multiple design parameters and constraints.","lead":"Researchers present a general method for calculating clinical trial sample sizes when several design choices must be chosen together and several goals conflict. It combines Monte Carlo simulation with a Gaussian-process surrogate model to search efficiently for good trial designs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own admission that dominated hypervolume is not monotonic reveals a gap in the EI criterion: Eq. (15) ignores downward revision of previously evaluated points, so the sequential search may not maximize the stated objective.","rationale":"The paper makes a useful and credible contribution: it translates expensive simulation-based SSD into a surrogate-modelled multi-objective optimisation problem, implements it with open-source R packages, and demonstrates on three realistic PACE-inspired examples that a set of non-dominated designs can be obtained in hours. The code is promised and the post-hoc verification of the final sets with much larger MC samples gives some empirical support that the returned designs are feasible. The framework is general and the three examples increase in dimensionality in a sensible way. The central claim is plausible but not fully proven, mainly because the search criterion has a known, acknowledged inconsistency that is not analysed.\n\nThe most load-bearing concern is not the Gaussian noise model per se, although that is related. The paper itself reports that H(A_i) is not monotonic because new evaluations revise the estimated feasibility of previously evaluated points. Yet the expected improvement in Eq. (15) is computed as if only the candidate's own feasibility is revised (Eqs. 12-14). In a noisy GP, adding a point changes the fitted hyperparameters and the entire posterior surface, so the feasibility classifications of all evaluated points can shift. The EI therefore overstates the expected gain of an evaluation, and the algorithm may spend budget on points whose improvement evaporates after the GP update. The reader's weakest_assumption focused on the noise model for the MC estimates; this is closely related since the joint revision is a consequence of the noise, but the sharper issue is that the optimisation criterion itself does not account for this revision. Thus my agreement is partial: the reader identified a real weakness, but I would argue the non-monotonic hypervolume is the more direct threat to the 'efficient optimisation' claim.\n\nThe concrete test would settle whether the concern is material. If the cumulative realised improvement tracks the cumulative EI closely, then the omitted joint-revision term is benign in practice despite the theoretical gap. If it does not, the paper's efficiency claim needs qualification or a corrected EI (e.g., a joint posterior sampling step). This does not warrant rejection: the method clearly can produce sensible designs, and the examples are encouraging. But it does warrant the conditional acceptance already recommended, with the requested sensitivity analysis and a comparison against a known Pareto front. I therefore leave the verdict unchanged.","tokens_in":18490,"tokens_out":7664,"duration_ms":79570,"concrete_test":"Use a small SSD problem with two design parameters and a known analytic power function (so the true Pareto front can be obtained by grid search). Run the proposed algorithm with N=100, recording at each iteration the EI value used to select the next point and the realized change in dominated hypervolume H(A_i) after the GP update. If the cumulative realized improvement over 30 iterations is below the cumulative EI by more than, say, 30%, or if the final approximation set's hypervolume is less than 80% of the true Pareto front's hypervolume (while a same-budget Sobol-only design achieves a smaller gap), the concern is confirmed. This directly tests whether the omitted joint-revision term materially misleads the search.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 5.1 the authors note that H(A_i) 'is not strictly increasing. This is because the evaluation of a new solution can lead to revised estimates of other solutions which were in the approximation set, such that they are then considered infeasible and removed from the set.' This observation exposes a structural mismatch in the optimization criterion. The expected improvement in Eq. (15), following [22,24], defines I = H(A*) - H(A) where A is the current approximation set and A* is the set after adding the candidate. The probability term in Eq. (15) accounts only for the posterior revision of the candidate's own feasibility (Eqs. 12-14), not for the joint revision of all previously evaluated points that occurs when the GP is refitted with new data and updated hyperparameters. As a result, EI is an over-optimistic estimate of the true expected improvement: it can select a point whose apparent contribution to hypervolume vanishes, or is negative, once old solutions are re-classified as infeasible. Because the algorithm has no other stopping rule than budget exhaustion and no correction for this bias, the central claim that the method 'efficiently' solves the SSD problem is not underpinned by a consistent monotone search. The paper demonstrates that the final sets from three runs are plausible and post-hoc verifies feasibility with larger N, but this does not establish that the surrogate-guided search is close to the true Pareto front or that the non-monotonicity did not waste a large fraction of the budget. This is load-bearing because the entire contribution rests on the ability of the EI criterion to direct evaluations toward the Pareto set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Gaussian-process-based sequential optimization framework for simulation-based sample size determination (SSD) in clinical trials. The SSD problem is formulated as a constrained multi-objective optimization over design parameters, with constraint functions estimated by Monte Carlo. The authors use a GP surrogate for each constraint, an EGO-style acquisition function based on hypervolume improvement multiplied by the predictive probability of feasibility, and illustrate the method on three PACE-inspired examples of increasing complexity. They also provide R code and simulated data.","tokens_in":18762,"tokens_out":7150,"duration_ms":71162,"significance":"If the central claim holds, the paper offers a practical and general tool for trial design problems where analytic power formulas are unavailable and where several design parameters and objectives must be balanced. The authors demonstrate concrete use cases, make the code available, and validate final designs with larger Monte Carlo samples. The main limitation is that the acquisition criterion does not account for the stochastic re-classification of previously evaluated points, so the theoretical link between the criterion and the stated objective (maximizing hypervolume) is incomplete; in addition, the efficiency claim is supported mainly by the examples rather than by comparison with alternative search strategies.","major_comments":[{"comment":"The expected improvement is defined as [H(A*) − H(A)] times the probability that the candidate is feasible, but the authors acknowledge in §5.1 that evaluating a new point can cause previously evaluated points to be reclassified as infeasible and removed from A. This means H(A*) in Eq. (15) is not the hypervolume that will actually be obtained after updating the GP; the expression overstates the expected improvement because it ignores the joint revision of the approximation set. Since the acquisition function is the core of the algorithm, this is a load-bearing gap. I recommend either (i) explicitly treating EI as a heuristic and providing an empirical justification, for example comparing with random or Sobol-based search with the same budget, or with a benchmark multi-objective optimizer on a small problem, or (ii) deriving an acquisition criterion that accounts for the joint posterior of all evaluated points. At minimum, the paper should report hypervolume trajectories and final sets from several independent runs.","section":"§4.3, Eq. (15); §5.1"},{"comment":"The noise model treats the Monte Carlo estimate as Gaussian with variance ω² = m(1−m)/N, where m is the GP predictive mean and N = 100 in the examples. For binary outcomes and probabilities near the constraint boundary (0.1 or 0.2), the normal approximation and the plug-in variance can be inaccurate; this directly affects the feasibility quantile q(x), the predictive distribution of q+(x), and the EI value. The authors should provide diagnostics, such as empirical coverage of the GP predictive intervals against repeated MC estimates, or move to a more appropriate noise model, such as a logit transform or an explicit binomial likelihood. This is not a purely theoretical concern because Table 3 contains N = 100 estimates that exceed the constraint while the points are nevertheless included in the final set.","section":"§4.3, Eqs. (12)–(14)"}],"minor_comments":[{"comment":"The first sentence contains a typo: 'An in PACE' should read 'As in PACE'.","section":"§2.2"},{"comment":"The line 'ρW = ρT = ρD = ρD = 0.9' contains a duplicated subscript; it should be 'ρW = ρT = ρD = 0.9'.","section":"§5.2"},{"comment":"The confidence level p used in Eq. (12) is not stated for any example, although it determines whether a candidate is considered feasible; please provide these values, since they affect the feasibility rule and thus the final sets.","section":"§4.3, §5"},{"comment":"The comparison labeled 'MLPowSim' is actually a custom Sobol-sequence procedure inspired by MLPowSim; please describe it as such to avoid overclaiming.","section":"§5.1"},{"comment":"Several rows report N = 100 estimates of α exceeding 0.2 (e.g., 0.21, 0.27) while still appearing in the final approximation set; an explanation is needed, e.g., that feasibility is based on the GP posterior rather than the individual MC estimate.","section":"Table 3"},{"comment":"The stopping rule is described only as budget exhaustion or lack of improvement; the examples should state the realized budget in a consistent way, such as total number of MC evaluations, not just wall-clock time.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful combination of existing techniques (GP surrogates, EGO, hypervolume improvement) applied to an important problem in clinical trial design. The main risk is that the acquisition function is not the true expected improvement under the algorithm's updates; this needs to be fixed or explicitly justified as a heuristic with additional empirical evidence. The missing confidence level p and the noise-model details should also be addressed. With those revisions, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nRead this one if you care about sample size determination when power formulas don't exist and the design has several knobs to turn. The paper wraps Gaussian-process surrogates and the EGO algorithm (expected hypervolume improvement with noisy feasibility constraints) around the simulation-based SSD problem, so instead of minimizing a single sample size you can optimize several design parameters against several conflicting objectives and a set of Pareto-optimal designs comes out. None of the algorithmic ingredients are new to the optimization literature, but the assembly for clinical trial design is not something the prior SSD tools (MLPowSim, ipdpower, SWSamp, SimSam) provide, and the paper is honest about that lineage.\n\nWhat's genuinely good: three realistic examples built around the PACE trial (cross-classified therapists/doctors, co-primary endpoints, small-sample pilot with type I error as a design variable). The first example shows the surrogate-guided search beats a fixed Sobol baseline. The final designs are verified with much larger Monte Carlo samples, a legitimate check. Code and data are on GitHub, and the paper openly discusses limitations, including the non-monotonicity of the dominated hypervolume across iterations. Citation pattern looks fine; citing Hooper's SimSam is appropriate.\n\nThe soft spots are worth naming, but I think the stress-test note overstates one of them. The EI criterion in Eq. (15) ignores the fact that new evaluations can re-classify previously evaluated points as infeasible—the paper says exactly this when explaining why H(A_i) is not strictly increasing. That is a real gap between the criterion and the stated objective, and the authors would have done well to flag it as a limitation of the acquisition rule rather than just an observation. But it does not sink the framework: noisy multi-objective search with a finite budget does not promise monotone convergence, the examples show steady improvement, and the verified outputs are plausible. I would call it medium-minor, with a request for a sensitivity analysis or a small problem with a known Pareto front to quantify how much the ignored re-classification costs.\n\nThe bigger practical concern is the N=100 Monte Carlo samples during optimization and the normal noise model backing the probability terms. For probabilities near zero or one the normal approximation is shaky, and the acquisition rule leans on it. A short sensitivity analysis with larger N, or at least reporting N=500 or 1000 for one example, would firm things up.\n\nWho this is for: applied trial statisticians and methodologists facing complex clustering or novel designs, plus anyone building software for simulation-based SSD. It deserves a serious referee; I'd send it out. Recommend revision with those two items—discuss (not necessarily fix) the re-classification gap, and add the sensitivity check.","headline":"A genuinely useful assembly of GP surrogates and EGO for multi-objective simulation-based trial design, with reproducible code and honest discussion; the EI criterion ignores re-classification of old points, but the stress-test overstates the damage.","tokens_in":19358,"tokens_out":4842,"would_cite":true,"duration_ms":45154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62P10","65C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that efficient optimisation algorithms based on surrogate models can solve complex clinical trial sample size problems with several design parameters, constraints, and conflicting objectives, returning a set of…","keywords":["simulation-based sample size determination","clinical trials","Gaussian process regression","efficient global optimisation","multi-objective optimisation","Pareto optimality","expected improvement","operating characteristics"],"falsifier":"Take a design problem with an analytic power formula, run the surrogate search using Monte Carlo estimates, and compare the returned Pareto set with the true Pareto curve; if the approximation set systematically excludes feasible designs or includes infeasible ones after precise re-estimation, the Gaussian process noise model or expected-improvement rule is the cause.","tokens_in":18220,"feed_emoji":"🩺","tokens_out":5935,"duration_ms":59587,"temperature":0.7,"pith_summary":"Simulation is a flexible way to estimate clinical trial power, but until now it has been practical only for problems with a single design parameter. This paper claims that by wrapping the Monte Carlo power calculation in a Gaussian-process surrogate model, sample size determination can be treated as a multi-objective optimisation over several design parameters and constraints at once. The method returns a set of Pareto-optimal designs, so a trial team can see trade-offs between, say, total patients and number of therapists or doctors instead of being handed one number. That matters because complex trials with clustering, co-primary endpoints, and small-sample pilot settings often have no closed-form power formula, and previous approaches either fixed one parameter or fell back on unrealistic assumptions.","feed_headline":"Simulation can size complex clinical trials in hours","feed_subtitle":"A Gaussian-process stand-in for Monte Carlo power lets designers explore patient, therapist, and doctor counts together.","key_machinery":"The machinery is a Gaussian process regression model used as a surrogate for the unknown constraint function, such as the type II error rate, with Monte Carlo noise modelled as normal with variance estimated from the simulation count. Around the surrogate sits an expected-improvement acquisition function: the gain in dominated hypervolume that evaluating a candidate design would bring, multiplied by the probability, read from the GP's predictive quantile, that the candidate will be feasible after evaluation. The quantile feasibility rule lets the user demand a chosen level of confidence that a constraint is satisfied despite Monte Carlo error.","core_discovery":"The central claim is that a surrogate-model optimisation loop can solve complex simulation-based sample size determination problems. Given a program that simulates a proposed trial and returns a binary rejection indicator, the method estimates power or type I error at a few initial designs, fits a Gaussian process to those noisy Monte Carlo estimates, then repeatedly evaluates the design that maximises an expected improvement measure that balances gains in dominated hypervolume against the probability the design will satisfy all operating-characteristic constraints. The paper demonstrates the loop on three increasingly complex problems, covering multilevel clustering with therapists and doctors, two correlated co-primary endpoints, and a small pilot trial with five design parameters including the nominal type I error rate, and in each case obtains an approximation set of nondominated designs whose constraints are confirmed by higher-precision simulation.","pith_inferences":["The same surrogate loop could be aimed at other expensive trial-design objectives, such as recruitment duration, expected cost, or expected sample size under adaptive stopping rules, since the simulation program and objective definitions are the only problem-specific pieces.","The algorithm's explicit noise model suggests a natural extension: allocate Monte Carlo effort adaptively, running more simulations where the surrogate is uncertain instead of using a fixed number per design.","The feasibility confidence parameter is effectively a dial for conservatism, and a systematic study of how it interacts with the number of simulations per design would give trial designers practical guidance on setting both."],"forward_implications":["Trial designers can ask for the full trade-off curve between conflicting objectives, such as number of participants versus number of care providers, rather than minimising one quantity with the others fixed.","Complex designs that previously had to be simplified to make analytic power formulas tractable can be simulated as actually planned, including cases where the analysis model sometimes fails to converge.","If a target power or type I error rate is revised after seeing preliminary results, the search can continue from existing Monte Carlo estimates instead of restarting from scratch.","The same recipe applies to almost any problem for which power can be estimated by simulation, including novel trial designs for which no dedicated sample-size software exists."],"supporting_citations":[{"why":"Supplies the Efficient Global Optimisation algorithm that forms the core sequential search loop.","marker":"[33]"},{"why":"Provides the noisy kriging optimisation framework and the predictive distribution of the revised feasibility quantile.","marker":"[24]"},{"why":"Gives the Gaussian process conditional predictive distribution and marginal likelihood used to fit the surrogate.","marker":"[19]"},{"why":"Provides the hypervolume-based expected improvement measure for multi-objective problems.","marker":"[23]"},{"why":"Supplies the constrained expected improvement penalty term used to discount designs unlikely to satisfy constraints.","marker":"[22]"},{"why":"Defines the simulation-based power calculation steps and the Monte Carlo error model the method relies on.","marker":"[2]"},{"why":"Provides the Gaussian process modelling software that makes surrogate fitting computationally tractable in the implementation.","marker":"[20]"}],"fun_headline_variants":["Surrogate model speeds complex trial sizing","Multi-parameter trial power via smart simulation","Fast simulation-based sizing for complex trials","Gaussian-process surrogate for trial power design","Efficient trial sizing with multiple design parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that each Monte Carlo estimate of a constraint function can be treated as the true value plus a normal error of known variance; with only one hundred simulations per design, that approximation can be unreliable, especially when power is close to 0 or 1, and a bad noise model can make the search certify designs that are actually underpowered.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate model speeds complex trial sizing","Multi-parameter trial power via smart simulation","Fast simulation-based sizing for complex trials","Gaussian-process surrogate for trial power design","Efficient trial sizing with multiple design parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1396,"prompt_tokens":860,"completion_tokens":536,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":472}},"tokens_in":476,"tokens_out":536,"duration_ms":514411,"temperature":1.0,"reasoning_tokens":472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:57:24.312044+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a design problem with an analytic power formula, run the surrogate search using Monte Carlo estimates, and compare the returned Pareto set with the true Pareto curve; if the approximation set systematically excludes feasible designs or includes infeasible ones after precise re-estimation, the Gaussian process noise model or expected-improvement rule is the cause.","supporting_citations":[{"cited_title":"Jones, Matthias Schonlau, and William J","cited_arxiv_id":null,"evidence_quote":"Supplies the Efficient Global Optimisation algorithm that forms the core sequential search loop."},{"cited_title":"Noisy kriging-based optim ization methods: A unified implementation within the DiceOptim package","cited_arxiv_id":null,"evidence_quote":"Provides the noisy kriging optimisation framework and the predictive distribution of the revised feasibility quantile."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Gaussian process conditional predictive distribution and marginal likelihood used to fit the surrogate."},{"cited_title":"Hypervolume-based expected improvement: Monotonicity proper ties and exact computation","cited_arxiv_id":null,"evidence_quote":"Provides the hypervolume-based expected improvement measure for multi-objective problems."},{"cited_title":"Sasena, Panos Papalambros, and Pierre Goovaerts","cited_arxiv_id":null,"evidence_quote":"Supplies the constrained expected improvement penalty term used to discount designs unlikely to satisfy constraints."},{"cited_title":"Sample size and power calculations for medical studies by simulation when closed form expressions are not a vail- able","cited_arxiv_id":null,"evidence_quote":"Defines the simulation-based power calculation steps and the Monte Carlo error model the method relies on."},{"cited_title":"DiceKriging , DiceOptim: Two R packages for the analysis of computer experiment s by kriging-based metamodeling and optimization","cited_arxiv_id":null,"evidence_quote":"Provides the Gaussian process modelling software that makes surrogate fitting computationally tractable in the implementation."}],"review_version":1}