{"id":"51e331af-c5ee-4ec3-94e5-b7a50c7eaa22","arxiv_id":"1908.05562","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A hypothesis test for external pilot trials that uses the power of the planned definitive trial as the test statistic yields controlled progression error rates, unlike conventional independent progression criteria.","lead":"External pilot trials often fail because progression decisions are based on ad hoc criteria. This paper proposes a formal hypothesis test that maps recruitment, follow-up, and adherence estimates to a stop/go decision based on the planned trial's power, with controlled error rates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Advertised type I/II error rates are not established: the suprema over composite hypotheses are assumed on the boundary and found by a heuristic optimizer, so the reported operating characteristics may be optimistic.","rationale":"The paper's central contribution is a principled way to turn pilot estimates into a stop/go decision with controlled error rates. That contribution is valuable, and the derivation of x(φ) and the pilot power h in Eq. (1) are a real step forward. The reader's conditional verdict is appropriate. The weakest point is not the modelling assumptions (constant follow-up, complete-case ITT, normal approximation) — those are stated and are standard. It is the computation of the type I/II error rates themselves. α and β are suprema over continuous composite hypotheses; the paper solves them with NSGA-II and, for the proposed method, never states or proves the boundary-attainment condition that would make the maximization tractable. NSGA-II is a metaheuristic: without a separate verification (grid, multistart, or monotonicity proof), the reported Pareto curves in Figures 2–4 could be optimistic. This matters because the paper's practical advice ('a sample size of around 50 per arm should be sufficient') is derived from those curves. A second, concrete anomaly compounds this: Section 2 defines the pilot recruitment declines as S∼NB(np), while Section 3.3 uses φ̂_r=2np/(2np+S), which corresponds to S∼NB(2np); the supplementary code URL is blank, so the reader cannot check which was implemented. These issues are fixable by a verification step and a code release, not by a change in the conceptual framework, so conditional acceptance is the right call.","tokens_in":16743,"tokens_out":14381,"duration_ms":147636,"concrete_test":"Recompute the operating characteristic for the reported design np=50, c=2.6422 (Section 4.1) using an independent dense grid over (φ_r,φ_f,φ_a)∈[0,1]^3 with 0.005 increments, evaluating h exactly as in Eq. (1) and taking the maximum over points with x(φ)≤x0 and x(φ)≥x1. Compare the resulting α and β with the claimed 0.09 and 0.23. If either exceeds the claimed value by more than rounding tolerance, the paper's error-control claim and the NSGA-II-based curves are not supported. In the same re-run, confirm that the code pairs S∼NB(2np) with φ̂_r=2np/(2np+S).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the pilot's error rates can be computed and controlled. This requires solving α(np,c)=max_{φ∈Φ0} h(np,c,φ) and β(np,c)=max_{φ∈Φ1}[1−h(np,c,φ)] exactly or with a certified global method. The paper does neither. Section 3.4 states that the optimization 'can be solved numerically using the NSGA-II algorithm', and the reported curves in Figures 2–4 are the result of this heuristic search; NSGA-II has no optimality guarantee and could miss the true maxima, making the plotted α and β lower bounds rather than actual error rates. The boundary-attainment assumption that would turn the 3D search into a 2D search is stated only for the conventional PC method ('we assume that the solutions will lie on the boundaries of the hypotheses', Section 3.5), not for the proposed test, and no monotonicity of h in φ is proved. If h is not monotone, or if h has interior maxima on Φ0, the type I error could exceed the advertised value at a parameter point that the optimizer did not visit. This directly threatens the practical recommendation (e.g., np≈50 per arm is sufficient) because it is based on these unverified error rates. Compounding this, the recruitment model is internally inconsistent: Section 2 defines S∼NB(np) while Section 3.3 uses φ̂_r=2np/(2np+S), which requires S∼NB(2np), so the code's actual negative-binomial parameter must be checked before any numerical claim can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a formal hypothesis-testing framework for progression decisions in external pilot trials. Feasibility of the planned definitive trial is quantified by its power, expressed as a function of recruitment, follow-up, and adherence rates. The null and alternative hypotheses correspond to the definitive trial power being below p0 or above p1. The test statistic is the plug-in estimate of the feasibility measure, x(φ̂), obtained from the pilot data. The authors show how the type I and II error rates of this test can be computed by enumerating the pilot sampling distribution and solving a bi-objective optimization for pilot sample size and critical value. The method is illustrated by re-designing TIGA-CUB and compared with conventional independent progression criteria, which are shown to have error rates no better than a coin toss in the considered scenarios. An extension incorporates an unknown outcome standard deviation. The paper includes a derivation of the definitive trial power and a reproducible implementation in R.","tokens_in":17125,"tokens_out":11317,"duration_ms":102379,"significance":"If the error-rate calculations were certified, this would be a valuable contribution to the design of external pilot trials, which currently rely on ad hoc progression criteria with little formal justification. Strengths of the paper include a careful modelling of the pilot data distribution, an explicit link between feasibility and definitive trial power, a complete implementation provided in the supplementary materials, and a concrete demonstration that conventional independent PCs can behave poorly. The main limitation is that the reported worst-case error rates rely on a heuristic optimizer without a proof that the optima are attained, which leaves the central design recommendation conditional.","major_comments":[{"comment":"The advertised type I and II error rates are defined as suprema of h(np,c,φ) over the composite null and alternative hypotheses, but the numerical solution uses NSGA-II, a stochastic metaheuristic with no global optimality guarantee. The paper does not prove that h(np,c,φ) is monotone in φ, nor that the suprema are attained on the boundary surfaces x(φ)=x0 and x(φ)=x1; the boundary-attainment assumption is stated only for the conventional PC method in Section 3.5, not for the proposed test. If the true maxima lie in the interior of Φ0 or Φ1, the curves in Figures 2–4 are lower bounds rather than worst-case error rates, in which case the recommendation that np≈50 per arm is sufficient is not supported. Please either prove monotonicity of h in each component of φ (which would justify restricting the search to the boundary surfaces and allow a grid-based certified search as in Section 3.5) or use a certified global optimization method and report the resulting maxima.","section":"Section 3.4, Eq. (2)"},{"comment":"The sampling model for the pilot is internally inconsistent. Section 2 defines S ∼ NB(np, φr) for the number of eligible patients who decline while recruiting to a target pilot sample size of np, whereas Section 3.3 defines the estimated recruitment rate as φ̂r = 2np/(2np + S) and states that np is the sample size per arm. These two statements are compatible only if S ∼ NB(2np, φr), i.e., if the pilot recruits until 2np consenting participants are obtained. Since the sampling distribution of S enters every probability calculation, the ambiguity must be resolved: please define clearly whether np is per arm or total and align the negative binomial specification in Section 2 with the estimator used in Section 3.3 and in the code.","section":"Section 2 and Section 3.3"}],"minor_comments":[{"comment":"The term 'one-sided type I error rate' is confusing because the definitive trial is described as a two-arm z-test and Section 4 uses a two-sided test at the 0.05 level; since the power formula uses z_{1-α}, the precise meaning of α (one-sided or per-sided) should be stated.","section":"Section 3.1"},{"comment":"The densities pf(.) and pa(.) in the simplified formula for h are not defined explicitly; please state that they are binomial densities for the total number followed up across both arms and the number of adherers, respectively.","section":"Section 3.3"},{"comment":"The data availability statement says the code is 'freely available at' but no URL is printed in the manuscript; please include the repository link.","section":"Data availability statement"},{"comment":"In the formula for h(np,c,φ,σ), the conditioning set of the sample variance density should be defined; as written, p̂σ²(σ̂² | s, a, f, φ) appears to depend on s and a, whereas the preceding text indicates that σ̂²|f follows a scaled chi-square distribution.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well-written and addresses a real gap in pilot-trial methodology. The main obstacle is the reliability of the error-rate optimisation, which is load-bearing for the design recommendations; the revision should resolve the monotonicity/boundary issue and correct the negative-binomial specification. The missing code URL should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper has a good idea—formalize external pilot progression as a hypothesis test on the planned definitive trial's power—and it works through the consequences in a careful, honest way. But the central numerical recommendations depend on error-rate calculations that are not fully established: the maxima over composite hypotheses are found with a heuristic, and the text has an internal inconsistency about the negative binomial recruitment model.\n\nWhat's new: previous warnings about hypothesis tests in pilots were about tests of effectiveness; this paper defines hypotheses on the power scale, which is a real shift. The power derivation for the definitive trial (truncated binomial recruitment, follow-up/adherence with correlation) is careful, and the pilot statistic's sampling distribution is computed by enumeration. The comparison with independent progression criteria is useful—showing that conjunctive PCs can have error rates near coin-toss level is an important practical message.\n\nThe bigger worry is the unstated boundary assumption in Section 3.4. The type I and II errors are defined as suprema over Φ0 and Φ1, but the paper only says (in Section 3.5, for the conventional PC method) that the solutions are assumed to lie on the boundaries. For the proposed test, no monotonicity of h is proved, and the optimization is handed to NSGA-II, which has no global optimality guarantee. So the curves in Figures 2–4 are plausibly lower bounds rather than true error rates. The stress-test note is right on both counts. That said, the issue is fixable: a proof of monotonicity or a certified global optimization would do; a grid search over the boundary (as they already do for the PC method) would at least be transparent.\n\nSecond soft spot: Section 2 defines S~NB(np, φr), but Section 3.3 uses φ̂r = 2np/(2np+S), which only makes sense if the total pilot target is 2np and S~NB(2np, φr). The code may resolve this, but the code statement is itself incomplete ('freely available at' with no URL). This needs clarification before the numbers can be reproduced.\n\nOverall: worth a serious referee. The conceptual contribution is solid, the worked example is instructive, and the critique of independent PCs is valuable. The soft spots are real but addressable, and they don't undercut the main idea. I'd send it to review, but I'd want the authors to fix the optimization justification and the NB inconsistency before publication.","headline":"A genuinely useful new framework for pilot trial progression decisions, but the advertised error rates rest on an unproven boundary assumption and a heuristic optimizer, so treat the numbers with caution.","tokens_in":17574,"tokens_out":3261,"would_cite":true,"duration_ms":28814,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62P10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that an external pilot trial's go/no-go decision be a formal hypothesis test on recruitment, follow-up and adherence, with type I and II error rates controlled by pilot sample size and critical value.","keywords":["external pilot trial","progression criteria","hypothesis test","definitive trial power","recruitment rate","follow-up rate","adherence rate","pilot sample size"],"falsifier":"For the TIGA-CUB setting with $n_p=50$ and $c=2.6422$, evaluate $h(n_p,c,\\varphi)$ at parameter triples strictly inside the null region $\\Phi_0$ rather than only on the boundary $x(\\varphi)=x_0$; if any interior triple gives a go probability larger than the reported type I error, the claimed error control is not guaranteed.","tokens_in":16564,"feed_emoji":"🧪","tokens_out":14309,"duration_ms":126707,"temperature":0.7,"pith_summary":"External pilot trials estimate recruitment, follow-up and adherence rates to decide whether a planned definitive trial is feasible. This paper argues that the go/no-go decision should be a formal hypothesis test rather than a checklist of independent progression criteria. Feasibility is defined by the statistical power the definitive trial would have under the true rates, and the pilot's estimates are condensed into a single test statistic. By choosing the pilot sample size and critical value, the researcher controls explicit type I and type II error rates, something the conventional checklist does not do. In the paper's analyses, conventional progression criteria perform no better than a coin toss, while the formal test gives reasonable error rates at around 50 participants per arm.","feed_headline":"Pilot go/no-go decisions become a formal hypothesis test","feed_subtitle":"Feasibility is the planned trial's power; pilot size and critical value then control both error rates.","key_machinery":"The central object is the function $x(\\varphi)=\\varphi_a\\mu\\sqrt{\\varphi_f E[N\\mid\\varphi_r]}/\\sqrt{4\\sigma^2+2\\mu^2\\varphi_a(1-\\varphi_a)}$, the standardized signal that determines definitive-trial power, where $E[N\\mid\\varphi_r]$ is the expected number of participants recruited under the definitive design. The method uses $x(\\hat\\varphi)$ as the pilot test statistic, so hypotheses are level sets of $x$ and the decision is 'go' when $x(\\hat\\varphi)>c$. The sampling machinery is the pilot power function $h(n_p,c,\\varphi)=\\Pr[x(\\hat\\varphi)>c\\mid n_p,\\varphi]$, built from a negative-binomial distribution for recruitment refusals and binomial or multinomial distributions for follow-up and adherence; error rates are the maxima of $h$ and $1-h$ over null and alternative regions. That reduction turns the choice of pilot sample size and critical value into a multi-objective optimization problem rather than a rule of thumb.","core_discovery":"The paper claims that an external pilot trial's progression decision can be redesigned as a test of feasibility, with hypotheses defined by the power of the planned definitive trial. The definitive trial's power is $g(\\varphi)=\\Phi(x(\\varphi)-z_{1-\\alpha})$, where $\\varphi=(\\varphi_r,\\varphi_f,\\varphi_a)$ are the recruitment, follow-up and adherence rates. Choosing power thresholds $p_0$ and $p_1$ partitions the parameter space into a null region $\\Phi_0=\\{x(\\varphi)\\le x_0\\}$ of infeasible trials and an alternative region $\\Phi_1=\\{x(\\varphi)\\ge x_1\\}$ of feasible trials. The pilot proceeds if and only if $x(\\hat\\varphi)>c$, where $\\hat\\varphi$ is the pilot estimate and $c$ a critical value. The paper then computes the type I and II error rates $\\alpha(n_p,c)=\\max_{\\varphi\\in\\Phi_0}\\Pr[x(\\hat\\varphi)>c\\mid\\varphi,n_p]$ and $\\beta(n_p,c)=\\max_{\\varphi\\in\\Phi_1}\\Pr[x(\\hat\\varphi)\\le c\\mid\\varphi,n_p]$, so $n_p$ and $c$ can be chosen prospectively to balance sampling cost against both errors. Re-designing TIGA-CUB shows the original 30-per-arm pilot has poor operating characteristics, roughly 50 per arm gives type I around 0.09 with type II around 0.23, and independent progression criteria, even with unlimited pilot size, have error rates no better than a coin toss. Extending the test to estimate an unknown outcome standard deviation raises both error rates, so a larger pilot is needed to maintain the same guarantees.","pith_inferences":["The authors' critique of independent progression criteria is a caution about conjunctive decision rules generally: requiring every estimated rate to clear a threshold creates a reverse-multiplicity effect, so a natural design heuristic is to define the decision on the downstream quantity of interest, power, rather than on each process estimate separately.","Because the error rates are explicit, funders and trial oversight committees could specify an acceptable probability of investing in an underpowered trial, and the pilot size would follow from that tolerance; this gives a concrete way to translate risk appetite into a sample size.","Extending the binary stop/go rule to a stop/modify/go decision could be done with two critical values, and the same error-rate calculations would show whether the intermediate decision actually improves long-run decisions or just adds a third action with unmeasured consequences."],"forward_implications":["A pilot team can pre-specify the decision rule as 'go if $x(\\hat\\varphi)>c$' and choose $n_p$ and $c$ from the calculated type I and type II error curves, making sample-size justification part of the same calculation.","In the settings modelled, increasing the pilot from 30 to 50 participants per arm materially improves error rates, and values around 50 per arm keep type II error close to 0.2 while holding type I error near 0.1 when $p_1=0.8$ and $p_0\\le0.65$.","Conventional independent progression criteria, which require all three estimated rates to pass their own thresholds, had error rates no better than a coin toss in the authors' scenarios, and larger pilot samples did not fix this.","If the pilot must also estimate the outcome standard deviation, both error rates rise; maintaining the same error control requires increasing sample size, for example from 50 to 70 per arm in the illustrative setting.","The formulation extends to designs whose power can be written as a function of the pilot-estimated parameters, including binary outcomes by normal approximation and cluster-randomised trials with known variance components."],"supporting_citations":[{"why":"supplies the external-pilot sample-size rules based on standard-deviation estimation that the proposed design replaces and against which its recommended sizes are compared.","marker":"[3]"},{"why":"identifies recruitment, follow-up and adherence as common progression-criteria targets and grounds the claim that their statistical properties are under-researched.","marker":"[6]"},{"why":"documents how imprecise pilot estimates and multiple required criteria distort progression decisions, motivating the formal hypothesis-test formulation.","marker":"[12]"},{"why":"states the usual warning against hypothesis tests in pilots, which the paper narrows to tests of effectiveness rather than feasibility.","marker":"[17]"},{"why":"supplies the multi-objective optimization algorithm used to trace the trade-off frontier between type I and type II error rates.","marker":"[20]"},{"why":"provides the TIGA-CUB pilot trial and its original progression criteria used as the worked re-design.","marker":"[23]"}],"fun_headline_variants":["Pilot trials get a formal go/no-go test","Feasibility tested as a hypothesis in pilots","Power-based decision rule for pilot studies","Hypothesis test sharpens pilot go/no-go","Formal test for pilot trial feasibility"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire design rests on the assumption that the worst combinations of recruitment, follow-up and adherence rates sit exactly on the boundary where the planned trial's power equals the chosen threshold; if they sit inside the region instead, the reported error rates are too low.","fun_headline_variants_meta":{"raw":{"variants":["Pilot trials get a formal go/no-go test","Feasibility tested as a hypothesis in pilots","Power-based decision rule for pilot studies","Hypothesis test sharpens pilot go/no-go","Formal test for pilot trial feasibility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1580,"prompt_tokens":1110,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":402}},"tokens_in":726,"tokens_out":470,"duration_ms":5011,"temperature":1.0,"reasoning_tokens":402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:09:51.671808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For the TIGA-CUB setting with $n_p=50$ and $c=2.6422$, evaluate $h(n_p,c,\\varphi)$ at parameter triples strictly inside the null region $\\Phi_0$ rather than only on the boundary $x(\\varphi)=x_0$; if any interior triple gives a go probability larger than the reported type I error, the claimed error control is not guaranteed.","supporting_citations":[{"cited_title":"Sample size requirements to estimate k ey de- sign parameters from external pilot randomised controlled trials: a simula- tion study","cited_arxiv_id":null,"evidence_quote":"supplies the external-pilot sample-size rules based on standard-deviation estimation that the proposed design replaces and against which its recommended sizes are compared."},{"cited_title":"Lancaster, Susanna Dodd, and Paula R","cited_arxiv_id":null,"evidence_quote":"states the usual warning against hypothesis tests in pilots, which the paper narrows to tests of effectiveness rather than feasibility."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the TIGA-CUB pilot trial and its original progression criteria used as the worked re-design."}],"review_version":1}