{"id":"e61f5a5d-1192-427c-9dad-dfff42349adf","arxiv_id":"1908.05955","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A Bayesian decision-theoretic framework for designing external pilot trials, using a piecewise-constant loss function to make red/amber/green progression decisions and to assess their error rates via simulation.","lead":"External pilot trials help decide whether a larger clinical trial is feasible, but their go/no-go rules are rarely designed with formal statistics. This paper proposes a Bayesian framework using a loss function to make progression decisions and to evaluate how often those decisions would be wrong.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equations (3)–(5) swap c2 and c3 in the expected-loss formulas, and Eq. (10) reverses the decision indicator; these internal inconsistencies undermine the printed derivation of the decision rule and require correction before the method is implementable.","rationale":"The reader's CONDITIONAL verdict is appropriate. The most load-bearing issue is that the paper's central decision-theoretic derivation is internally inconsistent: the expected-loss expressions (3)–(5) swap c2 and c3 relative to Table 1, and Eq. (10) reverses the inequality for the progression decision. These errors directly affect the argmin in Eq. (1), so a reader implementing from the printed equations would obtain incorrect decisions whenever c2≠c3, which is the intended use case. The errors are likely typographical—the TIGA-CUB example (Section 3) uses the two-decision special case with only c1 and c2, so it is unaffected, and the REACH example (Section 4) likely uses correct code—but the manuscript as written is not self-consistent. This is a correctness risk, not a rejection of the framework.\n\nThe loss-function representativeness issue, which the reader highlighted as weakest_assumption, is real but explicitly acknowledged in the Discussion ('the piecewise constant loss function we have proposed may not adequately represent the preferences of the decision maker'). Moreover, Section 2.4 offers an alternative calibration route: choose c to achieve desirable operating characteristics, which reframes c as tuning parameters rather than an exact preference model. This softens the concern considerably, though it does not eliminate the need for hypotheses and priors to be specified carefully.\n\nIndependent support: the paper provides reproducible code on GitHub, a nested Monte Carlo algorithm that is clearly described, and two worked examples with plausible results. The equations are fixable with an erratum. Therefore our test focuses on verifying that the printed equations do not change the reported OCs; if they do, the paper needs a correction before acceptance.","tokens_in":17272,"tokens_out":13424,"duration_ms":113395,"concrete_test":"Implement the decision rule exactly as printed in Eqs. (3)–(5) for the REACH example with c=(0.07,0.9,0.03), using posterior probabilities from the authors' GitHub code or from a simple simulation; then recompute the operating characteristics in Table 3 using the corrected formulas. If any of OC1, OC2, OC3 in Table 3 differs by more than 0.01 (the reported standard errors are 0.001–0.005), the results as reported do not follow from the printed equations. As a second check, rerun the TIGA-CUB OC1 calculation with Eq. (10) replacing I(p_G<c1) with I(p_G>c1); the value of OC1 should increase, and if it does not, the reversed indicator is not a typographical artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is a decision rule that minimizes posterior expected loss. Table 1 correctly specifies losses for each action under each hypothesis. However, the expected-loss equations derived from it are wrong: Eq. (3) gives E[L(r)]=(p_A+p_G)c3 but should be (p_A+p_G)c2; Eq. (4) gives p_R c1+p_R c2+p_G c2 but should be p_R(c1+c3)+p_G c3; Eq. (5) gives p_R c1+p_A c1+p_A c3 but should be p_R c1+p_A(c1+c2). The pattern is a systematic swap of c2 and c3. When c2 differs from c3 (e.g., REACH point a: c=(0.07,0.9,0.03)), the argmin changes for posterior probability vectors where p_R or p_G is moderate, so a reader implementing the printed equations will reach different progression decisions than intended. Equation (10) defines OC1 with I(p_G<c1), whereas the text states decision g is optimal when p_G>c1; the indicator is reversed. These are not cosmetic: they sit in the core derivation and in the primary operating-characteristic formula. The paper's own Discussion notes the piecewise-constant loss may not capture preferences, which is a separate acknowledged limitation; but the equation errors are unacknowledged and would lead any careful implementer astray. Because the method is otherwise coherent and the examples appear to use corrected logic, the right response is to require a correction/erratum, not to reject the framework.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript develops a Bayesian decision-theoretic framework for designing and analysing external pilot trials of complex interventions. The pilot data update an analysis prior; a red/amber/green progression decision is then chosen by minimising posterior expected loss under a piecewise-constant, additive loss parameterised by three costs c1,c2,c3. At the design stage, operating characteristics are defined as unconditional error probabilities with respect to a design prior and estimated by nested Monte Carlo simulation; the loss parameters can be varied to approximate a Pareto frontier of operating characteristics, and this is used to support sample-size choice. The method is illustrated on the TIGA-CUB and REACH pilot trials, and the authors provide code and simulated data on GitHub.","tokens_in":17577,"tokens_out":11495,"duration_ms":103347,"significance":"If the technical errors below are corrected, the paper makes a useful contribution to pilot-trial methodology. It addresses a genuine and under-studied problem: progression criteria for external pilots are routinely pre-specified but their operating characteristics are rarely evaluated. The proposed framework is coherent, handles multiple endpoints, small samples, multi-level models, and nuisance parameters within a single Bayesian workflow, and is demonstrated on two real trials. The use of a subjective design prior for assurance-type operating characteristics is standard Bayesian design rather than circularity, and the authors are appropriately explicit about the practical burden of prior and hypothesis elicitation. The GitHub repository containing code and simulated data strengthens reproducibility. The main risks are internal inconsistencies in the printed equations and notation rather than the conceptual framework; these need to be fixed before the method can be implemented from the paper as written.","major_comments":[{"comment":"The expected-loss equations are inconsistent with Table 1. From Table 1, the correct expressions are E[L(r)] = (p_A + p_G)c2, E[L(a)] = p_R(c1+c3) + p_G c3, and E[L(g)] = p_R c1 + p_A(c1+c2). The printed equations interchange c2 and c3 in every term. Because c2 and c3 differ in the REACH illustration (for point a, c2=0.9 and c3=0.03), a reader implementing the printed equations can obtain a different argmin and hence a different progression decision. These equations are the core decision rule and must be corrected; the fact that the special two-decision case in Section 3 is derived correctly suggests the numerical work may have used the right formulas, but the printed derivation is wrong.","section":"Section 2.2, Eqs. (3)-(5)"},{"comment":"The indicator in Eq. (10) is reversed. The text states that decision g is optimal whenever p_G > c1, so the probability of proceeding to an infeasible trial should be computed using I(p_G > c1 | x_f, x_a, n), not I(p_G < c1 | x_f, x_a, n). As printed, Eq. (10) counts the opposite decision and would give incorrect operating characteristics for OC1. This is a load-bearing formula for the TIGA-CUB example and needs correction.","section":"Section 3, Eq. (10)"},{"comment":"The meaning of the sample size n is inconsistent. The likelihood is written with a per-arm sample size n, giving 2n follow-up observations and n adherence observations, yet Eq. (8) gives posterior Beta parameters (1+x_f, 1+n-x_f) and (1+x_a, 1+n/2-x_a), which correspond instead to a total sample size n with n/2 adherence observations. The paragraph also says 'given a total sample size n', while later text says 'n = 30 per arm'. Depending on the intended convention, the posterior probabilities and the operating characteristics in Figures 1-2 change. The paper should adopt one convention consistently and ensure Eq. (8), Eq. (10), and the simulation code all match it.","section":"Section 3, Eq. (8) and surrounding text"}],"minor_comments":[{"comment":"The sentence reporting that the INA analysis prior leads to 'larger probabilities of an infeasible trial (OC1) and of unnecessary adjustment (OC2), while reducing the probability of discarding a promising intervention (OC3)' swaps the definitions of OC2 and OC3 from Section 2.3. It should say 'unnecessary adjustment (OC3)' and 'discarding a promising intervention (OC2)'.","section":"Section 4.2.2"},{"comment":"The caption states that the operating characteristics are evaluated at (c1,c2,c3) = (0.069, 0.116, 0.815), whereas the text says the parameters are set to point a of Table 3, (0.07, 0.9, 0.03). These are different cost vectors; the caption or the text must be corrected.","section":"Figure 6 caption"},{"comment":"In the dominance definition, the phrase 'If there exist c, c′ ∈ C*' should refer to two vectors in the sampled set C, not in C*, since C* is the set of non-dominated parameters being constructed.","section":"Section 2.4"},{"comment":"There is a typo in the sentence 'the our preferences for any one of the attributes E1,E2,E3 are independent...'; 'the our' should be 'our'.","section":"Section 2.2"},{"comment":"The Discussion correctly acknowledges that the piecewise-constant loss function may not adequately represent the decision maker's preferences. This is an important scope condition and should be stated at the point the loss function is introduced in Section 2.2, not only in the discussion.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The conceptual framework is sound and the examples appear to have been analysed with the correct logic, but the printed derivation contains several internal inconsistencies (Eqs. (3)-(5), Eq. (10), and the TIGA-CUB sample-size notation) that are load-bearing for implementation and for reproducing the reported operating characteristics. I recommend requiring a careful revision that re-derives all decision and operating-characteristic formulas from Table 1, fixes the indicator in Eq. (10), and resolves the n convention in Section 3. There is no need to reject the framework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a genuinely useful idea — Bayesian decision-theoretic progression rules for external pilot trials — but the printed equations in the core derivation have real typos. The framework is salvageable and the examples are informative, but don't implement from the paper as it stands.\n\nWhat's new: the authors adapt Bayesian decision theory to the traffic-light (red/amber/green) progression decisions that pilot trials actually use, with a piecewise-constant loss over the parameter space. That lets you trade off multiple endpoints (recruitment, adherence, follow-up) rather than applying independent thresholds, and you can choose the loss parameters by looking at operating characteristics rather than by pure elicitation. This goes beyond standard assurance calculations and is a sensible adaptation of admissible-design ideas from phase II. The two worked examples — TIGA-CUB and REACH — demonstrate the method on realistic, messy problems, and the code is on GitHub. The authors also flag the main conceptual limitation themselves: the additive piecewise-constant loss may not capture actual preferences.\n\nSoft spots: Section 2.2 has a systematic c2/c3 swap in equations (3)–(5). Table 1 is correct, but the expected loss formulas printed don't follow from it. Equation (10) has the indicator reversed: they say decision g is optimal when pG > c1, then write I(pG < c1). These are mechanical errors, not a flaw in the framework — the REACH example uses the correct logic — but they sit at the center of the method and would mislead a careful implementer. There are also two smaller label errors: the Figure 6 caption gives a cost vector (0.069, 0.116, 0.815) that doesn't match the point-a values (0.07, 0.9, 0.03) in the text and Table 3, and Section 4.2.2 swaps OC2 and OC3 when describing the INA results. The nested Monte Carlo computation is heavy; the paper acknowledges that and points to possible speedups, but it's a real barrier to uptake.\n\nThe circularity concern doesn't land. Using a design prior to compute operating characteristics is standard Bayesian design, and the cost parameters are used to explore a Pareto front, not fitted to force a conclusion.\n\nVerdict: the central claim holds up after corrections. This deserves peer review — I think a good referee would send it back for revision rather than reject it. It would be a solid contribution to the pilot-trial design literature once the equations are fixed. I'd bring it to a reading group mainly to talk about whether the loss function is flexible enough; the typos are annoying but not the interesting part.","headline":"A genuinely useful Bayesian framework for pilot-trial progression decisions, but the printed core equations have fixable typos and the paper needs a revision before anyone should implement from it.","tokens_in":18171,"tokens_out":4183,"would_cite":true,"duration_ms":37081,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62C10","62F15","62P10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that external pilot trial progression decisions can be made by minimising a three-parameter piecewise-constant loss function, with operating characteristics evaluable at the design stage.","keywords":["Bayesian decision theory","pilot trials","complex interventions","progression criteria","loss function","operating characteristics","assurance","sample size determination"],"falsifier":"For a simple conjugate model with two independent binomial outcomes, compute the paper's operating characteristics both by nested Monte Carlo and by exact enumeration of all possible pilot data sets; disagreement beyond Monte Carlo error would indicate a flaw in the algorithm. To test the preference assumption, ask a decision maker the two indifference gambles the paper describes and a third logically equivalent gamble; if the implied cost parameters differ, the additive three-parameter loss does not capture the stated preferences.","tokens_in":17015,"feed_emoji":"🎯","tokens_out":10290,"duration_ms":90865,"temperature":0.7,"pith_summary":"External pilot trials of complex interventions — treatments made of several interacting components — collect small samples to inform whether a confirmatory trial should go ahead, yet the thresholds used to decide this are usually chosen without assessing their statistical properties. This paper argues that the progression decision should be treated as a Bayesian decision problem: after the pilot, choose the action that minimises expected loss, where loss is defined over the whole parameter space so that trade-offs among recruitment, retention, adherence and potential efficacy can be made explicit. To keep this feasible, the loss is piecewise constant and summarised by three cost parameters, and a design-stage simulation uses a design prior to compute operating characteristics, so error probabilities and sample-size implications can be examined before data are collected. The method is demonstrated by revisiting two external pilot trials, including a cluster-randomised pilot of an intervention to increase physical activity in care home residents, and by showing how the loss parameters can be chosen to obtain a desired balance of errors.","feed_headline":"Minimise expected loss to set pilot trial progression rules","feed_subtitle":"A three-cost decision rule lets trialists pre-test go, amend and stop errors before the pilot begins.","key_machinery":"The load-bearing object is an additive, piecewise-constant loss function over the substantive parameter space, together with a partition of that space into three hypotheses $\\Phi_R$, $\\Phi_A$, $\\Phi_G$ corresponding to the ideal decisions. Its additivity makes the expected loss of each decision a linear function of the posterior probabilities $p_R,p_A,p_G$, so the minimising decision follows directly from an MCMC posterior sample; its piecewise constancy is what allows the whole preference structure to be collapsed into the three costs $c_1,c_2,c_3$. Around this sits a nested Monte Carlo scheme that draws parameters from a design prior, simulates pilot data, and repeats the Bayesian analysis, producing unconditional error probabilities that can be used to choose the sample size and to select loss parameters by multi-objective optimisation.","core_discovery":"The central claim is that progression decisions in external pilot trials can and should be made by minimising expected loss rather than by comparing point estimates to arbitrary thresholds. The substantive parameter space is partitioned into regions corresponding to the red, amber and green decisions; three error types are defined — proceeding to an infeasible main trial, discarding a promising intervention, and making unnecessary adjustments — and a loss function $L(d,\\phi)=c_1E_1+c_2E_2+c_3E_3$ assigns costs to each. Because the loss is piecewise constant, the expected loss of each action depends only on the posterior probabilities of the three hypothesis regions, which can be computed by MCMC even for complex multilevel models. At the design stage, sampling from a design prior and repeating the analysis yields unconditional probabilities of each error type, and a search over the cost parameters reveals the achievable trade-offs. In the worked examples, the resulting pilot designs have error probabilities near conventional levels, and one cost parameter — the cost of discarding a promising intervention — dominates the operating characteristics.","pith_inferences":["The same decision machinery could be applied to internal pilot or seamless phase II/III designs, with the amber action modelled concretely as an adaptive change to the main trial rather than left as an unmodelled option.","Because the three-cost loss assumes the cost of discarding an intervention does not depend on how effective it is, a natural test is to elicit indifference gambles at different points within a hypothesis and check whether the implied costs remain constant.","If the three costs were derived from the health-economic consequences of each error rather than from expert judgement, the operating-characteristic frontier could be converted into expected net benefit and compared directly with value-of-information analyses.","The finding that one cost parameter dominates the operating characteristics suggests a diagnostic: designs in which a single error dominates may need different hypotheses, larger samples, or different endpoints before costs are worth eliciting."],"forward_implications":["Progression criteria become decision rules with known error rates, so pilot sample size can be chosen to control the probability of a wrong go/no-go decision.","Trade-offs between feasibility endpoints — such as accepting lower adherence when potential efficacy is higher — can be articulated through the hypothesis partition instead of being ignored by independent thresholds.","Complex multilevel models with small samples can drive the decision, because only posterior probabilities of hypothesis regions are needed and these come from MCMC rather than closed-form criteria.","A multi-objective search over the three costs lets sponsors view the frontier of achievable error probabilities and select a design matching their preferences rather than a conventional default.","Using an informative prior for a parameter with very little pilot data lowers expected loss while shifting the error balance, making prior choice a substantive design decision."],"supporting_citations":[{"why":"Defines complex interventions and establishes the rationale for conducting pilot trials before a confirmatory trial.","marker":"[1]"},{"why":"Distinguishes external pilot and feasibility trials from the main confirmatory trial, fixing the object of study.","marker":"[2]"},{"why":"Supplies the trial-reporting extension that requires pre-specified progression criteria, motivating the red/amber/green decision framework.","marker":"[5]"},{"why":"Identifies the statistical challenges in pilot trials of complex interventions that the proposed method is designed to address.","marker":"[16]"},{"why":"Introduces assurance, the unconditional error probability used as the primary operating characteristic.","marker":"[17]"},{"why":"Provides the simulation-based Bayesian design framework and the distinction between design prior and analysis prior.","marker":"[19]"},{"why":"Supplies Bayesian hierarchical models for cluster-randomised trials, used in the care-home pilot example.","marker":"[22]"},{"why":"Provides the multi-attribute loss elicitation procedure that yields the three cost parameters.","marker":"[25]"},{"why":"Describes the motivating cluster-randomised pilot trial whose design is revisited in the example.","marker":"[27]"}],"fun_headline_variants":["Bayesian loss minimisation sets pilot trial go/no-go rules","Pilot trial go/no-go via Bayesian expected-loss rules","Expected-loss Bayes sets pilot trial thresholds","Bayesian loss choice decides pilot trial progression","Pilot trials set go/no-go by minimising expected loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The decision maker's preferences can be summarised by three fixed costs, one for each type of wrong decision, with no additional dependence on how far wrong the decision is or on which other wrong decisions occur at the same time.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian loss minimisation sets pilot trial go/no-go rules","Pilot trial go/no-go via Bayesian expected-loss rules","Expected-loss Bayes sets pilot trial thresholds","Bayesian loss choice decides pilot trial progression","Pilot trials set go/no-go by minimising expected loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3058,"prompt_tokens":982,"completion_tokens":2076,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":2000}},"tokens_in":598,"tokens_out":2076,"duration_ms":14386,"temperature":1.0,"reasoning_tokens":2000,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:59:33.266909+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a simple conjugate model with two independent binomial outcomes, compute the paper's operating characteristics both by nested Monte Carlo and by exact enumeration of all possible pilot data sets; disagreement beyond Monte Carlo error would indicate a flaw in the algorithm. To test the preference assumption, ask a decision maker the two indifference gambles the paper describes and a third logically equivalent gamble; if the implied cost parameters differ, the additive three-parameter loss does not capture the stated preferences.","supporting_citations":[{"cited_title":"Developing and evaluating complex intervention s: the new medical research council guidance","cited_arxiv_id":null,"evidence_quote":"Defines complex interventions and establishes the rationale for conducting pilot trials before a confirmatory trial."},{"cited_title":"Eldridge, Gillian A","cited_arxiv_id":null,"evidence_quote":"Distinguishes external pilot and feasibility trials from the main confirmatory trial, fixing the object of study."},{"cited_title":"CONSORT 2010 statement: extension to randomised pilot and feasibility trials","cited_arxiv_id":null,"evidence_quote":"Supplies the trial-reporting extension that requires pre-specified progression criteria, motivating the red/amber/green decision framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the simulation-based Bayesian design framework and the distinction between design prior and analysis prior."},{"cited_title":"Spiegelhalter","cited_arxiv_id":null,"evidence_quote":"Supplies Bayesian hierarchical models for cluster-randomised trials, used in the care-home pilot example."},{"cited_title":"Statistical Decision Theory","cited_arxiv_id":null,"evidence_quote":"Provides the multi-attribute loss elicitation procedure that yields the three cost parameters."},{"cited_title":"Research exploring physical activity in care homes (REACH): study pro- tocol for a randomised controlled trial","cited_arxiv_id":null,"evidence_quote":"Describes the motivating cluster-randomised pilot trial whose design is revisited in the example."}],"review_version":1}