{"id":"c58a63c0-5df2-4c2c-b626-b076419ffc3f","arxiv_id":"1908.07751","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper formalizes dual-criterion phase II designs that require both statistical significance and a clinically relevant effect estimate, and gives sample size and operating-characteristic methods.","lead":"This paper proposes a phase II trial design that declares success only when a result is both statistically significant and clinically relevant. It shows how to size such trials and compares the approach with standard designs in two oncology examples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sample-size formula in Eq. (1) omits the log transform for time-to-event endpoints; as printed it yields 73 events, not the reported 52.","rationale":"The reader's weakest assumption correctly identifies the log-scale omission in the sample-size formula. This is the most load-bearing issue because the sample size is the explicit quantitative deliverable of the dual-criterion design; the printed formula is not reproducible for the paper's own leading example. The underlying design logic remains sound, and the operating-characteristic tables are self-consistent once the log transform is assumed, so I do not see a fatal flaw. The formula must be corrected and the scale of NV and DV made explicit before the design is implemented by others. A secondary issue worth clarifying: for n ≥ nmin, the clinical criterion implies the statistical criterion under the planned model, so Table 2's case 4 is empty unless variability is larger than assumed; this does not change the conditional verdict but should be acknowledged in a revision.","tokens_in":11511,"tokens_out":13167,"duration_ms":137684,"concrete_test":"Recompute nmin for Example 1 using the printed formula with NV = 1, DV = 0.7, sigma = 2, alpha = 0.1 on the natural scale (≈73), then repeat with NV = log(1) = 0 and DV = log(0.7) (≈52). If the intended value is 52, amend the formula to define NV and DV as log-transformed parameters for time-to-event endpoints and confirm the corrected equation reproduces the 52-event example and the 309-event example.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 'Sample size' states nmin = σ² z_α² / (NV − DV)² for normally distributed data, and for time-to-event data says σ = 2. For Example 1 (NV = 1, DV = 0.7, α = 0.1), the printed formula gives 4 × 1.2816² / (1 − 0.7)² ≈ 73 events, but the paper reports nmin = 52. The reported value follows only if NV and DV are log hazard ratios: 4 × 1.2816² / (log(1) − log(0.7))² ≈ 52. The manuscript never states that the formula's NV and DV are on the log scale; it only says the calculation used approximate normality of the log-hazard-ratio. Because nmin is the central design output, an implementer following the equation as written will compute 73 events, changing the planned type-I error and trial size. This is a load-bearing omission, not a cosmetic typo.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a dual-criterion design for phase II proof-of-concept trials in which a GO decision requires both statistical significance and a clinically relevant effect estimate. The frequentist version requires a one-sided p-value below alpha and an estimate at least as favorable as a pre-specified decision value (DV); the Bayesian version requires a posterior probability above 1−alpha and a posterior median reaching the DV. The paper derives the minimum sample size needed for the estimate at the DV to imply significance, discusses operating characteristics including the 50% power-at-DV property, and presents two worked examples: a randomized time-to-event trial analyzed frequentist and a single-arm binary trial analyzed Bayesian, with comparisons to randomized screening designs and three-outcome designs.","tokens_in":11643,"tokens_out":27932,"duration_ms":223685,"significance":"The paper addresses a real and recurring practical problem: statistical significance alone is often insufficient for phase II GO/NO-GO decisions. Its main contribution is conceptual clarity: making the clinically relevant effect threshold an explicit design input and showing how the resulting operating characteristics differ from standard designs. If the two issues identified below are corrected, the manuscript would be a useful, accessible methods note for clinical statisticians. The example tables are informative, the comparison with alternative designs is helpful, and the distinction between the decision value and the alternative hypothesis is well explained. The work does not require heavy machinery, and its value lies in transparent communication rather than theoretical novelty.","major_comments":[{"comment":"The sample-size formula is stated for NV and DV as if they were on the natural effect scale, but the time-to-event example requires them to be on the log-hazard-ratio scale. With alpha=0.1, sigma=2, NV=1, DV=0.7, the printed expression gives sigma^2 z_alpha^2/(NV-DV)^2 = 4 x 1.2816^2 / 0.3^2 = 73 events, whereas the paper reports nmin=52. The reported value is obtained only if the denominator is (log(1)-log(0.7))^2, i.e. the null and decision values are log hazard ratios. The text mentions 'approximate normality of the log-hazard-ratio' but never states that NV and DV in the formula must be transformed accordingly. Because nmin is the central design output and directly determines the planned type-I error and power, this is a load-bearing omission. Please restate the formula in terms of the transformed parameter, for example theta = log(HR) with NV* = log(NV) and DV* = log(DV).","section":"Sample size section and Example 1"},{"comment":"In the discussion of Example 1 (design 1, n=70), the text says 'The type-I error is 0.032'. Table 3 reports for true HR=1.0: GO=0.068, NO-GO=0.900, inconclusive=0.032. The probability of a GO under the null is 0.068, not 0.032; 0.032 is the probability of the inconclusive outcome (significant p-value but HR estimate above the DV). Since type-I error control is one of the paper's central claims, this number must be corrected (to 0.068) or the definition must be clarified explicitly. The same table also shows P(significant | HR=1) = GO+inconclusive = 0.100, so if the authors intend to report the probability of any significant result, that should be stated.","section":"A randomized PoC design with time-to-event data, Table 3"}],"minor_comments":[{"comment":"The word 'Notingly' should be 'Notably'.","section":"Operating characteristics section"},{"comment":"The word 'ploting' should be 'plotting'.","section":"Final analysis section"},{"comment":"In design 1, the intervals defining GO, NO-GO, and inconclusive should be given explicitly (for example, GO: theta-hat <= 0.7, inconclusive: 0.7 < theta-hat <= 0.736, NO-GO: theta-hat > 0.736); readers currently have to infer the interval boundaries from the text.","section":"Table 3"},{"comment":"The sample-size formula would benefit from an equation number and a one-line definition of whether higher or lower values are favorable, to avoid sign confusion.","section":"Sample size section"},{"comment":"Describing Beta(0.0811,1) as 'unimodal' is debatable because this density is monotone decreasing on (0,1); consider calling it 'J-shaped' or 'boundary-mode'.","section":"Single-arm PoC design with binary data"}],"recommendation":"major_revision","confidential_remarks":"The two substantive issues (the scale of the sample-size formula and the misreported type-I error in the first example) are correctable within the scope of a revision. The paper is within the journal's scope and would be acceptable after these corrections. Code is only available on request, which limits reproducibility, but this is not a blocker."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. This paper is a clear, useful exposition of a dual-criterion phase II design: success requires both statistical significance and an effect estimate reaching a clinically pre-specified decision value. The central logic is sound, the examples are realistic, and the operating characteristics are internally consistent. The soft spot the stress-test flags is real and load-bearing: the sample size formula is mis-specified as printed for the time-to-event example. Plug NV=1, DV=0.7, sigma=2, alpha=0.1 into Eq. (1) and you get about 73 events; the text reports 52. The reported value only emerges if the denominator uses log(NV) - log(DV), meaning NV and DV are log-hazard-ratios. The manuscript says the calculation used approximate normality of the log-hazard-ratio but never states that the formula's inputs are on that scale. A statistician following the equation will oversize the trial by about 40% and change the planned type-I error and power.\n\nWhat the paper does well: it makes the clinical decision value an explicit design input, explains why power at the decision value is (about) 50% and why that is not a defect, and gives clear frequentist and Bayesian implementations. The comparison with standard designs and three-outcome designs is genuinely helpful. Tables 3 and 4 look right if you work through the numbers. The novelty is modest -- Neuenschwander et al. 2011 and Chuang-Stein et al. 2011 already contain the core idea, and the paper cites them honestly -- but there is value in this systematized presentation with worked examples.\n\nBeyond the formula issue, the weaknesses are minor. The R code is 'available from the author,' not shipped, so a skeptical reader cannot fully verify the grid search example or the operating characteristics. There are a few typos ('Notingly,' 'ploting') and the phrase 'power is implicit' is itself a little implicit in places. None of that changes the verdict. The citation pattern is fine; the earlier citations are appropriate and not used as crutches.\n\nWho is this for? Statisticians and clinical teams designing proof-of-concept trials, especially in oncology and other TTE settings. It deserves a serious referee. My recommendation: send it to peer review, but require the authors to fix the sample size formula (or clearly define the scale of NV and DV), and ideally post the R code. I would cite this paper for the dual-criterion concept, not for the sample size formula as printed.","headline":"A clear, practical dual-criterion phase II design paper whose central logic is sound, but the printed sample size formula omits the log transform for time-to-event endpoints, so implementers will compute 73 events instead of the reported 52.","tokens_in":12199,"tokens_out":4022,"would_cite":true,"duration_ms":40731,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62P10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Phase II success should require both a significant p-value and an effect estimate that passes a clinically chosen threshold.","keywords":["phase II clinical trials","dual-criterion design","clinical relevance","statistical significance","GO/NO-GO decisions","type-I error","sample size","proof-of-concept"],"falsifier":"A reader can check the formula by plugging in the paper's Example 1 values: with $NV=1$, $DV=0.7$, $\\sigma=2$, and $\\alpha=0.1$, the published formula gives roughly 73 events, whereas the paper reports 52; the reported 52 only appears when the denominator uses $\\log(1)-\\log(0.7)$. That discrepancy would falsify the formula as printed.","tokens_in":11278,"feed_emoji":"📊","tokens_out":6783,"duration_ms":512232,"temperature":0.7,"pith_summary":"The paper argues that standard phase II designs, which define success by a statistically significant p-value, can certify effects that clinicians do not consider meaningful. It proposes a dual-criterion design in which success requires both statistical significance (or its Bayesian analogue) and an effect estimate that reaches a pre-specified clinically motivated decision value. The authors show that this design controls the type-I error at the planned level, turns the clinical threshold into an explicit input, and yields GO, NO-GO, and inconclusive decisions that are easier to defend. If the paper is right, phase II proof-of-concept decisions become a joint statistical-clinical statement rather than a purely null-rejection exercise.","feed_headline":"Dual-criterion design makes clinical relevance a formal trial gate","feed_subtitle":"The design keeps GO/NO-GO decisions tied to an explicit clinical threshold, not just statistical significance.","key_machinery":"The carrying object is the dual criterion itself, together with the sample-size floor that makes it operative: the smallest $n$ at which a significant test result and an estimate at the decision value coincide. The formula $n_{\\min} = \\sigma^2 z_\\alpha^2 / (NV - DV)^2$ converts the two benchmarks into a required number of events or patients; for non-normal endpoints a grid search over $n$ plays the same role. The Bayesian version replaces the p-value with a posterior probability and the point estimate with the posterior median, and the paper uses a minimally informative Beta prior in its single-arm example.","core_discovery":"The paper claims that a phase II success criterion should be the conjunction of two conditions rather than a single p-value threshold: the one-sided p-value must fall below $\\alpha$ (or, Bayesianly, the posterior probability that the effect exceeds the null value must exceed $1-\\alpha$), and the effect estimate must reach a pre-specified clinical decision value. With both criteria met the decision is GO; with neither, NO-GO; with one of the two, the outcome is inconclusive. The paper shows that this design preserves type-I error control, makes the clinically relevant effect an explicit design input, and has a sample size floor $n_{\\min} = \\sigma^2 z_\\alpha^2 / (NV - DV)^2$ below which the clinical criterion cannot be met whenever significance holds. It also establishes that power at the decision value is about 50% and is not changed by increasing the sample size.","pith_inferences":["If the same criterion were applied in confirmatory trials, a significant but small effect would fail the clinical gate, formally preventing 'significant but trivial' phase III results; the paper only raises this as a possibility.","The formula can be inverted as a calibration rule: to achieve a target power at a desired effect, set the decision value below that effect and compute $n$; this inversion is not given in the paper.","Reported dual-criterion operating characteristics could be audited by checking that the implied estimate threshold equals the stated decision value at the minimum sample size, a consistency test readers can run from the paper's tables."],"forward_implications":["A dual-criterion design makes the clinical threshold a design input, so the protocol states the smallest effect estimate that justifies GO.","Because power at the decision value is about 50%, sample size increases only shift power for effects above the decision value; effects below it remain clinically irrelevant.","Planning with the minimum sample size avoids inconclusive outcomes entirely, while planning above it can produce significant-but-clinically-irrelevant results that need further judgment.","In the Bayesian version, a weak prior gives operating characteristics close to the frequentist design, but the posterior probability and p-value retain distinct evidential meanings.","The same dual-criterion logic is transferable to non-inferiority trials, bridging, dose-finding, and interim futility decisions."],"supporting_citations":[{"why":"Supplies the motivating problem: p-values alone do not convey clinical relevance.","marker":"[1]"},{"why":"Provides the three-outcome design used as the comparator in the single-arm binary example.","marker":"[9]"},{"why":"Provides the randomized phase II screening designs used as comparators in the time-to-event example.","marker":"[11]"},{"why":"Grounds the clinical decision value in the minimum clinically important difference tradition.","marker":"[16]"},{"why":"Extends the same statistical-clinical logic to proof-of-concept non-inferiority, a setting discussed as an application.","marker":"[18]"},{"why":"Used to argue that p-values and posterior probabilities, while numerically close, have different meanings.","marker":"[23]"}],"fun_headline_variants":["Two thresholds beat one for phase II trials","A second gate: clinical effect for trial GO","Phase II GO now needs significance plus clinical effect","Make clinical relevance a formal trial criterion","Dual-criterion design: p-value and effect size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the effect estimate follows a normal distribution with known variance, and for the time-to-event example that the null and decision values enter the formula as log hazard ratios; the paper does not state the log transform, and without it the formula yields the wrong sample size.","fun_headline_variants_meta":{"raw":{"variants":["Two thresholds beat one for phase II trials","A second gate: clinical effect for trial GO","Phase II GO now needs significance plus clinical effect","Make clinical relevance a formal trial criterion","Dual-criterion design: p-value and effect size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000494,"raw_usage":{"total_tokens":2442,"prompt_tokens":977,"completion_tokens":1465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1395}},"tokens_in":593,"tokens_out":1465,"duration_ms":162918,"temperature":1.0,"reasoning_tokens":1395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:56:42.268086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader can check the formula by plugging in the paper's Example 1 values: with $NV=1$, $DV=0.7$, $\\sigma=2$, and $\\alpha=0.1$, the published formula gives roughly 73 events, whereas the paper reports 52; the reported 52 only appears when the denominator uses $\\log(1)-\\log(0.7)$. That discrepancy would falsify the formula as printed.","supporting_citations":[{"cited_title":"The ASA’s statement on p-values: Context, process, and purpose","cited_arxiv_id":null,"evidence_quote":"Supplies the motivating problem: p-values alone do not convey clinical relevance."},{"cited_title":"A three-outcome design for phase II clinical trials","cited_arxiv_id":null,"evidence_quote":"Provides the three-outcome design used as the comparator in the single-arm binary example."},{"cited_title":"Design issues of randomized phase ii trials and a proposal for phase II screening trials","cited_arxiv_id":null,"evidence_quote":"Provides the randomized phase II screening designs used as comparators in the time-to-event example."},{"cited_title":"The role of the minimum clinically important difference and its impact on designing a trial","cited_arxiv_id":null,"evidence_quote":"Grounds the clinical decision value in the minimum clinically important difference tradition."},{"cited_title":"A proof of concept phase II non-inferiority criterion","cited_arxiv_id":null,"evidence_quote":"Extends the same statistical-clinical logic to proof-of-concept non-inferiority, a setting discussed as an application."},{"cited_title":"Testing a point null hypothesis: The irreconcilability of P values and evidence (C/R: P123-133, 135-139, 1201-1201)","cited_arxiv_id":null,"evidence_quote":"Used to argue that p-values and posterior probabilities, while numerically close, have different meanings."}],"review_version":1}