A Globally Calibrated Bayesian Optimal Phase II Design for Adaptive Enrichment Trials

Hisato Sunami; Masaaki Kuriki; Masahiro Kojima

REVIEW 3 major objections 3 minor 8 references

Jointly calibrating the all-comer and biomarker-positive thresholds in an adaptive enrichment BOP2 design keeps the probability of any false-positive efficacy claim at or below the nominal level, whereas independent calibration of the two c

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:17 UTC pith:BM3FJDXD

load-bearing objection A genuinely new exact enumeration for jointly calibrated enrichment BOP2, with a real but fixable scope gap in the 'global type I error' claim. the 3 major comments →

arxiv 2607.17692 v2 pith:BM3FJDXD submitted 2026-07-20 stat.ME

A Globally Calibrated Bayesian Optimal Phase II Design for Adaptive Enrichment Trials

Masahiro Kojima , Hisato Sunami , Masaaki Kuriki This is my paper

classification stat.ME MSC 62F1562L05

keywords adaptive enrichmentBayesian optimal phase II designBOP2global type I errorexact enumerationbiomarker-positive subgroupfutility monitoringbinary endpoint

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that an adaptive enrichment phase II trial—one that starts in an all-comer population and, after an all-comer futility boundary is crossed, can restrict enrollment to a biomarker-positive subgroup—can be designed so that the overall probability of making either efficacy claim is controlled when neither claim is true. The proposed method calibrates the all-comer and biomarker-positive monitoring thresholds jointly against the union of the two claims, rather than optimizing each population separately. Using an exact finite-state recursive enumeration, the authors show that under a point global null and for biomarker prevalences between 0.4 and 0.8, the maximum global false-positive rate is 9.9%, below the nominal 10%, while a comparator made of two independently calibrated BOP2 designs reaches 12.6%. The design's decision boundaries are fully prespecified before trial initiation, so no recalibration is required during the trial. A sympathetic reader would care because this makes adaptive enrichment practical in phase II without sacrificing control of false-positive conclusions.

Core claim

The paper claims that the global type I error of the full branching procedure is controlled by jointly calibrating the power-family posterior thresholds for the all-comer and biomarker-positive paths, and that this joint calibration is essential: an otherwise identical design formed by independently calibrating each BOP2 component inflates the probability of at least one false-positive claim from the nominal 10% to as high as 12.6% across the same prevalence range. For a binary endpoint, the operating characteristics are computed exactly by a recursive enumeration of all possible interim trajectories, which also accounts for the random number of biomarker-positive patients observed when enri

What carries the argument

The load-bearing mechanism is the BOP2 posterior-probability futility rule with sample-size-adaptive cutoff functions C_h(s)=1−λ_h (s/N_h)^γ_h, converted to integer count boundaries. The innovation is that the four tuning parameters (λ_A, γ_A, λ_+, γ_+) are selected jointly so that the maximum over the prevalence set of P(R_A ∪ R_+) is controlled at level α, instead of calibrating each population independently. This joint calibration is evaluated exactly by a finite-state recursion that sums over cohort-level binomial transitions, tracking the all-comer futility decision, the enrichment entry decision (which requires a minimum biomarker-positive sample size and a passing futility check), the

Load-bearing premise

The claim of global type I error control is verified only at the point where both response probabilities equal the null value p0, not across the entire composite null region, so it is an open question whether the error bound holds at other null configurations.

What would settle it

Compute, using the paper's exact recursion or by direct simulation, the global rejection probability at null configurations inside the region, such as (θ+, θ−)=(0.20, 0.10) or (0.15, 0.15), for the prespecified boundaries in Table 1. If any such null configuration yields P(R_A ∪ R_+) > 0.10, the 'global' error-control claim would not hold in the standard frequentist sense.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Trials can prospectively specify an adaptive enrichment path with two possible efficacy claims while maintaining a pre-set bound on the probability of any false-positive conclusion.
Because the operating characteristics come from exact enumeration, type I error and power can be reported without Monte Carlo uncertainty.
The same global-calibration principle is extended to complex categorical endpoints via a Dirichlet–multinomial formulation, as evaluated by simulation in the paper's supplement.
The decision boundaries can be tabulated before the trial starts, so the design is fully prespecified and does not require recalibration as data accumulate.
A separate-calibration comparator, which is the natural way to combine two ordinary BOP2 designs, overstates the error control of the combined procedure; this serves as a caution for other adaptive designs with multiple efficacy-claim paths.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper verifies error control only at the single null configuration θ+=θ-=p0; the null region is composite, so the 'global' label in the usual frequentist sense is supported only if the worst case occurs at that corner. A natural next check is to evaluate other null configurations with the same boundaries.
The exact recursive enumeration could be adapted to randomized or multi-arm enrichment designs, where the event of interest becomes a familywise error rate rather than a union of two claims.
The observed trade-off—a less stringent early all-comer boundary and more stringent later boundaries—suggests a general design principle for gated enrichment: spend less error early to allow enrichment, and recoup it later.
For complex categorical endpoints, the supplement's simulation-based calibration could be made exact by extending the finite-state recursion to multinomial counts, if the state space remains tractable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely new exact enumeration for jointly calibrated enrichment BOP2, with a real but fixable scope gap in the 'global type I error' claim.

read the letter

You should know two things about this paper. First, it does something new: it treats the all-comer and biomarker-positive BOP2 thresholds as one branching procedure and calibrates them against the union claim, using an exact recursion for the random-entry enrichment path. That is a real methodological step beyond Zhou et al. and the older enrichment designs, and the binary-endpoint analysis is internally coherent. Second, the main caveat is the type I error claim: the null hypothesis is composite, but the paper only calibrates and evaluates at the single point theta+ = theta- = p0. The abstract's phrase 'global type I error rate' is therefore stronger than what is actually demonstrated.

What the paper does well: the recursion in Section 2.3 is clean, and the tables are arithmetically consistent with it. The comparator — independently calibrated BOP2 embedded in the same branching procedure — is the natural one, and the inflation from 10.8–12.6% for that comparator is credible. The finding that joint calibration restores error control while improving power in all 30 reported scenarios is a solid, useful result. Code is provided, so the exact numbers are checkable. That matters.

The soft spots are modest in number. The point-null versus composite-null issue is the real one. The stress-test note is right that monotonicity is not obvious here: crossing the all-comer futility boundary gates entry into the enrichment path, so decreasing theta- can make futility stopping more likely while also changing the composition of the biomarker-positive data at entry. The worst-case type I error may or may not be at the corner, but the authors don't show it. They should either prove the relevant monotonicity or, at minimum, evaluate a few interior null configurations and clearly state that the guarantee is point-null only unless proven otherwise. They do hedge in places, saying 'prespecified point global null,' but the abstract and discussion leave the impression of full frequentist control. That needs fixing.

Minor: the selected (lambda_A, gamma_A, lambda_+, gamma_+) values are not reported, only the resulting boundaries, and the complex-categorical extension lives in an unavailable supplement. Neither is a serious problem.

This paper deserves a serious referee. The method is useful, the exact evaluation is a genuine contribution, and the open question about the null region is answerable. I would send it to review and ask for a monotonicity argument or a clearly qualified claim, plus a couple of null-region evaluations. I'd also bring it to reading group; it's a good example of a design paper where the stated guarantee is slightly ahead of the proof.

Referee Report

3 major / 3 minor

Summary. The paper proposes a Bayesian optimal phase II design for a single-arm adaptive enrichment trial with a binary endpoint. Enrollment starts in the all-comer population; at prespecified interim analyses, if the all-comer futility boundary is crossed and the accumulated biomarker-positive data satisfy a minimum-sample-size and futility requirement, enrollment is restricted to biomarker-positive patients, with post-enrichment monitoring and a final efficacy claim. The all-comer and biomarker-positive BOP2-type thresholds are jointly calibrated so that, over a prespecified set of prevalences, the probability of the union of the two efficacy claims under the point global null H0: theta+ = theta- = p0 is at most alpha. The paper derives an exact finite-state recursive enumeration for a binary endpoint and uses it to calibrate and evaluate the design. It reports that the proposed design controls the global type I error, with maximum rates 8.7%-9.9% across prevalences, whereas an independently calibrated BOP2 comparator yields 10.8%-12.6%, and that the proposed design has higher global power in all 30 evaluated alternative scenarios.

Significance. If the central claim is taken together with the point-null caveat, this is a useful and practical contribution: the exact recursive enumeration removes Monte Carlo error from operating-characteristic evaluation, the decision rules are fully prespecified before trial initiation, and the authors provide open-source code for reproducibility. The comparison with the independently calibrated BOP2 design is informative and highlights the inflation from separate calibration of the two paths. The main methodological question is whether the phrase 'global type I error control' is justified for the composite null, or only for the single point theta+ = theta- = p0.

major comments (3)

[Section 2.2 and Section 3.1] The design is calibrated and evaluated only at the point global null theta+ = theta- = p0. The constraint is max_{pi} P_{H0,global,pi}(R_A union R_+) <= alpha, and Section 3.1 reports type I error only under this single configuration. However, the actual null hypothesis for the pair of one-sided efficacy claims is composite: theta+ <= p0 and theta- <= p0. The paper neither proves that P(R_A union R_+) is monotone nondecreasing in theta+ and theta- over this null region nor evaluates any other null configuration. This is not a formal quibble: because enrichment is entered only after an all-comer futility boundary is crossed, decreasing theta- can increase the probability of crossing that boundary and thereby increase the opportunity to make a biomarker-positive claim, even when theta+ = p0. A worst-case type I error could therefore occur inside the null region rather than at the corner. T
[Abstract and Discussion] The wording alternates between 'controlled the global type I error rate' and 'under the prespecified point global null.' If the intended guarantee is only at the point null, the abstract and discussion should say so consistently to avoid implying the standard frequentist composite-null control. If the intended guarantee is over the composite null, the missing evaluation described in the previous comment is load-bearing for the paper's main contribution.
[Section 2.3, q+(m,x) recursion] The recursion is internally coherent, but the formula omits an explicit statement that when an interim biomarker-positive futility boundary is crossed, the conditional rejection probability is zero. The indicator function in the displayed formula implicitly handles this, but a one-sentence clarification would prevent reader confusion in what is otherwise the technical core of the paper.

minor comments (3)

[Table 2] The caption states that boldface indicates PRN-any, but the rendered table has no bold entries. Please make the primary measure visually distinct or remove the note.
[Section 3.1] The comparator is calibrated with a finer grid (lambda step 0.001 vs. 0.005 for the proposed design). This is unlikely to affect conclusions, but the asymmetry should be acknowledged as a possible source of small differences in power.
[General] The Supplementary Material is referenced for the categorical-endpoint extension and for further enumeration details, but it does not appear in the arXiv listing. Please ensure it is available to readers and reviewers.

Circularity Check

0 steps flagged

No significant circularity: the design is calibrated to an explicit constraint and the reported operating characteristics are exact evaluations of the selected rules, not fitted predictions.

full rationale

The derivation chain is self-contained. The paper's central claim is that jointly calibrating BOP2 thresholds for the branching enrichment procedure controls the global type I error at the prespecified point null θ+ = θ- = p0 over Π, while the independently calibrated comparator does not. The tuning parameters (λ_A, γ_A, λ_+, γ_+) are chosen by grid search to satisfy max_{π∈Π} P_{H0,π}(R_A ∪ R_+) ≤ α, so the reported type I error rates in Table 2 are exact evaluations of the selected design—an operating characteristic, not a prediction from a fitted input. Power at the working alternative (p1,p0) is used only as a tie-breaker among feasible designs, and 29 of 30 alternative scenarios are evaluated at configurations away from the calibration point, providing an independent check. The BOP2 threshold family is adopted from Zhou et al. (2017) with a standard citation; none of the load-bearing mathematical steps reduce to a self-citation. The only notable limitation—that point-null calibration does not by itself prove composite-null control—is a correctness/interpretation concern about the breadth of the type I error claim, not a circularity: the paper explicitly and repeatedly qualifies its claim as 'under the prespecified point global null.' No fitted parameter is renamed as an independent prediction, and no uniqueness theorem or ansatz is imported from the authors' own prior work.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The four BOP2 tuning parameters are the only fitted quantities; their final values are not printed. The most consequential unstated premise is the point-to-composite null extrapolation. No new entities beyond the trial design itself are introduced.

free parameters (4)

lambda_A (all-comer threshold tuning parameter) = not reported
BOP2 power-family slope parameter for the all-comer futility boundary; calibrated by grid search as part of the joint procedure.
gamma_A (all-comer threshold shape parameter) = not reported
BOP2 power-family exponent for the all-comer path; jointly calibrated.
lambda_+ (biomarker-positive threshold tuning parameter) = not reported
BOP2 power-family slope parameter for the biomarker-positive enrichment path; jointly calibrated.
gamma_+ (biomarker-positive threshold shape parameter) = not reported
BOP2 power-family exponent for the biomarker-positive path; jointly calibrated.

axioms (4)

domain assumption Data are generated as independent Bernoulli responses with fixed subgroup probabilities theta+ and theta-, and biomarker status is Bernoulli(pi); all patients are i.i.d.
Section 2.3 builds the exact transition probabilities on this i.i.d. Binomial model; if the trial has time trends or heterogeneous subgroups, the computed error rates do not apply.
standard math Posterior probability P(theta_h <= p0 | x) is monotone decreasing in x, so the posterior rule is equivalent to integer futility boundaries b_h(s).
Used in Section 2.1 to define b_h(s); true for the Beta-Binomial model.
ad hoc to paper Calibrating at the point global null theta+ = theta- = p0 is sufficient to claim 'global type I error control'; no proof is given that the error is controlled over the composite null {theta+ <= p0, theta- <= p0}.
Section 2.2 defines H0,global as this single point and Section 3.1 evaluates only this point; the extension to the full null region is never tested.
domain assumption The BOP2 power-family threshold grid contains a design satisfying the type I error constraint.
The calibration procedure depends on the existence of a feasible candidate on the specified (lambda,gamma) grids; the authors report finding one.

pith-pipeline@v1.3.0-alltime-deepseek · 10395 in / 32452 out tokens · 325159 ms · 2026-08-01T17:17:46.401999+00:00 · methodology

0 comments

read the original abstract

Adaptive enrichment can allow the development of an experimental treatment to continue when its activity is insufficient in an all-comer population but remains promising in a prespecified biomarker-positive subgroup. However, a straightforward sequential application of separately calibrated phase II designs to the two populations can inflate the probability of a false-positive efficacy conclusion. The Bayesian optimal phase II (BOP2) design uses posterior-probability thresholds for interim futility monitoring and final efficacy decisions in single-arm phase II trials. We develop a pathwise globally calibrated BOP2 framework for branching adaptive enrichment trials. At prespecified all-comer interim analyses, the trial either continues enrollment in the all-comer population or, after the all-comer futility boundary is crossed, transitions to a prespecified biomarker-positive enrichment path. The all-comer and biomarker-positive thresholds are jointly calibrated against the union of the two possible efficacy claims while accounting for the random biomarker-positive sample size available when enrichment is initiated. All decision rules remain prespecified before trial initiation. For a binary endpoint, we derive an exact finite-state recursive enumeration of the complete adaptive procedure, enabling both calibration and operating-characteristic evaluation without Monte Carlo error. Using this exact procedure, we showed that the proposed design controlled the global type I error rate under the prespecified point global null across the prespecified range of biomarker prevalence, whereas the separately calibrated BOP2 approach did not. Under alternative scenarios, efficacy claims arose through both the all-comer and biomarker-positive paths, with their relative contributions depending on biomarker prevalence and subgroup response probabilities.

discussion (0)

Reference graph

Works this paper leans on

8 extracted references

[1]

Jack and Yuan, Ying , journal=

Chen, Kai and Zhou, Heng and Lee, J. Jack and Yuan, Ying , journal=. 2026 , publisher=

2026
[2]

and Takeda, Kentaro , journal=

Xu, Xinling and Hashimoto, Atsuki and Yimer, Belay B. and Takeda, Kentaro , journal=. 2025 , publisher=

2025
[3]

and Kolamunnage-Dona, Ruwanthi , title =

Antoniou, Marcela and Jorgensen, Andrea L. and Kolamunnage-Dona, Ruwanthi , title =. PLOS ONE , year =
[4]

Pharmaceutical Statistics , year =

Jenkins, Martin and Stone, Andrew and Jennison, Christopher , title =. Pharmaceutical Statistics , year =
[5]

Biostatistics , year =

Simon, Noah and Simon, Richard , title =. Biostatistics , year =
[6]

and Turnbull, Bruce W

Magnusson, Baldur P. and Turnbull, Bruce W. , title =. Statistics in Medicine , year =
[7]

Statistics in Medicine , year =

Brannath, Werner and Zuber, Emmanuel and Branson, Michael and Bretz, Frank and Gallo, Paul and Posch, Martin and Racine-Poon, Amy , title =. Statistics in Medicine , year =
[8]

Jack and Yuan, Ying , journal=

Zhou, Heng and Lee, J. Jack and Yuan, Ying , journal=. 2017 , publisher=

2017

[1] [1]

Jack and Yuan, Ying , journal=

Chen, Kai and Zhou, Heng and Lee, J. Jack and Yuan, Ying , journal=. 2026 , publisher=

2026

[2] [2]

and Takeda, Kentaro , journal=

Xu, Xinling and Hashimoto, Atsuki and Yimer, Belay B. and Takeda, Kentaro , journal=. 2025 , publisher=

2025

[3] [3]

and Kolamunnage-Dona, Ruwanthi , title =

Antoniou, Marcela and Jorgensen, Andrea L. and Kolamunnage-Dona, Ruwanthi , title =. PLOS ONE , year =

[4] [4]

Pharmaceutical Statistics , year =

Jenkins, Martin and Stone, Andrew and Jennison, Christopher , title =. Pharmaceutical Statistics , year =

[5] [5]

Biostatistics , year =

Simon, Noah and Simon, Richard , title =. Biostatistics , year =

[6] [6]

and Turnbull, Bruce W

Magnusson, Baldur P. and Turnbull, Bruce W. , title =. Statistics in Medicine , year =

[7] [7]

Statistics in Medicine , year =

Brannath, Werner and Zuber, Emmanuel and Branson, Michael and Bretz, Frank and Gallo, Paul and Posch, Martin and Racine-Poon, Amy , title =. Statistics in Medicine , year =

[8] [8]

Jack and Yuan, Ying , journal=

Zhou, Heng and Lee, J. Jack and Yuan, Ying , journal=. 2017 , publisher=

2017