Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Bayesian Design of Experiments in the Presence of Nuisance Parameters

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BART-BvM turns Bayesian design calculations into a closed-form normal approximation, with nuisance parameters handled by one learned scale factor.

desk verdict A useful, honest computational trick—BvM plus BART to emulate the variance scale—that works in two examples, but the general claims outrun the evidence. read the letter →

arxiv 2508.03948 v1 pith:DFVVFAUM submitted 2025-08-05 stat.ME

Shirin Golchi , Luke Hagar This is my paper · ORCID
classification stat.ME MSC 62K0562F1562L10
keywords BayesiandesignofexperimentsassurancenuisanceparameterssamplesizedeterminationBernstein-vonMisesadditiveregressiontreesgroupsequentialdecisiontheory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Designing a Bayesian study usually requires Monte Carlo estimation of the sampling distribution of posterior summaries for every candidate sample size, and nuisance parameters make the search even heavier because they must be integrated out under the design prior. This paper claims the whole calculation can be collapsed into a closed-form approximation: after a one-time BART fit that learns how the asymptotic variance parameter varies over the parameter space, power, assurance, expected sample size, and expected cost follow by plugging values into a normal probability formula. The method is justified by the Bernstein-von Mises theorem, so it inherits both the speed of asymptotic formulas and the flexibility to average over uncertainty in every parameter. The authors caution that the approximation is asymptotic, so it is positioned as an efficient way to screen designs, with Monte Carlo kept for a few final options.

What carries the argument

Equation (8), the analytic power formula with a BART-emulated scale parameter $\lambda(\theta)$. Here $\lambda$ is the per-observation variance appearing in both the approximate posterior and the sampling distribution of the MLE; the empirical step estimates $\lambda$ from simulated posterior variances at a few points, and the regression step uses BART/SoftBART to interpolate it across the design-prior region. This single emulation carries the entire burden of nuisance-parameter dependence, because the mean structure $\sqrt{n}(\psi-\psi_0)/\lambda$ and the standard normal error are otherwise parameter-free.

What would settle it

In the logistic-regression example, at a planned sample size such as $n=100$, repeatedly simulate data sets at a fixed $\theta$ in the design-prior support, compute the posterior probability $\tau$, and histogram $\Phi^{-1}(\tau)$. If the histogram's variance is not close to 1 or its shape is not normal, equation (8) with BART-predicted $\lambda$ cannot give correct power; equivalently, compare equation (8) power against empirical rejection rates and look for deviations larger than Monte Carlo error.

Watch

Extended reading notes

Core claim

The core object is an analytic power function. Under the Bernstein-von Mises theorem, the probit of the posterior probability that $\psi(\theta)>\psi_0$ is approximately $\sqrt{n}/\lambda(\theta)\,(\psi(\theta)-\psi_0)$ plus a standard normal error, so power is $1-\Phi(\Phi^{-1}(u)-\sqrt{n}/\lambda(\theta)(\psi(\theta)-\psi_0))$. The only unknown is $\lambda(\theta)$, the per-observation variance scale of the posterior and of the MLE, which depends on nuisance parameters through the Fisher information. The paper estimates $\lambda$ by simulation at a modest set of parameter values and trains a Bayesian additive regression tree to predict $\lambda$ anywhere in the support of the design prior;

Load-bearing premise

The load-bearing premise is that the planned sample sizes are large enough for the Bernstein-von Mises normal approximation and for $\hat\lambda\approx\lambda^*$ to hold, so the probit of the posterior probability is approximately a standard normal plus a drift; at small sample sizes the power formula is biased, and the BART-emulated $\lambda$ cannot repair that bias.

Editorial extensions

If this is right

  • Once $\lambda$ is fitted, integrated power (Bayesian assurance) becomes a numerical integral of a closed-form normal probability, so comparing many candidate sample sizes and design priors no longer requires repeated posterior sampling.
  • For group sequential designs, the joint distribution in (13) gives stage-wise stopping probabilities and expected sample sizes directly from one fitted $\lambda$, making adaptive-design screening much cheaper.
  • Decision-theoretic criteria such as integrated expected cost become easy to evaluate across many designs, enabling optimization over thresholds, sample sizes, and the number and spacing of interim analyses.
  • Because the method is parameterized only by $\lambda$, the same BART fit can be reused to evaluate different decision thresholds or different candidate designs without additional training.
  • The paper states the approach generalizes from one-sided to interval hypotheses, extending its use to equivalence and other two-sided design problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same fitted $\lambda$ should only be trusted within the training region: if the design prior is widened or shifted substantially, the BART model must be retrained on the new support.
  • Because the joint distribution in (13) is threshold-agnostic, futility boundaries or two-sided decision rules could be added by changing only the limits of integration, without any new BART fit; the examples stop only for efficacy.
  • A practical diagnostic suggested by the approach but not developed is to run full Monte Carlo at one or two design points, such as a small and a moderate sample size, as a calibration check before trusting Equation (8) across the whole parameter space.
  • The method's small-sample weakness could potentially be mitigated by the small-sample corrections the authors mention, but such corrections are not part of the current procedure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes BART-BvM, an approximation to Bayesian operating characteristics for fixed and group sequential designs in the presence of nuisance parameters. The authors combine the Bernstein-von Mises theorem with asymptotic normality of the MLE to approximate the probit-transformed posterior probability of success as approximately √n/λ(ψ(θ)-ψ0)+ε, where λ is a parameter-dependent variance scale learned by Bayesian additive regression trees (BART) from a small Monte Carlo training set. This yields analytic power, assurance, and stopping probabilities. The approach is demonstrated on a logistic regression example and on a hypothetical Bayesian group sequential design based on the PLATINUM-CAN trial, with leave-one-out cross-validation deviations below 0.04. The paper includes a public code repository.

Significance. If the issues below are resolved, the method has clear practical value: it offers a fast approximation to Monte Carlo-based Bayesian design, especially for nuisance-parameter settings, and it extends naturally to group sequential designs through a joint canonical distribution. The public code and the LOOCV validation on two examples are strengths. However, the central equations contain a definitional inconsistency in the scale parameter λ, and the claim of instantaneous power curves over a range of sample sizes is not fully supported by the validation. The approach is explicitly acknowledged by the authors to be not broadly generalizable, which further tempers the advertised scope.

major comments (3)
  1. [Section 2.2, Eqs. (5)-(8) and Section 2.3, Eq. (13)] The definition of λ is internally inconsistent. The text states 'let λ̂/n be the variance in (5) and λ*/n be the variance in (6)', so λ is the asymptotic variance of √n ψ̂. Under that definition, ε = √n/λ(ψ̂_n - ψ*) has variance 1/λ, not 1, and the approximation γ_n ≈ √n/λ(ψ* - ψ0) + ε, and hence Eq. (8), does not follow. The group sequential covariance calculation in Section 2.3 uses Var(ψ̂_k) = λ²/n_k, and the empirical construction in Section 3 ('√n times the standard deviation of this sampling distribution') corresponds to treating λ as the standard-deviation scale (Var = λ²/n). The paper should consistently define λ²/n as the variance and re-derive the equations, or explicitly state that λ denotes the standard-deviation scale. As printed, a literal reader of (5)-(8) will obtain a power function that is off by a factor of √λ.
  2. [Section 3, Figure 2] The claim of 'instant estimation of integrated power curves for any sample size' is not validated. The BART model is trained at n = 500 and the LOOCV in Figure 1 is only at n = 500. Figure 2 shows BART-BvM power curves over a range of sample sizes, but the Monte Carlo points are only for n = 500. If the finite-sample scale λ depends on n, the curves at other sample sizes will be biased. The GSD example partially checks this (n = 350, 500, 700), but the fixed-design sample-size sweep is not empirically supported. The paper should add Monte Carlo checks at several sample sizes or restrict the claim to sample sizes near the training n.
  3. [Section 2.3 and Supplement A.2] The assumption that λ is constant across stages is load-bearing for Eq. (13), but it is justified only by asymptotic argument and by one LOOCV exercise (design D1). The reported maximum deviation below 0.04 is reassuring but does not establish general validity for other stage spacings, sample sizes, or analysis models. The authors themselves state in the Discussion that the satisfactory performance is not broadly generalizable. A sensitivity analysis that trains λ at different interim sample sizes, or a diagnostic for assessing constancy of λ across stages, would substantially strengthen the GSD claim.
minor comments (5)
  1. [Section 2.2] Typo: 'identifiablity' should be 'identifiability'.
  2. [Equation (16)] The symbol I is called an 'identity function' but it is used as an indicator function; please correct the terminology.
  3. [Figure 2 caption] Please state explicitly that the yellow dots are Monte Carlo estimates obtained at n = 500 only, to avoid the impression that the displayed curves are validated at all sample sizes.
  4. [Section 5] The caveat that the approximation 'may be poor for small sample sizes and in more complex models' is important and should appear earlier (e.g., in the abstract or when Eq. (8) is introduced), not only in the discussion.
  5. [References] Several author names contain accent-mark LaTeX artifacts (e.g., 'Fl´ orez Rivera'); please fix the encoding.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BART-BvM derives an analytic power approximation from BvM and validates it against independent Monte Carlo power estimates.

full rationale

The paper's claimed contribution is the analytic approximation of the power function, Eq. (8), obtained from the Bernstein-von Mises normal approximation and a BART surrogate for the scale parameter λ. The derivation chain is: (5)-(6) are standard BvM/MLE asymptotics; the simplification λ̂≈λ* is stated as a large-sample assumption; λ(θ) is estimated from Monte Carlo sampling distributions at a small Latin-hypercube training set and interpolated by BART; Eq. (8) then follows by a probit transformation of the approximate normal decision statistic. The BART model is trained on λ values, not on power values, so the LOOCV comparisons in Section 3 and Supplement A.2 against full Monte Carlo power are external benchmarks rather than in-sample fits. The GSD extension (Section 2.3) uses the same λ at all stages under the stated assumption that λ is constant across sample sizes; this is an asymptotic modeling assumption, tested at n=350,500,700 in Supplement A.2, not a circular reduction. Self-citations to Golchi (2022), Golchi and Willard (2024), and Hagar et al. appear only as related work or suggested extensions and do not carry the derivation; the load-bearing distributional results are cited to external standard references (van der Vaart; Jennison and Turnbull). The Discussion's caveat that performance may be poor for small samples or complex models is an honest limitation, not an admission of circularity. No equation in the paper reduces to its own fitted output, and no parameter is fit to the target operating characteristic. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central formula (8) has no fitted parameters: λ is a variance parameter estimated from simulations and predicted by BART, not tuned to match the power target. The main assumptions are the asymptotic normal approximations and the BART surrogate accuracy.

free parameters (2)
  • BART hyperparameters (e.g., number of trees, SoftBART regularization defaults) = not reported
    The manuscript does not specify the exact BART prior/hyperparameter settings used for training λ; these choices affect prediction quality and reproducibility.
  • Training set size for λ simulation = 40 (Section 3), 60 (Section 4)
    The Latin hypercube training set size is chosen by the authors; BART prediction accuracy and the cost of training depend on this.
assumptions (6)
  • standard math Bernstein-von Mises theorem: the posterior distribution of ψ is asymptotically normal with mean ψ̂ and variance λ̂/n (Eq. 5).
    Invoked in Section 2.2 to derive the normal approximation of the decision statistic τ.
  • standard math Asymptotic normality of the MLE: sqrt(n)(ψ̂_n - ψ*) converges to N(0, λ*) (Eq. 6).
    Used jointly with BvM to express the probit-transformed τ as a linear function of a standard normal.
  • standard math Joint canonical distribution of group sequential MLEs (incremental information structure) leading to Cov(ε_j, ε_k) = sqrt(n_j/n_k) (Eqs. 11-13).
    Needed to extend the approximation to group sequential designs.
  • domain assumption Regularity conditions: compact parameter space, continuity, identifiability, unique MLE.
    Stated in Section 2.2 as required for the BvM and MLE asymptotics to hold.
  • domain assumption For the sample sizes considered, λ̂ ≈ λ* = λ, so the probit-transformed decision statistic has unit variance and mean sqrt(n)(ψ*-ψ0)/λ.
    Stated in Section 2.2 ('For large enough sample sizes'); this is the key finite-sample assumption, acknowledged as potentially poor for small n.
  • domain assumption BART (Eq. 7) accurately predicts λ(θ) across the design-prior region Θd from a finite Latin hypercube training set.
    The entire method hinges on BART interpolation of λ; the paper validates this with LOOCV but provides no theoretical guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bayesian Design of Experiments in the Presence of Nuisance Parameters." pith.science (2026). https://pith.science/paper/DFVVFAUM

@misc{pith2026250803948,
  author       = {Pith},
  title        = {Pith review of: Bayesian Design of Experiments in the Presence of Nuisance Parameters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DFVVFAUM}},
  note         = {Machine review of arXiv:2508.03948}
}
read the original abstract

Design of experiments has traditionally relied on the frequentist hypothesis testing framework where the optimal size of the experiment is specified as the minimum sample size that guarantees a required level of power. Sample size determination may be performed analytically when the test statistic has a known asymptotic sampling distribution and, therefore, the power function is available in analytic form. Bayesian methods have gained popularity in all stages of discovery, namely, design, analysis and decision making. Bayesian decision procedures rely on posterior summaries whose sampling distributions are commonly estimated via Monte Carlo simulations. In the design of scientific studies, the Bayesian approach incorporates uncertainty about the design value(s) instead of conditioning on a single value of the model parameter(s). Accounting for uncertainties in the design value(s) is particularly critical when the model includes nuisance parameters. In this manuscript, we propose methodology that utilizes the large-sample properties of the posterior distribution together with Bayesian additive regression trees (BART) to efficiently obtain the optimal sample size and decision criteria in fixed and adaptive designs. We introduce a fully Bayesian procedure that incorporates the uncertainty associated with the model parameters including the nuisance parameters at the design stage. The proposed approach significantly reduces the computational burden associated with Bayesian design and enables the wide adoption of Bayesian operating characteristics.

Figures

Figures reproduced from arXiv: 2508.03948 by the authors.

Figure 1
Figure 1. (a) LOOCV BART predictions of λ versus simulated values on the log scale and (b) LOOCV BART￾BvM estimates of power versus Monte Carlo estimates of power for the 40 samples across Θd together with 95% credible intervals. the power integrated over the design distribution of the nuisance parameters. These estimates are obtained by plugging λˆ into (8) for a Monte Carlo sample drawn from the design prior and computing t… view at source ↗
Figure 2
Figure 2. The curves are BART-BvM estimates of power integrated over the nuisance parameters versus the conditional main effect of treatment for a range of sample sizes. The dots are the Monte Carlo estimates with N = 500. article. Bayesian analysis model We define the baseline “hazard” of experiencing lesion resolution as a piece￾wise constant function and let s be the time since randomization in days. We allow the baseline … view at source ↗
Figure 3
Figure 3. BART-BvM curves for cumulative stopping probabilities integrated over the nuisance parameters versus the log hazard ratio for designs D1 and D2. worse with respect to these criteria, the choice should ultimately be made considering the feasibility of an additional analysis. It is also straightforward to formally incorporate the cost associated with each analysis into the cost function. We investigated the quality of… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: LOOCV BART predictions of λ versus simulated values on the log scale together with 95% credible intervals. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Deviation between BART-BvM estimates and Monte Carlo estimates of cumulative power through the (a) first, n = 350, (b) second, n = 500, and (c) last, n = 700, analyses of D1. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 24 canonical work pages

  1. [1]

    Basu, D. (1977). On the elimination of nuisance parameters. Journal of the American Statistical Association , 72(358):355--366

    work page 1977

  2. [2]

    and Mortera, J

    Berger, J. and Mortera, J. (1994). Robust bayesian hypothesis testing in the presence of nuisance parameters. Entropy (Basel) , 40(2):357--373

    work page 1994

  3. [3]

    O., Liseo, B., and Wolpert, R

    Berger, J. O., Liseo, B., and Wolpert, R. L. (1988). The Likelihood Principle . Institute of Mathematical Statistics

    work page 1988

  4. [4]

    O., Liseo, B., and Wolpert, R

    Berger, J. O., Liseo, B., and Wolpert, R. L. (1999). Integrated likelihood methods for eliminating nuisance parameters . Statistical Science , 14(1):1 -- 28

    work page 1999

  5. [5]

    M., Carlin, B

    Berry, S. M., Carlin, B. P., Lee, J. J., and Muller, P. (2010). Bayesian adaptive methods for clinical trials . CRC press

    work page 2010

  6. [6]

    Best, N., Maxine, A., Beat, N., Gaëlle, S.-H., and Wandel, S. (2024). Beyond the Classical Type I Error : Bayesian Metrics for Bayesian Designs Using Informative Priors . Statistics in Biopharmaceutical Research , 0(0):1--14. Publisher: ASA Website \_eprint: https://doi.org/10.1080/19466315.2024.2342817

    arXiv 2024

  7. [7]

    Calderazzo, S., Wiesenfarth, M., and Kopp-Schneider, A. (2020). A decision-theoretic approach to bayesian clinical trial design and evaluation of robustness to prior-data conflict. Biostatistics , 23(1):328–44

    work page 2020

  8. [8]

    A., George, E

    Chipman, H. A., George, E. I., and McCulloch, R. E. (2010). Bart: Bayesian additive regression trees. The Annals of Applied Statistics , 4(1):266--298

    2010

Show all 31 references
  1. [9]

    Chuang-Stein, C. (2006). Sample size and the probability of a successful trial. Pharmaceutical Statistics , 5(4):305--309. \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/pst.232

    2006 doi

  2. [10]

    Flórez Rivera, A., Esteves, L., Fossaluza, V., and de Bragança Pereira, C. (2024). On the nuisance parameter elimination principle in hypothesis testing. Entropy (Basel) , 26(2):117

    2024

  3. [11]

    Golchi, S. (2022). Estimating design operating characteristics in B ayesian adaptive clinical trials. Canadian Journal of Statistics , 50(2):417--436

    2022

  4. [12]

    and Willard, J

    Golchi, S. and Willard, J. J. (2024). Estimating the sampling distribution of posterior decision summaries in B ayesian clinical trials. Biometrical Journal , 66(8):e70002

    2024

  5. [13]

    and De Santis, F

    Gubbiotti, S. and De Santis, F. (2011). A B ayesian method for the choice of the sample size in equivalence trials. Australian & New Zealand Journal of Statistics , 53(4):443--460

    2011

  6. [14]

    and Golchi, S

    Hagar, L. and Golchi, S. (2025). Design of B ayesian clinical trials with clustered data and multiple endpoints. arXiv preprint arXiv:2501.13218

    2025

  7. [15]

    Hagar, L., Golchi, S., and Klein, M. B. (2025). Sequential design with posterior and posterior predictive probabilities. arXiv preprint arXiv:2504.00856

    2025 arXiv

  8. [16]

    and Stevens, N

    Hagar, L. and Stevens, N. T. (2025). An economical approach to design posterior analyses. Journal of the American Statistical Association , page doi.org/10.1080/01621459.2025.2476221

    2025

  9. [17]

    and Turnbull, B

    Jennison, C. and Turnbull, B. W. (1999). Group Sequential Methods with Applications to Clinical Trials . CRC Press

    1999

  10. [18]

    Kalbfleisch, J. D. and Sprott, D. A. (1970). Application of likelihood methods to models involving large numbers of parameters. Brazilian Journal of Probability and Statistics , 32(2):175--194

    1970

  11. [19]

    Klein, M. (2024). Tecovirimat in non-hospitalized patients with monkeypox (platinum-can). https://clinicaltrials.gov/study/NCT05534165

    2024

  12. [20]

    Lehmann, E. L. and Casella, G. (1998). Theory of Point Estimation . Springer Science & Business Media

    1998

  13. [21]

    Linero, A. R. and Yang, Y. (2018). Bayesian regression tree ensembles that adapt to smoothness and sparsity. Journal of the Royal Statistical Society Series B: Statistical Methodology , 80(5):1087--1110

    2018

  14. [22]

    D., Beckman, R

    McKay, M. D., Beckman, R. J., and Conover, W. J. (1979). A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics , 21(2):239--245

    1979

  15. [23]

    W., and Campbell, M

    O'Hagan, A., Stevens, J. W., and Campbell, M. J. (2005). Assurance in clinical trial design. Pharmaceutical Statistics , 4(3):187--201. \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/pst.175

    2005 doi

  16. [24]

    and Pereira, C

    Pericchi, L. and Pereira, C. (2016a). Adaptative significance levels using optimal decision rules: Balancing by weighting the error probabilities. Brazilian Journal of Probability and Statistics , 30(1):70--90

  17. [25]

    and Pereira, C

    Pericchi, L. and Pereira, C. (2016b). Adaptative significance levels using optimal decision rules: balancing by weighting the error probabilities. Brazilian Journal of Probability and Statistics , 30:70–90

  18. [26]

    and Oakley, J

    Ren, S. and Oakley, J. E. (2014). Assurance calculations for planning clinical trials with time-to-event outcomes. Statistics in Medicine , 33(1):31--45

    2014

  19. [27]

    J., Abrams, K

    Spiegelhalter, D. J., Abrams, K. R., and Myles, J. P. (2004). Bayesian Approaches to Clinical Trials and Healthcare Evaluation , volume 13. John Wiley & Sons

    2004

  20. [28]

    Spiegelhalter, D. J. and Freedman, L. S. (1986). A predictive approach to selecting the size of a clinical trial, based on subjective clinical opinion. Statistics in Medicine , 5(1):1--13. \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.4780050103

    1986 doi

  21. [29]

    J., Freedman, L

    Spiegelhalter, D. J., Freedman, L. S., and Parmar, M. K. (1994). Bayesian approaches to randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society) , 157(3):357--387

    1994

  22. [30]

    van der Vaart, A. W. (1998). Asymptotic Statistics . Cambridge University Press

    1998

  23. [31]

    and Gelfand, A

    Wang, F. and Gelfand, A. E. (2002). A simulation-based approach to B ayesian sample size determination for performance under a given model and for separating models. Statistical Science , 17(2):193--208

    2002

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.