pith. sign in

arxiv: 1907.04185 · v1 · pith:ILDYVDNYnew · submitted 2019-07-09 · 📊 stat.AP

Predictively Consistent Prior Effective Sample Sizes

Beat Neuenschwander , Sebastian Weber , Heinz Schmidli , Anthony O'Hagan This is my paper

Pith reviewed 2026-05-24 23:59 UTC · model grok-4.3

classification 📊 stat.AP
keywords effective sample sizeprior ESSpredictive consistencyBayesian statisticsclinical trial designhistorical datainformation ratiosubgroup analysis
0
0 comments X

The pith

The expected local-information-ratio ESS is predictively consistent, so that the expected posterior ESS after N observations equals the prior ESS plus N.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current methods for calculating the effective sample size of a prior fail a basic predictive consistency check. This check requires that the expected ESS of the posterior predictive after N observations equals the prior ESS plus N. The paper introduces the expected local-information-ratio ESS and shows that it meets the consistency requirement. The new measure adjusts the values given by earlier methods in non-conjugate settings such as normal data with a Student-t prior and exponential data with a generalized gamma prior. Accurate prior ESS values help in designing clinical trials that borrow information from historical controls without mis-weighting it.

Core claim

The expected local-information-ratio ESS is introduced and shown to be predictively consistent, requiring the expected posterior-predictive ESS for a sample of size N to be the sum of the prior ESS and N. It corrects the ESS of current methods, as shown for normally distributed data with a heavy-tailed Student-t prior and exponential data with a generalized Gamma prior. Finally, two applications are discussed: the prior ESS for the control group derived from historical data, and the posterior ESS for hierarchical subgroup analyses.

What carries the argument

The expected local-information-ratio ESS, which defines the prior's contribution through the expected local information ratio between prior and posterior predictive.

If this is right

  • Prior ESS for a control group derived from historical data can be obtained consistently.
  • Posterior ESS for hierarchical subgroup analyses can be calculated consistently.
  • ESS values are corrected for normally distributed data with a heavy-tailed Student-t prior.
  • ESS values are corrected for exponential data with a generalized Gamma prior.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Adopting the predictive consistency criterion might require re-examination of other existing ESS methods beyond those discussed.
  • The approach could be tested on additional prior-likelihood pairs to check whether the consistency property holds more generally.
  • Reliable ESS values under this criterion would alter how much historical information is effectively used when planning sample sizes.

Load-bearing premise

Predictive consistency, where the expected posterior-predictive ESS after N observations equals prior ESS plus N, is the appropriate criterion against which all ESS methods should be judged.

What would settle it

A calculation for a specific prior and likelihood where the expected local-information-ratio ESS after N observations does not equal prior ESS plus N would show the claimed predictive consistency fails.

Figures

Figures reproduced from arXiv: 1907.04185 by Anthony O'Hagan, Beat Neuenschwander, Heinz Schmidli, Sebastian Weber.

Figure 1
Figure 1. Figure 1: Median and 95%-intervals for event rates of historical ankylosing spondylitis trials and MAP [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗
read the original abstract

Determining the sample size of an experiment can be challenging, even more so when incorporating external information via a prior distribution. Such information is increasingly used to reduce the size of the control group in randomized clinical trials. Knowing the amount of prior information, expressed as an equivalent prior effective sample size (ESS), clearly facilitates trial designs. Various methods to obtain a prior's ESS have been proposed recently. They have been justified by the fact that they give the standard ESS for one-parameter exponential families. However, despite being based on similar information-based metrics, they may lead to surprisingly different ESS for non-conjugate settings, which complicates many designs with prior information. We show that current methods fail a basic predictive consistency criterion, which requires the expected posterior-predictive ESS for a sample of size $N$ to be the sum of the prior ESS and $N$. The expected local-information-ratio ESS is introduced and shown to be predictively consistent. It corrects the ESS of current methods, as shown for normally distributed data with a heavy-tailed Student-t prior and exponential data with a generalized Gamma prior. Finally, two applications are discussed: the prior ESS for the control group derived from historical data, and the posterior ESS for hierarchical subgroup analyses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that existing prior effective sample size (ESS) methods fail a predictive consistency criterion requiring that the expected posterior-predictive ESS after N observations equals prior ESS plus N. It introduces an expected local-information-ratio ESS that satisfies the criterion by construction, shows that it corrects the numerical values produced by current methods in two non-conjugate examples (normal data with Student-t prior; exponential data with generalized gamma prior), and illustrates use in clinical-trial design with historical controls and in hierarchical subgroup analyses.

Significance. If the central derivation holds, the work supplies a coherent, update-consistent definition of prior ESS that is directly usable in sample-size calculations for experiments incorporating external information. The explicit corrections for the two non-conjugate cases and the applications to trial design constitute concrete, falsifiable contributions.

major comments (2)
  1. [§3] §3, Eq. (7)–(9): the manuscript states that the new ESS recovers the standard values for conjugate exponential-family cases, yet provides no explicit verification (e.g., for a normal–normal or gamma–gamma pair); without this check the claim that the method “corrects” rather than supplants existing practice remains unanchored.
  2. [§5.2] §5.2: the predictive-consistency property is shown to hold by construction for the local-information-ratio definition, but the paper does not examine whether the resulting ESS is invariant under reparameterization of the sampling model; this invariance is load-bearing for the hierarchical-subgroup application in §6.
minor comments (2)
  1. [Table 2] Table 2: the reported ESS values for the Student-t prior should include the Monte-Carlo standard error or the number of replications used to obtain the expectation.
  2. The notation for the local information ratio is introduced without an explicit statement of the measure used to define “information” (Kullback–Leibler, Fisher, etc.); a one-sentence clarification would remove ambiguity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful review and the recommendation of minor revision. The comments identify two areas where additional verification would strengthen the manuscript. We address each below and will make the corresponding revisions.

read point-by-point responses

  1. Referee: §3, Eq. (7)–(9): the manuscript states that the new ESS recovers the standard values for conjugate exponential-family cases, yet provides no explicit verification (e.g., for a normal–normal or gamma–gamma pair); without this check the claim that the method “corrects” rather than supplants existing practice remains unanchored.

    Authors: We agree that an explicit verification is needed to anchor the claim. In the revised manuscript we will add a short calculation in §3 for the normal–normal conjugate case, confirming that the expected local-information-ratio ESS equals the conventional value n₀. A parallel check for the gamma–gamma pair can be included if space allows. revision: yes

  2. Referee: §5.2: the predictive-consistency property is shown to hold by construction for the local-information-ratio definition, but the paper does not examine whether the resulting ESS is invariant under reparameterization of the sampling model; this invariance is load-bearing for the hierarchical-subgroup application in §6.

    Authors: We acknowledge the omission. Because the local information ratio is constructed from the Fisher information, which transforms by the square of the Jacobian under reparameterization, the ESS is expected to be invariant. We will add a brief argument or numerical check in §5.2 to confirm this property and thereby support the §6 application. revision: yes

Circularity Check

0 steps flagged

No significant circularity; new ESS defined independently and shown to satisfy stated criterion

full rationale

The paper introduces the expected local-information-ratio ESS via an independent definition based on local information ratios and then demonstrates (rather than assumes by construction) that it satisfies the predictive consistency criterion requiring E[posterior-predictive ESS after N observations] = prior ESS + N. Current methods are shown to fail this criterion on specific examples, but the new quantity's success is presented as a derived property. No equations reduce the result to a fitted input, self-citation chain, or redefinition of the target quantity itself. The central derivation remains self-contained once the predictive consistency criterion is granted as the evaluation standard.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the predictive consistency criterion as the evaluation standard and on the local-information-ratio construction as the definition of the new ESS; both are introduced in the paper rather than derived from external benchmarks.

axioms (1)
  • domain assumption Predictive consistency criterion: expected posterior-predictive ESS after observing a sample of size N equals prior ESS plus N
    This is the basic requirement used to reject current methods and to motivate the new definition.
invented entities (1)
  • expected local-information-ratio ESS no independent evidence
    purpose: A new scalar measure of prior information that satisfies predictive consistency for non-conjugate priors
    Introduced in the paper as the quantity that corrects existing ESS calculations; no independent evidence outside the definition is provided.

pith-pipeline@v0.9.0 · 5747 in / 1330 out tokens · 22904 ms · 2026-05-24T23:59:41.236365+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

51 extracted references · 51 canonical work pages

  1. [1]

    May 19, 2015

    21st Century Cures Act. May 19, 2015. http://docs.house.gov/meetings/IF/IF00/ 20150519/103516/BILLS-1146ih.pdf

    work page 2015

  2. [2]

    Anti-interleukin-17A monoclonal antibody secukinumab in ankylosing spondylitis: a randomized, double-blind, placebo- controlled trial

    Baeten D, Baraliakos X, Braun J, Sieper J, Emery P, van der Heijde D, McInnes I, van Laar J, Landewe R, Wordsworth P, Wollenhaupt J, Kellner H, Paramarta J, Wei J, Brachat A, Bek S, Laurent D, Li Y, Wang Y, Bertolino A, Gsteiger S, Wright AM and Hueber W. Anti-interleukin-17A monoclonal antibody secukinumab in ankylosing spondylitis: a randomized, double-...

    work page 2013

  3. [3]

    Power prior distributions for regression models

    Chen MH and Ibrahim JG. Power prior distributions for regression models. Statistical Science , 15(1):46–60, 2000

    work page 2000

  4. [4]

    Phase II multicenter trial of imatinib in 10 histologic subtypes of sarcoma using a Bayesian hierarchical statistical model

    Chugh R, Wathen K, Maki R, Benjamin R, Patel R, Myers P, Priebat D, Reinke D, Thomas D, Keohan M, Samuels B, and Baker L. Phase II multicenter trial of imatinib in 10 histologic subtypes of sarcoma using a Bayesian hierarchical statistical model. Journal of Clinical Oncology 2009; 27(19): 3148–3153

    work page 2009

  5. [5]

    Approximating priors by mixtures of natural conjugate priors

    Dallal S, Hall W. Approximating priors by mixtures of natural conjugate priors. Journal of the Royal Statistical Society Serires B 1983; 45: 278-286

    work page 1983

  6. [6]

    Longitudinal model-based meta-analysis in rheumatoid arthritis: an application toward model-based drug development

    Demin I, Hamren B, Luttringer O, Pillai G, and Jung T. Longitudinal model-based meta-analysis in rheumatoid arthritis: an application toward model-based drug development. Clinical Pharmacology and Therapeutics 2012; 92(3): 251–261

    work page 2012

  7. [7]

    Diaconis and D

    P. Diaconis and D. Ylvisaker. Quantifying prior opinion. Bayesian Statistics (Proceedings of the Second Valencia International Meeting) 1984; 2:133–148

    work page 1984

  8. [8]

    NICE DSU Technical Support Document 2: A Generalised Linear Modelling Framework for Pairwise and Network Meta-Analysis of Randomised Controlled Trials, 2011

    Dias S, Welton N, Sutton A, and Ades A. NICE DSU Technical Support Document 2: A Generalised Linear Modelling Framework for Pairwise and Network Meta-Analysis of Randomised Controlled Trials, 2011

    work page 2011

  9. [9]

    Innovative Medicines initiative 2: Europe’s fast track to better medicines 2014

    European Comission. Innovative Medicines initiative 2: Europe’s fast track to better medicines 2014. 11

    work page 2014

  10. [10]

    Reflection paper on the use of extrapolation in the development of medicines for paediatrics 2018

    European Medicines Agency. Reflection paper on the use of extrapolation in the development of medicines for paediatrics 2018. https://www.ema.europa.eu/documents/scientific-guideline/adopted- reflection-paper-use-extrapolation-development-medicines-paediatrics-revision-1 en.pdf [Online; ac- cessed 20-December 2018]

    work page 2018

  11. [11]

    Concept paper on extrapolation of efficacy and safety in medicine development 2013

    European Medicines Agency. Concept paper on extrapolation of efficacy and safety in medicine development 2013. https://www.ema.europa.eu/documents/scientific-guideline/concept- paper-extrapolation-efficacy-safety-medicine-development en.pdf [Online; accessed 20-December 2018]

    work page 2013

  12. [12]

    Bayesian design of proof-of- concept trials

    Fisch R, Jones I, Jones J, Kerman J, Rosenkranz GK and Schmidli H. Bayesian design of proof-of- concept trials. Therapeutic Innovation & Regulatory Science 2015; 49(1): 155-162

    work page 2015

  13. [13]

    Innovation/Stagnation - Challenge and Opportunity on the Critical Path to New Medical Products 2004

    Food and Drug Administration (FDA). Innovation/Stagnation - Challenge and Opportunity on the Critical Path to New Medical Products 2004. [Online; accessed 6-September-2015]

    work page 2004

  14. [14]

    Guidance for Industry: Non-Inferiority Clinical Trials 2016

    Food and Drug Administration (FDA). Guidance for Industry: Non-Inferiority Clinical Trials 2016. https://www.fda.gov/downloads/Drugs/Guidances/UCM202140.pdf [Online; accessed 20-December- 2018]

    work page 2016

  15. [15]

    Guidance for the use of Bayesian statistics in medical device clinical trials 2010

    Food and Drug Administration (FDA). Guidance for the use of Bayesian statistics in medical device clinical trials 2010. www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/ GuidanceDocuments/ ucm071072.htm (accessed Jan 2014)

    work page 2010

  16. [16]

    Paving the Way for Personalized Medicine: FDA’s Role in a New Era of Medical Product Development 2013

    Food and Drug Administration (FDA). Paving the Way for Personalized Medicine: FDA’s Role in a New Era of Medical Product Development 2013

    work page 2013

  17. [17]

    Guidance for Industry and Food and Drug Administration Staff: Leveraging Existing Clinical Data for Extrapolation to Pediatric Uses of Medical Devices 2016

    Food and Drug Administration (FDA). Guidance for Industry and Food and Drug Administration Staff: Leveraging Existing Clinical Data for Extrapolation to Pediatric Uses of Medical Devices 2016. https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm444591.pdf [Online; accessed 20-December-2018]

    work page 2016

  18. [18]

    Framework for FDA’s Real-World Evidence Program 2018

    Food and Drug Administration (FDA). Framework for FDA’s Real-World Evidence Program 2018

    work page 2018

  19. [19]

    Prior distributions for variance parameters in hierarchical models

    Gelman A. Prior distributions for variance parameters in hierarchical models. Bayesian Analysis 2006; 1(3): 515–534

    work page 2006

  20. [20]

    A Bayesian approach to randomized controlled trials in children utilizing information from adults: the case of Guillain-Barre syndrome

    Goodman S, Sladky J. A Bayesian approach to randomized controlled trials in children utilizing information from adults: the case of Guillain-Barre syndrome. Clinical Trials 2005; 2(10): 305–310

    work page 2005

  21. [21]

    Borrowing strength with nonexchage- able priors over subpopulations

    Leon-Novelo LG, Bekele BN, M¨ uller P, Quintana F, Wathen K. Borrowing strength with nonexchage- able priors over subpopulations. Biometrics 2012; 68:550-558

    work page 2012

  22. [22]

    A closer look at combining data among a small number of binomial experiments

    Malec D. A closer look at combining data among a small number of binomial experiments. Statistics in Medicine 2001; 12: 1811–1824

    work page 2001

  23. [23]

    Determining the effective sample size of a parametric prior

    Morita S, Thall PF and M¨ uller P. Determining the effective sample size of a parametric prior. Bio- metrics 2008; 64: 595–602

    work page 2008

  24. [24]

    Prior effective sample size in conditionally independent hierarchical models

    Morita S, Thall PF, and M¨ uller P. Prior effective sample size in conditionally independent hierarchical models. Bayesian Analysis 2012; 7(3):561–614. 12

    work page 2012

  25. [25]

    Modeling and simulation in clinical drug development

    Nedelman J, Bretz F, Fisch R, Georgieva A, Hsu C, Kahn J, Kawai L, Lowe P, Maca J, Pinheiro J, Rossini A, Schmidli H, Steimer J, and Yu J. Modeling and simulation in clinical drug development. In: Pharmaceutical Sciences Encyclopedia 2010; 43: 1–29

    work page 2010

  26. [26]

    A note on the power prior

    Neuenschwander B, Branson M, and Spiegelhalter DJ. A note on the power prior. Statistics in Medicine, 28:3562–3566, 2009

    work page 2009

  27. [27]

    Summarizing historical in- formation on controls in clinical trials

    Neuenschwander B, Capkun-Niggli G, Branson M, and Spiegelhalter DJ. Summarizing historical in- formation on controls in clinical trials. Clinical Trials 2010; 7: 5–18

    work page 2010

  28. [28]

    Robust exchangeability designs for early phase clinical trials with multiple strata

    Neuenschwander B, Wandel S, Roychoudhury S, and Bailey S. Robust exchangeability designs for early phase clinical trials with multiple strata. Pharmaceutical Statistics 2015; 15: 123-134

    work page 2015

  29. [29]

    On the use of co-data in clinical trials.Statistics in Biopharmaceutical Research 2016; 8(3): 1-10

    Neuenschwander B, Roychoudhury S, and Schmidli H. On the use of co-data in clinical trials.Statistics in Biopharmaceutical Research 2016; 8(3): 1-10

    work page 2016

  30. [30]

    On outlier rejection phenomena in Bayes inference

    O’Hagan, A. On outlier rejection phenomena in Bayes inference. Journal of the Royal Statistical Society, Series B 1979; 41, 358–367

    work page 1979

  31. [31]

    Bayesian heavy-tailed models and conflict resolution: a review

    O’Hagan A and Pericchi L. Bayesian heavy-tailed models and conflict resolution: a review. Brazilian Journal of Probability and Statistics 2012; 26: 372–401

    work page 2012

  32. [32]

    Experience with reviewing Bayesian medical device trials

    Pennello G and Thompson L. Experience with reviewing Bayesian medical device trials. Journal of Biopharmaceutical Statistics 2008; 18(1): 81–115

    work page 2008

  33. [33]

    The combination of randomized and historical controls in clinical trials.Journal of Chronic Diseases, 29(3):175–188, 1976

    Pocock, SJ. The combination of randomized and historical controls in clinical trials.Journal of Chronic Diseases, 29(3):175–188, 1976

    work page 1976

  34. [34]

    On the Half-Cauchy prior for a global scale parameter

    Polson NG and Scott JG. On the Half-Cauchy prior for a global scale parameter. Bayesian Analysis 2012; 7: 887–902

    work page 2012

  35. [35]

    SAS user guide: Statistics

    SAS Institute. SAS user guide: Statistics. The FMM procedure . Cary, NC: SAS Institute Inc., 2014

    work page 2014

  36. [36]

    Robust meta-analytic-predictive priors in clinical trials with historical control information.Biometrics 2014; 70: 1023–32

    Schmidli H, Gsteiger S, Roychoudhury S, O’Hagan O, Spiegelhalter DJ and Neuenschwander B. Robust meta-analytic-predictive priors in clinical trials with historical control information.Biometrics 2014; 70: 1023–32

    work page 2014

  37. [37]

    Bayesian Approaches to Clinical trials and Health-Care Evaluation

    Spiegelhalter DJ, Abrams KR and Myles JP. Bayesian Approaches to Clinical trials and Health-Care Evaluation. 2004; Chichester: Wiley

    work page 2004

  38. [38]

    Effective sample size for computing prior hyperparameters in Bayesian phase I-II dose-finding

    Thall PF, Herrick RC, Nguyen HQ, Venier JJ, and Norris JC. Effective sample size for computing prior hyperparameters in Bayesian phase I-II dose-finding. Clinical Trials 2014; 11(6):657–666

    work page 2014

  39. [39]

    Hierarchical Bayesian ap- proaches to phase II trials in diseases with multiple subtypes

    Thall PF, Wathen J, Bekele B, Champlin R, Baker L, and Benjamin R. Hierarchical Bayesian ap- proaches to phase II trials in diseases with multiple subtypes. Journal of Clinical Oncology 2003; 22: 763–780

    work page 2003

  40. [40]

    S. Weber. RBesT: R Bayesian Evidence Synthesis Tools , 2019. R package version 1.4-0

    work page 2019

  41. [41]

    Applying meta-analytic predictive priors with the R Bayesian evidence synthesis tools

    Weber S, Li Y, Seaman J III, Kakizume T, and Schmidli H. Applying meta-analytic predictive priors with the R Bayesian evidence synthesis tools. pre-print 2019. arXiv:1907.00603 13 Supporting Information The on-line material is available at Biometrics website on Wiley Online Library. It contains ESSELIR functions for mixtures of normal and Beta distributio...

    work page arXiv 2019

  42. [42]

    Angiosarcoma 2/15 (13) 65 60 50 36

    work page

  43. [43]

    Ewing 0/13 (0) 57 46 36 24

    work page

  44. [44]

    Fibrosarcoma 1/12 (8) 60 54 44 31

    work page

  45. [45]

    Leiomyosarcoma 6/28 (21) 78 72 64 47

    work page

  46. [46]

    Liposarcoma 7/29 (24) 76 68 59 43

    work page

  47. [47]

    MFH 3/29 (10) 74 69 59 46

    work page

  48. [48]

    Osteosarcoma 5/26 (19) 79 73 64 47

    work page

  49. [49]

    MPNST 1/5 (20) 57 48 39 23

    work page

  50. [50]

    Rhabdomysarcoma 0/2 (0) 54 43 33 18

    work page

  51. [51]

    Synovial 2/20 (15) 72 66 57 42 18 Figure 1: Median and 95%-intervals for event rates of historical ankylosing spondylitis trials and MAP event rate for new trial (left panel), and MAP prior density (solid line) with two-component Beta mixture approximation (dashed line) (right panel). 19

    work page