Pith. sign in

REVIEW 2 major objections 4 minor 27 references

A decomposition of Fisher's information to inform sample size for developing or updating fair and precise clinical prediction models -- Part 3: continuous outcomes

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper derives a closed-form sample size formula that targets the width of confidence intervals around individual predicted continuous outcomes, and shows that meeting it may require far more participants than existing…

desk verdict A sound and useful extension of the same group's Fisher-information sample-size framework to continuous outcomes, with a real but fixable flaw in the way the 'ensure all individuals' target is defined. read the letter →

arxiv 2507.23548 v1 pith:4V2EUTEW submitted 2025-07-31 stat.ME

Rebecca Whittle , Richard D Riley , Lucinda Archer , Gary S Collins , Amardeep Legha , Kym IE Snell , Joie Ensor This is my paper
classification stat.ME MSC 62J0562F1262P10
keywords clinicalpredictionmodelssamplesizeconfidenceintervalwidthFisherinformationmatrixintervalsfairnesscontinuousoutcomeslinearregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a formula for the sample size needed to develop or update a linear prediction model with a continuous outcome so that the confidence interval around each individual's predicted value is no wider than a target. The calculation decomposes the variance of the estimated regression coefficients into a Fisher unit information matrix, built from the predictor distribution and the residual variance (or R-squared), divided by the sample size. That matrix lets researchers either check whether an existing dataset is large enough or solve directly for the required sample size once they pick a target width. In the worked example for forced expiratory volume in children, the previous criterion-based minimum of 237 participants corresponds to a mean interval width of 0.206 litres, while guaranteeing every width no wider than 0.3 requires 694 participants.

What carries the argument

The load-bearing object is the decomposition $\operatorname{var}(\hat{\boldsymbol{\beta}}) = \frac{1}{n} \mathbf{I}(\boldsymbol{\beta})^{-1}$ of the linear-regression coefficient variance into a Fisher unit information matrix $\mathbf{I}(\boldsymbol{\beta}) = \frac{\mathbf{X}'\mathbf{X}}{\sigma^2}$ and the sample size $n$. The quadratic form $\mathbf{x}_{new} \mathbf{I}(\boldsymbol{\beta})^{-1} \mathbf{x}_{new}'$ converts that matrix into the variance of a prediction at a specific predictor profile, and Equation (12) inverts it to solve for the required sample size at a target confidence interval width; prediction-interval versions add $\sigma^2$ inside the square root, which sets a floor on the width no matter how large $n$ becomes.

What would settle it

Simulate many datasets at the recommended sample size from the same predictor distribution but with slightly heavier tails, refit the linear model, and compare the largest observed confidence interval width to the target; if the width exceeds the target in more than a small fraction of simulations, the claim that the computed sample size guarantees the chosen precision fails.

Watch

Extended reading notes

Core claim

The central claim is that, for a linear regression predicting a continuous outcome, the epistemic uncertainty of a prediction for a new person with predictor vector $\mathbf{x}_{new}$ is $\operatorname{var}(\hat{y}_{new}) = \frac{1}{n} \mathbf{x}_{new} \mathbf{I}(\boldsymbol{\beta})^{-1} \mathbf{x}_{new}'$, where $\mathbf{I}(\boldsymbol{\beta})$ is the Fisher unit information matrix estimated from a pilot or simulated dataset. Rearranged, the sample size that forces every predicted value's 95% confidence interval within a chosen width $W$ is $n = \frac{\mathbf{x}_{new} \mathbf{I}(\boldsymbol{\beta})^{-1} \mathbf{x}_{new}'}{(W/(2z_{\alpha/2}))^2}$. The paper further claims that the same machinery supports fairness checks by comparing interval widths across subgroups, extends to prediction intervals where the residual variance $\sigma^2$ imposes a lower bound on achievable width, and works from published summary statistics when individual data are unavailable, by reconstructing the predictor covariance from univariable and multivariable regression coefficients.

Load-bearing premise

The calculation assumes the predictor distribution and residual variance (or R-squared) fed into the Fisher unit information matrix truly represent the target population; if the pilot or simulated data miss the tail, the maximum required sample size is biased, as shown by the authors excluding the most extreme 0.001% of simulated individuals to stabilise their estimate.

Editorial extensions

If this is right

  • Researchers can use an existing or pilot dataset to decide whether it already provides clinically acceptable prediction precision, without fitting the final model.
  • If only summary statistics from published models are available, the predictor covariance can be reconstructed from univariable and multivariable coefficients and used to simulate the joint distribution needed for the matrix.
  • Applying the calculation separately within subgroups reveals whether one group, such as women or men, would systematically receive wider intervals, informing fairness assessments.
  • For prediction intervals, no sample size can drive the width below roughly $2z_{\alpha/2}\sigma$, so if the residual variance is large the effort should shift to finding stronger predictors rather than collecting more data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same unit-information decomposition could be reused to compare candidate predictor sets before data collection: adding a weak predictor worsens the quadratic form and raises the required sample size, so the formula gives a direct cost of model complexity.
  • The sensitivity of the maximum sample size to extreme predictor profiles suggests that a robust planning rule would use a trimmed quantile of the required-$n$ distribution (as the paper does implicitly) and would validate the chosen quantile against the pilot data rather than the full sample.
  • For updating an existing model, the formula could be inverted to ask how much new data is needed to shrink the model's prediction intervals to a target, which the paper mentions but does not develop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper extends a line of work on sample size calculations for clinical prediction models to continuous outcomes. The proposed method uses the Fisher unit information matrix of a linear regression model, estimated from an existing or simulated dataset, to compute the variance of a predicted outcome for a new individual's predictor profile. From this, the authors derive closed-form expressions for the expected confidence interval width at a given sample size (Equation 10) and for the sample size needed to achieve a target width (Equation 12). They also provide an extension to prediction intervals (Equations 13-15) and illustrate the method with an example predicting forced expiratory volume (FEV) from age, height, and sex, reporting that 694 patients would be needed to ensure all anticipated interval widths are no wider than 0.3 litres. The paper includes a fairness assessment by subgroup and a sensitivity analysis assuming conditional independence of predictors.

Significance. If the proposed methodology is sound, it addresses a genuine gap in sample size guidance for continuous-outcome prediction models: existing criteria target overfitting and parameter estimation, but not the precision of individual predictions. The derivations are transparent and rely on standard linear regression theory, which makes the core formulas easy to verify. The authors also provide practical implementation materials, including R code and a software module, which increases the potential utility of the work. The extension to prediction intervals and the explicit consideration of subgroup-level precision are useful contributions. However, the applied headline claim rests on a maximum-over-individuals calculation that is sensitive to extreme predictor profiles and is therefore not a stable population-level guarantee; this needs to be addressed before the method can be recommended for routine use.

major comments (2)
  1. [Applied Example, Option B and Discussion] The 'all anticipated interval widths ≤ 0.3' target is defined as the maximum of Equation (12) over individuals in the observed or simulated dataset. As the authors acknowledge in the Discussion, this maximum varies substantially with the simulation seed and is driven by a small number of extreme predictor profiles, so much so that they excluded the most extreme 0.001% of simulated individuals to obtain stable estimates. This trimming changes the estimand: the reported n=694 (Scenario 1) and n=733 (Scenario 2) are no longer guarantees for all individuals, and for continuous predictors with unbounded support the required sample size can be unbounded. The paper should either state the target as a quantile (e.g., '99.9% of widths ≤ 0.3'), or justify the trimming as part of a well-defined robustness rule, or present a different summary (e.g., mean or upper quantile) as the primary target. As written, the abstract's claim that the method 'calculates a minimum of 694 patients would be needed to ensure all anticipated interval widths were ≤0.3' is not supported by a stable population-level estimand.
  2. [Step 2, Scenario 2, Equation (2)] The procedure for reconstructing the predictor covariance matrix from univariable and multivariable regression coefficients is underdetermined for more than three predictors. With p predictors, the user knows the variances var(x_j) and the covariances cov(x_j, y) from univariable models, and the p multivariable coefficients supply p equations of the form cov(x_j, y) = sum_k beta_k cov(x_j, x_k). The unknowns are the p(p-1)/2 pairwise covariances among predictors; these can be solved only when p(p-1)/2 ≤ p, i.e., p ≤ 3. The manuscript presents Equation (2) as a general solution, but the notation β_multi^{-1} is not well-defined for a vector, and the system is not solvable for larger p without additional information. The authors should state this limitation explicitly and provide guidance for settings with more than three core predictors, such as obtaining the full covariance matrix from data custodians or using a priori assumptions.
minor comments (4)
  1. [Step 3, Equation (4)] Equation (4) defines I(β) using E(X^T X), but Equation (6) and all subsequent calculations use the observed X^T X from the pilot or simulated dataset. The distinction between the population expectation and the finite-sample estimate should be clarified, and the text should state that the method conditions on the observed predictor distribution as an approximation to the target population.
  2. [Step 4, Equations (9)-(13)] The degrees of freedom for the t-distribution are used inconsistently: Equation (9) uses n−p, while Equation (10) and Equation (13) use n−p−1. In a linear regression with p parameters including the intercept, the residual degrees of freedom are n−p for both confidence intervals and prediction intervals. The authors should standardize the definition of p (number of parameters vs. number of predictors) and use a consistent df throughout, as this will affect the computed widths and sample sizes at small n.
  3. [Step 4, Equation numbering] The equation numbering is inconsistent: the displayed equation after 'CI width =' is labelled (6), but the text refers to it as Equation (10), and a later displayed equation is labelled (7) while the text refers to Equation (11). These labels should be corrected so that cross-references and the final published equations match.
  4. [Discussion] The sentence 'as outlined by Riley at. (12)' contains a typo ('at.' should be 'et al.'), and the software package 'pmstabilityss' is mentioned but not described or referenced; the authors should provide a citation or a brief description of how to access it.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the sample-size formulas follow algebraically from standard linear-regression variance theory, and the paper's applied claims are transparent about their dependence on assumed inputs and simulated-data trimming.

full rationale

The central derivation chain is self-contained. Equation (8), var(y_new) = (1/n) x_new I(beta)^{-1} x_new', is the standard linear-regression variance-of-prediction formula written through Fisher's unit information matrix, and Equation (12) is a direct algebraic rearrangement to solve for n given a target CI width. Nothing in this chain is fitted from the quantity it claims to predict. The paper requires an assumed R^2 or residual variance as an external input; it states, 'For illustrative purposes, we assume an anticipated R^2 value of 0.77, chosen to reflect a plausible level of explained variance... based on previous prediction models for FEV and previous analyses using subsets of the same data.' That is a transparency about the input, not a circular step, because the method does not estimate R^2 from the same data and then present it as a validated prediction. The applied '694 patients' figure is the maximum of Eq. (12) over observed/simulated individuals for a 0.3 width target, so it is by construction the smallest n making all observed widths below target; this is the intended design calculation, not a circular prediction. The paper candidly reports a limitation that 'the maximum required sample size could vary substantially depending on the random seed... To mitigate this, we excluded the most extreme 0.001% of simulated individuals.' That is a robustness caveat about the stability of a maximum-based estimand, not a circularity. Self-citations to the authors' binary and time-to-event companion papers (refs 14-15) are motivational and comparative, not load-bearing; no uniqueness theorem is imported from those papers, and the formulas here are derived rather than asserted by citation. Accordingly, no specific circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities, but it relies on several assumptions about the target population and the pilot data. The main free parameters are the assumed R^2 and the trimming threshold, both of which materially affect the reported sample sizes.

free parameters (2)
  • Anticipated R^2 = 0.77
    Assumed based on previous prediction models for FEV and prior analyses using subsets of the same data; directly influences the Fisher unit information matrix via the residual variance.
  • Extreme-value trimming threshold = 0.001% (top 10 of 1 million simulated individuals)
    Chosen by hand to stabilize the maximum required sample size in the simulated-data scenario; affects the reported scenario 2 result of 733.
assumptions (4)
  • domain assumption The prediction model is a correctly specified linear regression with the identified core predictors and constant residual variance.
    The Fisher information matrix and variance formulas are only valid under these linear regression assumptions; invoked throughout Steps 3 and 4.
  • domain assumption The pilot dataset or simulated data accurately represent the joint distribution of the core predictors in the target population.
    The Fisher unit information matrix is computed from this distribution, so any mismatch directly biases the sample size estimates.
  • standard math The residual variance is related to R^2 by sigma^2 = (1 - R^2) * sum(y_i - ybar)^2 / N, using N rather than the bias-corrected denominator.
    This is an approximation to the population variance; using N instead of n-p-1 is a simplification that may slightly understate uncertainty in small samples.
  • ad hoc to paper When reconstructing the predictor covariance matrix from univariable and multivariable regression coefficients, predictors are either continuous or binary, and the equations are solvable.
    This reconstruction is a key step for Scenario 2 but depends on the reported coefficients being internally consistent and on the normality assumption for simulation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A decomposition of Fisher's information to inform sample size for developing or updating fair and precise clinical prediction models -- Part 3: continuous outcomes." pith.science (2026). https://pith.science/paper/4V2EUTEW

@misc{pith2026250723548,
  author       = {Pith},
  title        = {Pith review of: A decomposition of Fisher's information to inform sample size for developing or updating fair and precise clinical prediction models -- Part 3: continuous outcomes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4V2EUTEW}},
  note         = {Machine review of arXiv:2507.23548}
}
read the original abstract

Clinical prediction models enable healthcare professionals to estimate individual outcomes using patient characteristics. Current sample size guidelines for developing or updating models with continuous outcomes aim to minimise overfitting and ensure accurate estimation of population-level parameters, but do not explicitly address the precision of predictions. This is a critical limitation, as wide confidence intervals around predictions can undermine clinical utility and fairness, particularly if precision varies across subgroups. We propose methodology for calculating the sample size required to ensure precise and fair predictions in models with continuous outcomes. Building on linear regression theory and the Fisher's unit information matrix, our approach calculates how sample size impacts the epistemic (model-based) uncertainty of predictions and allows researchers to either (i) evaluate whether an existing dataset is sufficiently large, or (ii) determine the sample size needed to target a particular confidence interval width around predictions. The method requires real or synthetic data representing the target population. To assess fairness,the approach can evaluate prediction precision across subgroups. Extensions to prediction intervals are included to additionally address aleatoric uncertainty. Our methodology provides a practical framework for examining required sample sizes when developing or updating prediction models with continuous outcomes, focusing on achieving precise and equitable predictions. It supports the development of more reliable and fair models, enhancing their clinical applicability and trustworthiness.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 26 canonical work pages

  1. [1]

    Development and validation of a prognostic model to predict birth weight: individual participant data meta- analysis

    Allotey J, Archer L, Snell KIE, Coomar D, Massé J, Sletner L, et al. Development and validation of a prognostic model to predict birth weight: individual participant data meta- analysis. bmjmed. 2024 Aug;3(1):e000784

    work page 2024

  2. [2]

    Uncertainty of risk estimates from clinical prediction models: rationale, challenges, and approaches

    Riley RD, Collins GS, Kirton L, Snell KI, Ensor J, Whittle R, et al. Uncertainty of risk estimates from clinical prediction models: rationale, challenges, and approaches. BMJ. 2025 Feb 13;e080749

    work page 2025

  3. [3]

    Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating [Internet]

    Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating [Internet]. Cham: Springer International Publishing; 2019 [cited 2025 May 19]. (Statistics for Biology and Health). Available from: http://link.springer.com/10.1007/978-3-030-16399-0

    doi:10.1007/978-3-030-16399-0 2019

  4. [4]

    Sample sizes of prediction model studies in prostate cancer were rarely justified and often insufficient

    Collins SD, Peek N, Riley RD, Martin GP . Sample sizes of prediction model studies in prostate cancer were rarely justified and often insufficient. Journal of Clinical Epidemiology. 2021 May;133:53–60

    work page 2021

  5. [5]

    Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

    Dhiman P , Ma J, Qi C, Bullock G, Sergeant JC, Riley RD, et al. Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review. BMC Med Res Methodol. 2023 Aug 19;23(1):188

    work page 2023

  6. [6]

    Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review

    Dhiman P , Ma J, Andaur Navarro CL, Speich B, Bullock G, Damen JAA, et al. Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review. BMC Med Res Methodol. 2022 Apr 8;22(1):101. 34

    work page 2022

  7. [7]

    Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

    Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020 Apr 7;m1328

    work page 2020

  8. [8]

    Stability of clinical prediction models developed using statistical or machine learning methods

    Riley RD, Collins GS. Stability of clinical prediction models developed using statistical or machine learning methods. Biometrical J. 2023 Dec;65(8):2200302

    work page 2023

Show all 27 references
  1. [9]

    Clinical prediction models and the multiverse of madness

    Riley RD, Pate A, Dhiman P , Archer L, Martin GP , Collins GS. Clinical prediction models and the multiverse of madness. BMC Med. 2023 Dec 18;21(1):502

    2023

  2. [10]

    Importance of sample size on the quality and utility of AI-based prediction models for healthcare

    Riley RD, Ensor J, Snell KIE, Archer L, Whittle R, Dhiman P , et al. Importance of sample size on the quality and utility of AI-based prediction models for healthcare. The Lancet Digital Health. 2025 Jun;100857

    2025

  3. [11]

    TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods

    Collins GS, Moons KGM, Dhiman P , Riley RD, Beam AL, Van Calster B, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024 Apr 16;e078378

    2024

  4. [12]

    Minimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes

    Riley RD, Snell KIE, Ensor J, Burke DL, Harrell FE, Moons KGM, et al. Minimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes. Statistics in Medicine. 2019 Mar 30;38(7):1262–75

    2019

  5. [13]

    Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease

    Pate A, Emsley R, Sperrin M, Martin GP , Van Staa T. Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease. Diagn Progn Res. 2020 Dec;4(1):14

    2020

  6. [14]

    Sample size for developing a prediction model with a binary outcome: targeting precise individual risk 35 estimates to improve clinical decisions and fairness [Internet]

    Riley RD, Collins GS, Whittle R, Archer L, Snell KI, Dhiman P , et al. Sample size for developing a prediction model with a binary outcome: targeting precise individual risk 35 estimates to improve clinical decisions and fairness [Internet]. arXiv; 2024 [cited 2024 Nov 28]. Av...

    2024 arXiv

  7. [15]

    A decomposition of Fisher’s information to inform sample size for developing fair and precise clinical prediction models -- Part 2: time-to-event outcomes [Internet]

    Riley RD, Collins GS, Archer L, Whittle R, Legha A, Kirton L, et al. A decomposition of Fisher’s information to inform sample size for developing fair and precise clinical prediction models -- Part 2: time-to-event outcomes [Internet]. arXiv; 2025 [cited 2025 May 16]. Availabl...

    2025 arXiv

  8. [16]

    Calculating the sample size required for developing a clinical prediction model

    Riley RD, Ensor J, Snell KIE, Harrell FE, Martin GP , Reitsma JB, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020 Mar 18;m441

    2020

  9. [17]

    PMSAMPSIZE: Stata module to calculate the minimum sample size required for developing a multivariable prediction model

    Ensor J. PMSAMPSIZE: Stata module to calculate the minimum sample size required for developing a multivariable prediction model. Boston College Department of Economics; 2018. (Statistical Software Components S458569)

    2018

  10. [18]

    pmsampsize: Sample Size for Development of a Prediction Model

    Ensor J. pmsampsize: Sample Size for Development of a Prediction Model. https://CRANR-projectorg/package=pmsampsize; 2023

    2023

  11. [19]

    Adaptive sample size determination for the development of clinical prediction models

    Christodoulou E, Van Smeden M, Edlinger M, Timmerman D, Wanitschek M, Steyerberg EW , et al. Adaptive sample size determination for the development of clinical prediction models. Diagn Progn Res. 2021 Dec;5(1):6

    2021

  12. [20]

    A guide to systematic review and meta-analysis of prognostic factor studies

    Riley RD, Moons KGM, Snell KIE, Ensor J, Hooft L, Altman DG, et al. A guide to systematic review and meta-analysis of prognostic factor studies. BMJ. 2019 Jan 30;k4597. 36

    2019

  13. [21]

    Predictors of systolic blood pressure response to treadmill exercise: the Lipid Research Clinics Program Prevalence Study

    Criqui MH, Haskell WL, Heiss G, Tyroler HA, Green P , Rubenstein CJ. Predictors of systolic blood pressure response to treadmill exercise: the Lipid Research Clinics Program Prevalence Study. Circulation. 1983 Aug;68(2):225–33

    1983

  14. [22]

    Predictors of high central blood pressure in young with isolated systolic hypertension

    Radchenko G, Torbas O, Sirenko Y . Predictors of high central blood pressure in young with isolated systolic hypertension. VHRM. 2016 Aug;Volume 12:321–8

    2016

  15. [23]

    Ethnicity and blood pressure

    Oparil S, Wright JT. Ethnicity and blood pressure. J Clin Hypertens (Greenwich). 2005 Jun;7(6):357–64

    2005

  16. [24]

    Sex differences in systolic blood pressure response to exercise testing: a real-world clinical analysis

    Li S, Zhang T, He J, He C, Liu F, Yao Y , et al. Sex differences in systolic blood pressure response to exercise testing: a real-world clinical analysis. J Hum Hypertens [Internet]. 2025 Jun 30 [cited 2025 Jun 30]; Available from: https://www.nature.com/articles/s41371-025-01037-1

    2025

  17. [25]

    Fundamentals of biostatistics

    Rosner B. Fundamentals of biostatistics. 8th edition. Australia Brazil Mexico Singapore United Kingdom United States: Cengage Learning; 2016. 927 p

    2016

  18. [26]

    Determinants of lung function across childhood in the Severe Asthma Research Program (SARP) 3

    Gaffin JM, Petty CR, Sorkness RL, Denlinger LC, Phillips BR, Ly NP , et al. Determinants of lung function across childhood in the Severe Asthma Research Program (SARP) 3. J Allergy Clin Immunol. 2023 Jan;151(1):138-146.e9

    2023

  19. [27]

    Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods

    Hüllermeier E, Waegeman W. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach Learn. 2021 Mar;110(3):457– 506

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.