Pith. sign in

REVIEW 2 major objections 1 minor 25 references

A beta-binomial model respecting randomization and its comparison to the standard beta-binomial model that ignores randomization for the meta-analysis of rare events

T0 review · 2 major / 1 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A beta-binomial model for rare events that respects randomization performs well in simulations of real meta-analyses.

desk verdict The paper gives a common-beta BBM that conditions on study totals to respect randomization, but the simulation evidence for its advantage is tied to how the data were generated and does not clearly show bias in the standard model under realistic conditions. read the letter →

arxiv 2606.27971 v1 pith:NLLMNL6R submitted 2026-06-26 stat.ME

Tim Mathes , Maxi Schulz , Oliver Kuss This is my paper · ORCID
classification stat.ME
keywords beta-binomialmodelmeta-analysisrareeventsrandomizationsimulationstudyzerogeneralizedlinearmixedmodelsecologicalbias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a beta-binomial model that accounts for randomization in meta-analyses of rare events by using a common beta and conditioning on the total counts in each study's four-fold table. This addresses the limitation of the standard model which ignores randomization. A simulation study designed to reflect actual Cochrane and non-Cochrane reviews demonstrates that the new model performs well overall. The standard model tends to have issues when study sample sizes vary greatly, although it can handle high heterogeneity better. This matters for combining evidence from studies with zero events without introducing design-related bias.

What carries the argument

Common-beta beta-binomial model with conditioning on the total sum of counts in each study's four-fold table to preserve randomization in parameter estimation.

What would settle it

A new simulation study or application to real data where the model respecting randomization shows worse performance metrics like bias or coverage than the standard model when study sizes are unequal.

Watch

Extended reading notes

Core claim

The beta-binomial model that respects randomization is constructed using a common-beta approach and by conditioning on the total sum of counts in a study's four-fold table when estimating parameters. This preserves the randomization. In simulations mirroring real meta-analyses, it performs well. Ignoring randomization appears problematic mainly when sample sizes differ substantially across studies, but the standard model performs better under high heterogeneity. The new model is generally preferred to avoid ecological bias.

Load-bearing premise

The simulation study accurately reflects the structure and heterogeneity patterns of real-world meta-analyses of rare events.

Editorial extensions

If this is right

  • The standard BBM is usually not biased when randomization is balanced and study sizes are similar.
  • Ignoring randomization is problematic with very different study sample sizes.
  • The BBM respecting randomization shows very similar results to the standard one in the simulation study.
  • The respecting model avoids potential ecological bias inherent in ignoring randomization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This conditioning technique might apply to other statistical models for binary data in meta-analysis.
  • Guidelines for meta-analysis of rare events could recommend the new model to ensure respect for study design.
  • Empirical tests on published meta-analyses with known randomization could further validate the approach.
  • High heterogeneity scenarios may still favor the standard model in practice despite the bias risk.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper proposes a common-beta beta-binomial model (BBM) for meta-analysis of rare events that respects randomization by conditioning on the total event count in each study's four-fold table when estimating parameters. It compares this model to the standard BBM (which ignores randomization) in a simulation study designed to mirror real Cochrane and non-Cochrane meta-analyses of rare events, concluding that the randomization-respecting BBM performs well overall, is preferable to avoid potential ecological bias (especially with large differences in study sizes), and that the standard BBM is usually unbiased when randomization is balanced and study sizes are similar, though it can outperform under high heterogeneity.

Significance. If the simulation accurately reproduces the joint distributions of study sizes, event rates, randomization balances, and heterogeneity patterns from real reviews, the work supplies a practical alternative to inverse-variance methods that avoids continuity corrections while addressing a structural limitation of the standard BBM. The explicit attempt to ground the simulation in empirical review characteristics is a methodological strength.

major comments (2)
  1. [Methods] Methods (simulation study description): The data-generating mechanism for the four-fold tables must be specified in sufficient detail to verify that randomization ratios are permitted to vary across studies according to the empirical distribution observed in the Cochrane and non-Cochrane reviews being mirrored; without this, it is impossible to confirm that the conditioning step is evaluated under conditions where ecological bias would actually arise.
  2. [Results] Results (performance comparison): The statement that the standard BBM 'tended to perform better when heterogeneity was high' requires a quantitative breakdown (e.g., by heterogeneity level and study-size imbalance) showing the magnitude of any performance difference; the current qualitative summary leaves unclear whether the advantage is practically meaningful or confined to extreme parameter regions.
minor comments (1)
  1. [Abstract] Abstract: 'studys four-fold table' contains a typo and should read 'study's four-fold table'.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which highlight important areas for clarification in our simulation study and results presentation. We address each major comment below and will revise the manuscript accordingly to improve transparency and rigor.

read point-by-point responses

  1. Referee: [Methods] Methods (simulation study description): The data-generating mechanism for the four-fold tables must be specified in sufficient detail to verify that randomization ratios are permitted to vary across studies according to the empirical distribution observed in the Cochrane and non-Cochrane reviews being mirrored; without this, it is impossible to confirm that the conditioning step is evaluated under conditions where ecological bias would actually arise.

    Authors: We agree that the current description of the simulation study lacks sufficient detail on the data-generating process for the four-fold tables. In the revised manuscript, we will expand the Methods section to explicitly describe how randomization ratios are sampled from the empirical distributions observed in the Cochrane and non-Cochrane reviews, including the full specification of the joint distributions for study sizes, event rates, and heterogeneity. This will confirm that the conditioning approach is evaluated under realistic conditions where ecological bias may arise. revision: yes

  2. Referee: [Results] Results (performance comparison): The statement that the standard BBM 'tended to perform better when heterogeneity was high' requires a quantitative breakdown (e.g., by heterogeneity level and study-size imbalance) showing the magnitude of any performance difference; the current qualitative summary leaves unclear whether the advantage is practically meaningful or confined to extreme parameter regions.

    Authors: We concur that the qualitative statement requires supporting quantitative detail to allow readers to assess the practical relevance of any performance differences. In the revision, we will add a quantitative breakdown in the Results section, including tables or supplementary figures that stratify performance metrics (such as bias, coverage, and mean squared error) by heterogeneity level and study-size imbalance, reporting the magnitude of differences across the simulated scenarios. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; central comparison rests on independent simulation benchmarks

full rationale

The paper introduces a common-beta BBM with conditioning on study totals to respect randomization and evaluates it via a simulation study designed to mirror real Cochrane and non-Cochrane meta-analyses of rare events. No equations, parameter fits, or first-principles derivations reduce to the target quantities by construction; the reported performance advantage is assessed against externally generated data structures rather than self-referential fitting or self-citation chains. The simulation serves as an independent benchmark, satisfying the criteria for a self-contained result with no load-bearing circular steps.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract only; no explicit free parameters, axioms, or invented entities are stated. The approach builds on the existing beta-binomial framework without introducing new entities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A beta-binomial model respecting randomization and its comparison to the standard beta-binomial model that ignores randomization for the meta-analysis of rare events." pith.science (2026). https://pith.science/paper/NLLMNL6R

@misc{pith2026260627971,
  author       = {Pith},
  title        = {Pith review of: A beta-binomial model respecting randomization and its comparison to the standard beta-binomial model that ignores randomization for the meta-analysis of rare events},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLLMNL6R}},
  note         = {Machine review of arXiv:2606.27971}
}
read the original abstract

Background: One of the suggested models for meta-analysis with rare events is the beta-binomial model (BBM). The main advantage of this model compared to inverse-variance models, is that it uses information from zero cells without needing a continuity correction. A disadvantage of the standard BBM is that it ignores randomization. Here we introduce a BBM that respects randomization. Methods: The main idea to preserve randomization is using a common-beta BBM. We illustrate that randomization is preserved by conditioning on the total sum of counts in a studys four-fold table when estimating the model parameters. We perform a simulation study reflecting real-world meta-analyses to compare the models. In addition, we explore in which situations ignoring randomization could be problematic. Results: The BBM that respects randomization performs well in the simulation study that mirrors real meta-analyses in Cochrane and non-Cochrane reviews, respectively. Ignoring randomization appears to be problematic in situations with very different sample sizes of the studies included in the meta-analysis. However, the BBM ignoring randomization tended to perform better when heterogeneity was high. Conclusion: The results show that using the standard BBM, which ignores randomization is usually not biased when the randomization is balanced and the size of studies included in the meta-analysis is not very different. However, as possible ecological bias due to ignoring randomization is an inherent disadvantage of the model and the BBM that respects randomization shows very similar results in the simulation study, it may be generally preferred. Key words Beta-binomial model, generalized linear mixed models, meta-analyses, simulation study, rare events, zero events

Figures

Figures reproduced from arXiv: 2606.27971 by the authors.

Figure 1
Figure 1. Different summarization levels of the BB models and the consequences of respecting or ignoring randomization [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. converged runs under H1 [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Bias under H1 [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages

  1. [1]

    A re-analysis of about 60,000 sparse data meta-analyses suggests that using an adequate method for pooling matters

    Schulz M, Kramer M, Kuss O, et al. A re-analysis of about 60,000 sparse data meta-analyses suggests that using an adequate method for pooling matters. Research Synthesis Methods 2024; 15: 978-987. DOI: https://doi.org/10.1002/jrsm.1748

    work page doi:10.1002/jrsm.1748 2024

  2. [2]

    Utilization of the evidence from studies with no events in meta- analyses of adverse events: an empirical investigation

    Xu C, Zhou X, Zorzela L, et al. Utilization of the evidence from studies with no events in meta- analyses of adverse events: an empirical investigation. BMC Medicine 2021; 19: 141. DOI: 10.1186/s12916-021-02008-2

    work page doi:10.1186/s12916-021-02008-2 2021

  3. [3]

    The impact of correction methods on rare-event meta- analysis

    Zabriskie BN, Cole N, Baldauf J, et al. The impact of correction methods on rare-event meta- analysis. Research Synthesis Methods 2024; 15: 130-151. DOI: https://doi.org/10.1002/jrsm.1677

    work page doi:10.1002/jrsm.1677 2024

  4. [4]

    Journal of the American Statistical Association 107, 737–753

    Bhaumik DK, Amatya A, Normand SL, et al. Meta-Analysis of Rare Binary Adverse Event Data. J Am Stat Assoc 2012; 107: 555-567. DOI: 10.1080/01621459.2012.664484

    work page doi:10.1080/01621459.2012.664484 2012

  5. [5]

    Random-effects meta-analysis of few studies involving rare events

    Günhan BK, Röver C and Friede T. Random-effects meta-analysis of few studies involving rare events. Research Synthesis Methods 2020; 11: 74-90. DOI: https://doi.org/10.1002/jrsm.1370

    work page doi:10.1002/jrsm.1370 2020

  6. [6]

    Rare events meta-analysis using the Bayesian beta-binomial model

    Jansen K and Holling H. Rare events meta-analysis using the Bayesian beta-binomial model. Research Synthesis Methods 2023; 14: 853-873. DOI: https://doi.org/10.1002/jrsm.1662

    work page doi:10.1002/jrsm.1662 2023

  7. [7]

    Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless

    Kuss O. Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless. Stat Med 2015; 34: 1097-1116. 20141201. DOI: 10.1002/sim.6383

    work page doi:10.1002/sim.6383 2015

  8. [8]

    A general framework for the use of logistic regression models in meta-analysis

    Simmonds MC and Higgins JPT. A general framework for the use of logistic regression models in meta-analysis. Statistical Methods in Medical Research 2014; 25: 2858-2877. DOI: 10.1177/0962280214534409

    work page doi:10.1177/0962280214534409 2014

Show all 25 references
  1. [9]

    Comparison of random-effects meta-analysis models for the relative risk in the case of rare events: A simulation study

    Beisemann M, Doebler P and Holling H. Comparison of random-effects meta-analysis models for the relative risk in the case of rare events: A simulation study. Biometrical Journal 2020; 62: 1597-

    2020

  2. [10]

    DOI: https://doi.org/10.1002/bimj.201900379

    doi

  3. [11]

    Performance of several types of beta-binomial models in comparison to standard approaches for meta-analyses with very few studies

    Felsch M, Beckmann L, Bender R, et al. Performance of several types of beta-binomial models in comparison to standard approaches for meta-analyses with very few studies. BMC Medical Research Methodology 2022; 22: 319. DOI: 10.1186/s12874-022-01779-3. 80

    2022 doi

  4. [12]

    Random-effects meta-analysis models for the odds ratio in the case of rare events under different data-generating models: A simulation study

    Jansen K and Holling H. Random-effects meta-analysis models for the odds ratio in the case of rare events under different data-generating models: A simulation study. Biometrical Journal 2023; 65: 2200132. DOI: https://doi.org/10.1002/bimj.202200132

    2023 doi

  5. [13]

    A comparison of methods for meta-analysis of a small number of studies with binary outcomes

    Mathes T and Kuss O. A comparison of methods for meta-analysis of a small number of studies with binary outcomes. Research Synthesis Methods 2018; 9: 366-381. DOI: https://doi.org/10.1002/jrsm.1296

    2018 doi

  6. [14]

    Contrast-Based and Arm-Based Models for Network Meta-Analysis

    Karahalios A, McKenzie JE and White IR. Contrast-Based and Arm-Based Models for Network Meta-Analysis. Methods Mol Biol 2022; 2345: 203-221. DOI: 10.1007/978-1-0716-1566-9_13

    2022 doi

  7. [15]

    Beta-binomial models for meta-analysis with binary outcomes: Variations, extensions, and additional insights from econometrics

    Mathes T and Kuss O. Beta-binomial models for meta-analysis with binary outcomes: Variations, extensions, and additional insights from econometrics. Research Methods in Medicine & Health Sciences 2021; 2: 82-89. DOI: 10.1177/2632084321996225

    2021 doi

  8. [16]

    Econometric Models for Count Data with an Application to the Patents-R & D Relationship

    Hausman J, Hall BH and Griliches Z. Econometric Models for Count Data with an Application to the Patents-R & D Relationship. Econometrica 1984; 52: 909-938. DOI: 10.2307/1911191

    1984 doi

  9. [17]

    A Simple Approach to Fit the Beta-binomial Model

    Guimarães P. A Simple Approach to Fit the Beta-binomial Model. The Stata Journal 2005; 5: 385-394. DOI: 10.1177/1536867x0500500307

    2005 doi

  10. [18]

    Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study

    Page MJ, Shamseer L, Altman DG, et al. Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLOS Medicine 2016; 13: e1002028. DOI: 10.1371/journal.pmed.1002028

    2016 doi

  11. [19]

    Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews

    Turner RM, Davey J, Clarke MJ, et al. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol 2012; 41: 818-827. 20120329. DOI: 10.1093/ije/dys041

    2012 doi

  12. [20]

    Analysing data and undertaking meta-analyses

    Deeks JJ, Higgins JP, Altman DG, et al. Analysing data and undertaking meta-analyses. Cochrane Handbook for Systematic Reviews of Interventions. 2019, pp.241-284

    2019

  13. [21]

    Improved tests for a random effects meta-regression with a single covariate

    Knapp G and Hartung J. Improved tests for a random effects meta-regression with a single covariate. Stat Med 2003; 22: 2693-2710. Journal Article 2003/08/27. DOI: 10.1002/sim.1482

    2003 doi

  14. [22]

    Off‐pump versus on‐pump coronary artery bypass grafting for ischaemic heart disease

    Møller CH, Penninga L, Wetterslev J, et al. Off‐pump versus on‐pump coronary artery bypass grafting for ischaemic heart disease. Cochrane Database of Systematic Reviews 2012. DOI: 10.1002/14651858.CD007224.pub2

    2012 doi

  15. [23]

    Simpson's paradox visualized: the example of the rosiglitazone meta-analysis

    Rücker G and Schumacher M. Simpson's paradox visualized: the example of the rosiglitazone meta-analysis. BMC Med Res Methodol 2008; 8: 34. 20080530. DOI: 10.1186/1471-2288-8-34

    2008 doi

  16. [24]

    How to observe the principle of concurrent control in an arm- based meta-analysis using SAS procedures GLIMMIX and BGLIMM

    Piepho H-P and Madden LV. How to observe the principle of concurrent control in an arm- based meta-analysis using SAS procedures GLIMMIX and BGLIMM. Research Synthesis Methods 2022; 13: 821-828. DOI: https://doi.org/10.1002/jrsm.1576

    2022 doi

  17. [25]

    A comparison of seven random-effects models for meta- analyses that estimate the summary odds ratio

    Jackson D, Law M, Stijnen T, et al. A comparison of seven random-effects models for meta- analyses that estimate the summary odds ratio. Stat Med 2018; 37: 1059-1085. 20180108. DOI: 10.1002/sim.7588

    2018 doi

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.