Efficient Bayes Factor Sensitivity Analysis via Posterior Density Ratios
Pith reviewed 2026-05-09 21:25 UTC · model grok-4.3
The pith
A single model fit with a hyperprior recovers the full Bayes factor sensitivity curve via density ratios.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The Bayes factor at any hyperparameter value γ_x equals the Bayes factor at a reference value γ_0 multiplied by the ratio of two posterior densities for γ in an extended model that places a hyperprior on γ. Once the extended model is fit, the ratio for any γ_x is approximated by the importance-weighted marginal density estimator applied to the MCMC output. Because the sensitivity parameter enters only through the prior, the likelihood term cancels and the estimator simplifies to a ratio of prior density values at the sampled parameter draws, requiring no further likelihood evaluations.
What carries the argument
The decomposition identity expressing the Bayes factor at γ_x as the anchor Bayes factor at γ_0 times a Savage-Dickey density ratio in the hyperprior-extended model, estimated by the importance-weighted marginal density estimator reduced to prior density ratios.
If this is right
- The entire sensitivity curve over any range of hyperparameter values follows from one MCMC run on the extended model.
- The estimator remains accurate with small MCMC sample sizes and outperforms kernel density estimation over the full range.
- The method extends directly to simultaneous sensitivity analysis over several hyperparameters.
- It applies without modification to sensitivity checks inside Bayesian model averaging.
- No additional likelihood computations are required after the initial extended-model fit.
Where Pith is reading between the lines
- The same single-fit density-ratio strategy could be adapted to explore sensitivity of other posterior summaries such as means or intervals.
- Interactive software tools could let users vary the hyperparameter on demand after one upfront fit.
- The cancellation property may generalize to other ratio-based estimators that separate prior and likelihood contributions.
- Testing the approach on hierarchical models with many hyperparameters would show whether the computational savings scale to larger problems.
Load-bearing premise
The sensitivity parameter affects the model only through the prior on the model parameters, so the data likelihood cancels exactly in the density ratio estimator.
What would settle it
In the univariate Bayesian t-test validation case, if the sensitivity curve recovered from the single extended-model fit deviates from the exact Bayes factors computed separately at each hyperparameter value beyond ordinary MCMC sampling error.
Figures
read the original abstract
Bayes factor sensitivity analysis examines how the evidence for one hypothesis over another depends on the prior distribution. In complex models, the standard approach refits the model at each hyper-parameter value, and the total computational cost scales linearly in the grid size. We propose a method that recovers the entire sensitivity curve from a single additional model fit. The key identity decomposes the Bayes factor at any hyper-parameter value $\gamma_x$ into an ``anchor'' Bayes factor at a fixed reference $\gamma_0$ and a Savage--Dickey density ratio in an extended model that places a hyper-prior on $\gamma$. Once this extended model is fit, the Bayes factor at any $\gamma_x$ follows from the anchor value and a ratio of two posterior density ordinates. To approximate this ratio, we employ the importance-weighted marginal density estimator (IWMDE). Because the sensitivity parameter enters the model only through the prior distribution on the model parameters, the data likelihood cancels in the IWMDE, reducing it to a simple ratio of prior density evaluations on the MCMC draws, without any additional likelihood computation. The resulting estimator is fast, remains accurate even with small MCMC samples, and substantially outperforms kernel density estimation across the full sensitivity range. The method extends naturally to simultaneous sensitivity over multiple hyper-parameters and to Bayesian model averaging. We illustrate it on a univariate Bayesian $t$-test with exact Bayes factors for validation, a bivariate informed $t$-test, and a Bayesian model-averaged meta-analysis, obtaining accurate sensitivity curves at a fraction of the brute-force cost.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes an efficient method for Bayes factor sensitivity analysis over prior hyperparameters. By augmenting the model with a hyperprior on the sensitivity parameter γ and performing a single MCMC fit of the extended model, the Bayes factor at any target value γ_x is recovered from a fixed anchor Bayes factor at γ_0 multiplied by a Savage–Dickey posterior density ratio. This ratio is approximated via the importance-weighted marginal density estimator (IWMDE); because γ enters only through the conditional prior p(θ|γ), the likelihood cancels and the estimator reduces to a ratio of prior density evaluations at existing MCMC draws. The approach is validated on a univariate t-test against exact Bayes factors, shown to produce accurate curves on a bivariate informed t-test and a Bayesian model-averaged meta-analysis, and claimed to outperform kernel density estimation while extending naturally to multiple hyperparameters and model averaging.
Significance. If the central identity and IWMDE reduction hold, the method substantially lowers the computational cost of prior sensitivity analysis in models where repeated refits are prohibitive, enabling routine exploration of hyperparameter effects from a single MCMC run. The explicit cancellation of the likelihood and reliance on standard MCMC output are attractive features that could be adopted in applied Bayesian workflows.
minor comments (3)
- §3 (IWMDE implementation): the manuscript should clarify the precise form of the importance weights and the choice of proposal density used in the IWMDE step, including any tuning parameters, to allow exact reproduction of the reported curves.
- Figure 2 and 3 captions: the reported pointwise error bands are not defined (e.g., whether they are Monte Carlo standard errors or quantiles across replications); adding this detail would strengthen the empirical validation.
- The abstract states that the method 'substantially outperforms kernel density estimation'; a brief quantitative comparison (e.g., integrated squared error or maximum deviation) in the main text or supplement would make this claim easier to assess.
Simulated Author's Rebuttal
We thank the referee for their positive summary of the manuscript and for recommending minor revision. We appreciate the recognition that the proposed identity and IWMDE-based estimator can substantially reduce the computational burden of Bayes factor sensitivity analysis while relying only on standard MCMC output.
Circularity Check
No significant circularity; derivation relies on standard external identities
full rationale
The central identity decomposes the Bayes factor via the marginal likelihood definition after introducing an auxiliary hyperprior on γ, yielding m(data|γ_x)/m(data|γ_0) = [p(γ_x|data)/p(γ_0|data)] × [p(γ_0)/p(γ_x)]. This is a direct algebraic consequence of the joint posterior factorization p(θ,γ|data) ∝ L(data|θ)p(θ|γ)p(γ), with the data likelihood canceling in the IWMDE ratio of prior ordinates. Both the Savage-Dickey density ratio and the IWMDE estimator are invoked from prior literature rather than derived or fitted within the paper; no equation reduces the target sensitivity curve to a quantity defined by the same fitted parameters or self-citation chain. The method therefore remains self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption The sensitivity parameter enters the model exclusively through the prior on the model parameters.
- domain assumption MCMC draws from the extended model are available and sufficiently accurate for density ratio estimation.
Reference graph
Works this paper leans on
-
[1]
Alcock, J. (2011). Back from the future: P arapsychology and the B em affair. Skeptical Inquirer 35, 31--39
2011
-
[2]
, Scheibehenne, B
Andraszewicz, S. , Scheibehenne, B. , Rieskamp, J. , Grasman, R. , Verhagen, J. & Wagenmakers, E.-J. (2015). An introduction to B ayesian hypothesis testing for management research. Journal of Management 41, 521--543
2015
-
[3]
, Gronau, Q
Barto s , F. , Gronau, Q. F. , Timmers, B. , Otte, W. M. , Ly, A. & Wagenmakers, E.-J. (2021). Bayesian model-averaged meta-analysis in medicine. Statistics in Medicine 40, 6743--6761
2021
-
[4]
& Maier, M
Barto s , F. & Maier, M. (2020). RoBMA : A n R package for robust B ayesian meta-analyses. https://CRAN.R-project.org/package=RoBMA. R package version 3.5.0
2020
-
[5]
, Maier, M
Barto s , F. , Maier, M. , Wagenmakers, E.-J. , Doucouliagos, H. & Stanley, T. D. (2022). Robust B ayesian meta-analysis: M odel-averaging across complementary publication bias adjustment methods. Research Synthesis Methods 14, 99--116
2022
-
[6]
& Wagenmakers, E.-J
Barto s , F. & Wagenmakers, E.-J. (2023). A general approximation to nested B ayes factors with informed priors. Stat 12, e600
2023
-
[7]
Bem, D. J. (2011). Feeling the future: E xperimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 100, 407--425
2011
-
[8]
Berger, J. O. (1982). The robust B ayesian viewpoint. Tech. rep., Purdue University
1982
-
[9]
Berger, J. O. (1990). Robust B ayesian analysis: S ensitivity to the prior. Journal of Statistical Planning and Inference 25, 303--328
1990
-
[10]
Berger, J. O. , Moreno, E. , Pericchi, L. R. , Bayarri, M. J. , Bernardo, J. M. , Cano, J. A. , De la Horra, J. , Martín, J. , Ríos-Insúa, D. , Betrò, B. , Dasgupta, A. , Gustafson, P. , Wasserman, L. , Kadane, J. B. , Srinivasan, C. , Lavine, M. , O’Hagan, A. , Polasek, W. , Robert, C. P. , Goutis, C. , Ruggeri, F. , Salinetti, G. & Sivaganesan, S. (1994...
1994
-
[11]
Box, G. E. & Tiao, G. C. (1962). A further look at robustness via B ayes's theorem. Biometrika 49, 419--432
1962
-
[12]
, Gelman, A
Carpenter, B. , Gelman, A. , Hoffman, M. D. , Lee, D. , Goodrich, B. , Betancourt, M. , Brubaker, M. , Guo, J. , Li, P. & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software 76, 1--32
2017
-
[13]
Chen, M.-H. (1994). Importance-weighted marginal B ayesian posterior density estimation. Journal of the American Statistical Association 89, 818--824
1994
-
[14]
Chib, S. (1995). Marginal likelihood from the G ibbs output. Journal of the American Statistical Association 90, 1313--1321
1995
-
[15]
, Canale, A
Corradin, R. , Canale, A. & Nipoti, B. (2021). BNPmix : An R package for B ayesian nonparametric modeling via P itman- Y or mixtures. Journal of Statistical Software 100, 1--33
2021
-
[16]
& Wickham, H
Deng, H. & Wickham, H. (2014). Density estimation in r
2014
-
[17]
& Van de Schoot, R
Depaoli, S. & Van de Schoot, R. (2017). Improving transparency and replication in B ayesian statistics: T he WAMBS - C hecklist. Psychological Methods 22, 240
2017
-
[18]
, Kass, R
Diciccio, T. , Kass, R. , Raftery, A. & Wasserman, L. (1997). Computing B ayes factors by combining simulation and asymptotic approximations. Journal of the American Statistical Association 92, 903--915
1997
-
[19]
Dickey, J. M. (1971). The weighted likelihood ratio, linear hypotheses on normal location parameters. The Annals of Mathematical Statistics 42, 204--223
1971
-
[20]
Dickey, J. M. (1973). Scientific reporting and personal probabilities: S tudent's hypothesis. Journal of the Royal Statistical Society: Series B (Methodological) 35, 285--305
1973
-
[21]
Dienes, Z. (2014). Using B ayes to get the most out of non-significant results. Frontiers in Psychology 5:781
2014
-
[22]
& Rufibach, K
Dümbgen, L. & Rufibach, K. (2009). Maximum likelihood estimation of a log-concave density and its distribution function: Basic properties and uniform consistency. Bernoulli 15, 40--68
2009
-
[23]
Fowlie, A. (2024). The B ayes factor surface for searches for new physics. The European Physical Journal C 84, 426
2024
-
[24]
Fragoso, T. M. , Bertoli, W. & Louzada, F. (2018). Bayesian model averaging: A systematic review and conceptual classification. International Statistical Review 86, 1--28
2018
-
[25]
Francis, G. (2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review 19, 151--156
2012
-
[26]
Franck, C. T. & Gramacy, R. B. (2020). Assessing B ayes factor surfaces using interactive visualization and computer surrogate modeling. The American Statistician 74, 359--369
2020
-
[27]
Gelfand, A. E. & Smith, A. F. M. (1990). Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association 85, 398--409
1990
-
[28]
Goodman, S. N. (1999). Toward evidence-based medical statistics. 2: T he B ayes factor. Annals of internal medicine 130, 1005--1013
1999
-
[29]
Gronau, Q. F. , Heck, D. W. , Berkhout, S. W. , Haaf, J. M. & Wagenmakers, E.-J. (2021). A primer on B ayesian model-averaged meta-analysis. Advances in Methods and Practices in Psychological Science 4, 1--19
2021
-
[30]
Gronau, Q. F. , Ly, A. & Wagenmakers, E.-J. (2020). Informed B ayesian t -tests. The American Statistician 74, 137--143
2020
-
[31]
Gronau, Q. F. , Sarafoglou, A. , Matzke, D. , Ly, A. , Boehm, U. , Marsman, M. , Leslie, D. S. , Forster, J. J. , Wagenmakers, E.-J. & Steingroever, H. (2017). A tutorial on bridge sampling. Journal of Mathematical Psychology 81, 80--97
2017
-
[32]
, Matthews, R
Held, L. , Matthews, R. , Ott, M. & Pawel, S. (2022). Reverse- B ayes methods for evidence assessment and research synthesis. Research Synthesis Methods 13, 295--314
2022
-
[33]
, Gronau, Q
Hinne, M. , Gronau, Q. F. , van den Bergh, D. & Wagenmakers, E.-J. (2020). A conceptual introduction to B ayesian model averaging. Advances in Methods and Practices in Psychological Science 3, 200--215
2020
-
[34]
Hoeting, J. A. , Madigan, D. , Raftery, A. E. & Volinsky, C. T. (1999). Bayesian model averaging: a tutorial. Statistical Science 14, 382--401
1999
-
[35]
Jeffreys, H. (1939). Theory of probability. Oxford, UK: Oxford University Press, 1st ed
1939
-
[36]
Jeffreys, H. (1948). Theory of Probability. Oxford, UK: Oxford University Press, 2nd ed
1948
-
[37]
Johnson, V. E. , Pramanik, S. & Shudde, R. (2023). Bayes factor functions for reporting outcomes of hypothesis tests. Proceedings of the National Academy of Sciences 120, e2217331120
2023
-
[38]
Kass, R. E. & Raftery, A. E. (1995). B ayes factors. Journal of the American Statistical Association 90, 773--795
1995
-
[39]
Komunjer, I. (2007). Asymmetric power distribution: Theory and applications to risk measurement. Journal of Applied Econometrics 22, 891--921
2007
-
[40]
Kooperberg, C. (2020). Polspline: polynomial spline routines. https://CRAN.R-project.org/package=polspline. R package version 1.1.25
2020
-
[41]
& Stone, C
Kooperberg, C. & Stone, C. J. (1991). A study of logspline density estimation. Computational Statistics & Data Analysis 12, 327--347
1991
-
[42]
Kruschke, J. K. (2021). Bayesian analysis reporting guidelines. Nature Human Behaviour 5, 1282--1291
2021
-
[43]
Lavine, M. (1991). Sensitivity in B ayesian statistics: T he prior and the likelihood. Journal of the American Statistical Association 86, 396--399
1991
-
[44]
Lee, M. D. & Vanpaemel, W. (2018). Determining informative priors for cognitive models. Psychonomic Bulletin & Review 25, 114--127
2018
-
[45]
, Paulo, R
Liang, F. , Paulo, R. , Molina, G. , Clyde, M. A. & Berger, J. O. (2008). Mixtures of g priors for B ayesian variable selection. Journal of the American Statistical Association 103, 410--423
2008
-
[46]
, Martino, L
Llorente, F. , Martino, L. , Delgado, D. & L \'o pez-Santiago, J. (2023). Marginal likelihood computation for model selection and hypothesis testing: An extensive review. SIAM Review 65, 3--58
2023
-
[47]
, Kim, W
Lodewyckx, T. , Kim, W. , Lee, M. D. , Tuerlinckx, F. , Kuppens, P. & Wagenmakers, E.-J. (2011). A tutorial on B ayes factor estimation with the product space method. Journal of Mathematical Psychology 55, 331--347
2011
-
[48]
& Wong, W
Meng, X.-L. & Wong, W. H. (1996). Simulating ratios of normalizing constants via a simple identity: A theoretical exploration. Statistica Sinica 6, 831--860
1996
-
[49]
Morey, R. D. & Rouder, J. N. (2015). BayesFactor : C omputation of B ayes factors for common designs. R package version 0.9.12-4.3
2015
-
[50]
Morey, R. D. , Rouder, J. N. , Pratte, M. S. & Speckman, P. L. (2011). Using MCMC chain outputs to efficiently estimate B ayes factors. Journal of Mathematical Psychology 55, 368--378
2011
-
[51]
, Musfeld, P
Oberauer, K. , Musfeld, P. & Aust, F. (2025). Reliability, bias, and computational cost of estimating the B ayes factor using bridge sampling and the S avage-- D ickey density ratio. Manuscript submitted for publication. OSF: https://osf.io/wg8b4/
2025
-
[52]
Pawel, S. , Ly, A. & Wagenmakers, E.-J. (2024). Evidential calibration of confidence intervals. The American Statistician 78, 47--57
2024
-
[53]
Plummer, M. (2003). JAGS : A program for analysis of B ayesian graphical models using G ibbs sampling. In Proceedings of the 3rd international workshop on distributed statistical computing, vol. 124. Vienna, Austria
2003
-
[54]
Rouder, J. N. , Speckman, P. L. , Sun, D. , Morey, R. D. & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review 16, 225--237
2009
-
[55]
& Duembgen, L
Rufibach, K. & Duembgen, L. (2023). logcondens: Estimate a log-concave probability density from iid observations. R package version 2.1.8
2023
-
[56]
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods 17, 551
2012
-
[57]
, Keetelaar, S
Sekulovski, N. , Keetelaar, S. , Haslbeck, J. & Marsman, M. (2024). Sensitivity analysis of prior distributions in B ayesian graphical modeling: G uiding informed prior choices for conditional independence testing. Advances. in /psychology 2, e92355
2024
-
[58]
Sheather, S. J. & Jones, M. C. (1991). A reliable data-based bandwidth selection method for kernel density estimation. Journal of the Royal Statistical Society. Series B (Methodological) 53, pp. 683--690
1991
-
[59]
& Stern, H
Sinharay, S. & Stern, H. S. (2002). On the sensitivity of B ayes factors to the prior distributions. The American Statistician 56, 196--201
2002
-
[60]
Skilling, J. (2006). Nested sampling for general bayesian computation. Bayesian Analysis 1, 833--859
2006
-
[61]
Spiegelhalter, D. J. , Abrams, K. R. & Myles, J. P. (2004). Bayesian approaches to clinical trials and health-care evaluation. Chichester: John Wiley & Sons
2004
-
[62]
Stanley, T. D. & Doucouliagos, H. (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods 5, 60--78
2014
-
[63]
Department of Health and Human Services , Food and Drug Administration , Center for Drug Evaluation and Research (CDER) & Center for Biologics Evaluation and Research (CBER) (2026)
U.S. Department of Health and Human Services , Food and Drug Administration , Center for Drug Evaluation and Research (CDER) & Center for Biologics Evaluation and Research (CBER) (2026). Use of B ayesian methodology in clinical trials of drug and biological products: Guidance for industry
2026
-
[64]
, van den Bergh, D
van Doorn, J. , van den Bergh, D. , B \"o hm, U. , Dablander, F. , Derks, K. , Draws, T. , Etz, A. , Evans, N. J. , Gronau, Q. F. , Haaf, J. M. et al. (2020). The JASP guidelines for conducting and reporting a B ayesian analysis. Psychonomic Bulletin & Review , 1--14
2020
-
[65]
Vanpaemel, W. (2010). Prior sensitivity in theory testing: A n apologia for the B ayes factor. Journal of Mathematical Psychology 54, 491--498
2010
-
[66]
Vevea, J. L. & Hedges, L. V. (1995). A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, 419--435
1995
-
[67]
, Beek, T
Wagenmakers, E.-J. , Beek, T. , Dijkhoff, L. , Gronau, Q. F. , Acosta, A. , Adams Jr, R. , Albohn, D. , Allard, E. , Benning, S. D. , Blouin-Hudon, E.-M. et al. (2016). Registered replication report: S track, M artin, & S tepper (1988). Perspectives on Psychological Science 11, 917--928
2016
-
[68]
, Gronau, Q
Wagenmakers, E.-J. , Gronau, Q. F. , Dablander, F. & Etz, A. (2022). The support interval. Erkenntnis 87, 589--601
2022
-
[69]
, Lodewyckx, T
Wagenmakers, E.-J. , Lodewyckx, T. , Kuriyal, H. & Grasman, R. (2010). Bayesian hypothesis testing for psychologists: A tutorial on the S avage-- D ickey method. Cognitive Psychology 60, 158--189
2010
-
[70]
Wand, M. (2025). KernSmooth: Functions for Kernel Smoothing Supporting Wand & Jones (1995). R package version 2.23-26
2025
-
[71]
Wand, M. P. (1994). Fast computation of multivariate kernel estimators. Journal of Computational and Graphical Statistics 3, 433--445
1994
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.