REVIEW
Is the Jeffreys' scale a reliable tool for Bayesian model comparison in cosmology?
Cosmology and Nongalactic Astrophysics · Submitted 29 Oct 2012 · 35 references
Pith's verdict: Correct analytic Bayes-factor formulas and a genuinely useful basis-dependence table, wrapped around a Jeffreys-scale critique whose mock-data demonstration is largely built into the prior choice.
T0 · AI review
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
The paper argues that a fixed Bayes-factor threshold cannot be a universal rule for choosing between models, because the evidence ratio for linear models depends on parameter covariances and prior widths, not just on how much better one fit
The reading
What carries the argument
The load-bearing object is the exact evidence integral for a linear model with a Gaussian likelihood. With Gaussian priors of inverse covariance $H$ and a Fisher matrix $F$ from the least-squares fit, completing the square gives $B_1 = e^{-chi^2_{min}/2 - c/2}|I_M + H^{-1}F|^{-1/2}$, where $c$ measures the offset between the prior centre and the best fit; with flat priors of width $Delta a_i$ the evidence becomes a product of error functions of $D^{perp}_{ii} Delta a_i / (2sqrt{2})$, with $D^perp$ from the Cholesky decomposition of the rotated Fisher matrix. Both forms make explicit that the evidence ratio is a product of a fit term and a volume term, and the volume term is what the Jeffreys thresholds ignore. The paper pairs this with an analytic Figure of Merit, the inverse volume of the $n$-$\sigma$ confidence ellipsoid, $mathrm{FoM}(M) = |F|^{1/2} Gamma(M/2+1) pi^{-M/2} (delta chi^2)^{-M/2}$, which lets the authors separate the effect of adding parameters from the effect of changing basis (monomials versus Legendre versus Chebyshev) at fixed parameter count.
Extended reading notes
Core claim
The central claim is that Jeffreys' scale is not a 'one size fits all' tool: its thresholds are stated in terms of the Bayes factor alone, but the Bayes factor for linear models is not a function of model complexity alone. The authors derive the exact evidence for Gaussian priors, $B_1 = e^{-chi^2_{min}/2 - c/2}|I_M + H^{-1}F|^{-1/2}$, and for flat priors an error-function product over the prior widths, and both reduce to the schematic form $ln B_{12} = -Delta chi^2_{min}/2 + ln G(M_1,M_2)$ where $G$ carries all the covariance and prior-volume information. The demonstration uses 31 mock points generated from $f(x) = a + (x-b)e^{-c x^2}$ and fits them with polynomials of degree $M_1$ and $M_2$. Pairs such as $(M_1,M_2)=(4,14)$ come out at $ln B_{12} approx 1.2$, which the scale calls indecisive, and $(4,10)$ with wider priors comes out at $ln B_{12} = 0$, which the scale calls a dead heat, even though ten extra parameters are being added. In the reverse direction, comparing the true generating model against high-degree polynomials yields strong support against the true model for degrees up to $M sim 10$–$12$. The authors describe these as Type I (rejecting a true model) and Type II (failing to reject a false one) failures of the scale.
Load-bearing premise
The headline demonstration sets each prior width proportional to the fitted parameter error, $Delta a_i = n sigma_i$ (or $H_{ii} = n sigma_i^2$), with $n=3$ or $7$; because the prior then scales with the posterior, models whose extra parameters are poorly constrained are guaranteed to come out near a tie, so the failure of the scale is partly built into that choice rather than discovered from the data, and no prior fixed independently of the fitted errors is tested.
Editorial extensions
If this is right
- A Bayes factor quoted without its prior specification and Fisher matrices is not a self-contained piece of evidence, since the same fit improvement can move the verdict across several Jeffreys thresholds when the prior widths change.
- Adding parameters to a model whose extra directions are poorly constrained by the data need not lower the evidence much, so an Occam penalty read off a fixed scale can understate how little the data actually say.
- Comparing a candidate physical model against a flexible polynomial expansion can favor the polynomial even when the physical model is the true generator, so such comparisons need a prior on the expansion coefficients rather than a default width.
- Because the Figure of Merit depends on the polynomial basis through a Jacobian determinant, parameter constraints and the associated FoM rankings are basis-dependent at fixed model dimension, and orthogonal bases such as Chebyshev give tighter contours than monomials for the same data.
Reading between the lines
- The failure mode the authors document is structurally the same one that appears whenever an evidence ratio is dominated by the Occam volume rather than the likelihood: the remedy is to report the evidence as a function of prior width, not as a single number compared against a ladder.
- A natural follow-up is a calibration study: for a fixed data set, sweep the prior width continuously and record where each Jeffreys threshold is crossed, which would convert the scale from a verdict into a sensitivity curve.
- The same decomposition should apply to any model that is linear in a subset of its parameters after profiling the rest, which suggests the criticism extends well beyond polynomial fitting to, for example, linear perturbation spectra with nuisance amplitudes.
- If the thresholds are arbitrary in the linear case where everything is computable, the situation can only be less controlled for the non-linear cosmological models the authors mention, where the evidence itself must be estimated numerically.
Original abstract
We are entering an era where progress in cosmology is driven by data, and alternative models will have to be compared and ruled out according to some consistent criterium. The most conservative and widely used approach is Bayesian model comparison. In this paper we explicitly calculate the Bayes factors for all models that are linear with respect to their parameters. We do this in order to test the so called Jeffreys' scale and determine analytically how accurate its predictions are in a simple case where we fully understand and can calculate everything analytically. We also discuss the case of nested models, e.g. one with $M_1$ and another with $M_2\supset M_1$ parameters and we derive analytic expressions for both the Bayes factor and the Figure of Merit, defined as the inverse area of the model parameter's confidence contours. With all this machinery and the use of an explicit example we demonstrate that the threshold nature of Jeffreys' scale is not a "one size fits all" reliable tool for model comparison and that it may lead to biased conclusions. Furthermore, we discuss the importance of choosing the right basis in the context of models that are linear with respect to their parameters and how that basis affects the parameter estimation and the derived constraints.
The compiled record
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
The abstract, sentence by sentence
We are entering an era where progress in cosmology is driven by data, and alternative models will have to be compared and ruled out according to some consistent criterium. The most conservative and widely used approach is Bayesian model comparison. In this paper we explicitly calculate the Bayes factors for all models that are linear with respect to their parameters. We do this in order to test the so called Jeffreys' scale and determine analytically how accurate its predictions are in a simple case where we fully understand and can calculate everything analytically. We also discuss the case of nested models, e.g. one with $M_1$ and another with $M_2\supset M_1$ parameters and we derive analytic expressions for both the Bayes factor and the Figure of Merit, defined as the inverse area of the model parameter's confidence contours. With all this machinery and the use of an explicit example we demonstrate that the threshold nature of Jeffreys' scale is not a "one size fits all" reliable tool for model comparison and that it may lead to biased conclusions. Furthermore, we discuss the importance of choosing the right basis in the context of models that are linear with respect to their parameters and how that basis affects the parameter estimation and the derived constraints.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | We are entering an era where progress in cosmology is driven by data, and alternative models will have to be compared and ruled out according to some consistent criterium. | background 1.00 | 0.04 | 0.02 | asserted 0.97 |
| s1 | The most conservative and widely used approach is Bayesian model comparison. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s2 | In this paper we explicitly calculate the Bayes factors for all models that are linear with respect to their parameters. | method 0.58 | 0.96 | 0.02 | asserted 0.98 |
| s3 | We do this in order to test the so called Jeffreys' scale and determine analytically how accurate its predictions are in a simple case where we fully understand and can calculate everything analytically. | method 0.66 | 0.86 | 0.04 | asserted 0.80 |
| s4 | We also discuss the case of nested models, e.g. one with $M_1$ and another with $M_2\supset M_1$ parameters and we derive analytic expressions for both the Bayes factor and the Figure of Merit, defined as the inverse area of the model parameter's confidence contours. | contribution 0.63 | 0.95 | 0.03 | asserted 0.98 |
| s5 | With all this machinery and the use of an explicit example we demonstrate that the threshold nature of Jeffreys' scale is not a "one size fits all" reliable tool for model comparison and that it may lead to biased conclusions. | result 0.74 | 0.95 | 0.94 | asserted 0.84 |
| s6 | Furthermore, we discuss the importance of choosing the right basis in the context of models that are linear with respect to their parameters and how that basis affects the parameter estimation and the derived constraints. | contribution 0.90 | 0.58 | 0.04 | asserted 0.90 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
Figures
Figures reproduced from arXiv:1210.7652 by the authors.
References
How the literature uses it
52 arXiv papers cite this in 68 sentences
Papers citing it · 60 sentences without a confident label
Uses its method or tool 4 papers
-
First, we consider the Akaike Information Criterion (AIC) [ref] , defined as AIC ≡-2 lnℒ_max+2 N_tot=χ_min^2 +2 N_tot , where $N_tot$ corresponds to the total number of free parameters of the considered model and $ℒ_max$ corresponds to the maximum likelihood.
Late-transition vs smooth $H(z)$ deformation models for the resolution of the Hubble crisis · Model selection · 0.97
-
In order to compare two models $ℳ_1$ and $ℳ_2$, we can make use of the ratio of the Bayesian evidences calculated by the Harmonic Mean Approximation (see Ref. [ref] ), given by:
Cosmological Constraints on Higgs-Dilaton Inflation · Bayesian analysis of $w_0w_a$CDM vs. HDM · 0.99
-
To do this, having two models $M_1$ and $M_2$ the Jeffreys scale with respect of $Δlnε= lnε_M_1- lnε_M_2$ is as the following: [ref] : * for $Δlnε< 1.1$ there is a weak evidence against model $M_2$. * for $1.1<Δlnε< 3$ there is a definite evidence against model $M_2$. * and finally for $3<Δlnε$ there is a strong evidence against model $M_2$.
A Bayesian comparison between $\Lambda$CDM and phenomenologically emergent dark energy models · Observational data and Bayesian inference · 0.88
Cites it as background 1 paper
-
It can also be found in many other references, as in [ref] , but we add this information here too for completeness.
Quantifying the evidence for the current speed-up of the Universe with low and intermediate-redshift data. A more model-independent approach · Computation of evidences and other quantities of interest · 0.94
What it cites · 5 sentences without a confident label
Uses the method or tool of 1 paper
-
Then, this gives [ref] , [ref]
Everything you wanted to know about Data Analysis and Fitting but were afraid to ask · Minimization · 0.93
Cites as support 1 paper
-
It is model predictiveness, not model simplicity, which is "rewarded" in Bayesian model comparison [ref] , [ref] .
Should we doubt the cosmological constant? · Conclusions · 0.99
Cites as background 11 papers
-
On the other hand, the Bayesian evidence is based on Bayes theorem, see Refs. [ref] , [ref] for in-depth reviews, which expresses the posterior distribution $P(u,M|D)$ for the parameters $u$ of the model $M$ given the data $D$, in terms of the likelihood distribution function $L(D| u,M)$ within a given set of priors $π(u,M)$ [equation] where the likelihood can be obtained from $L(D| u,M)=exp(-χ^2(u)/2)$.
Statistical methods for cosmological parameter selection and estimation · Introduction · 0.96
-
On the other hand, the Bayesian evidence is based on Bayes theorem, see Refs. [ref] , [ref] for in-depth reviews, which expresses the posterior distribution $P(u,M|D)$ for the parameters $u$ of the model $M$ given the data $D$, in terms of the likelihood distribution function $L(D| u,M)$ within a given set of priors $π(u,M)$ [equation] where the likelihood can be obtained from $L(D| u,M)=exp(-χ^2(u)/2)$.
Bayes in the sky: Bayesian inference and model selection in cosmology · Introduction · 0.96
-
Several examples of the use of the Jeffreys' scale in cosmology and astronomy can by found in [ref] and references there-in.
Limitations of Bayesian Evidence Applied to Cosmology · The Jeffreys' scale revisited · 0.95
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.