Pith. Sign in

REVIEW

Is the Jeffreys' scale a reliable tool for Bayesian model comparison in cosmology?

Cosmology and Nongalactic Astrophysics · Submitted 29 Oct 2012 · 35 references

Pith's verdict: Correct analytic Bayes-factor formulas and a genuinely useful basis-dependence table, wrapped around a Jeffreys-scale critique whose mock-data demonstration is largely built into the prior choice.

T0 · AI review

How this review was made: An AI referee (selfhosted/deepseek-v4.1-flash) read the paper on November 10, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

  1. T0 AI review An AI read the paper and answered a fixed list of questions.
  2. T1 Audit Automatic checks that the references and files the paper relies on exist and match.
  3. T2 Compute The paper's code or calculations re-run with the same results.
  4. T3 Formal statement The claims written out in a formal math language a computer can read.
  5. T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 5 major objections · 8 minor · Ask about this paper · The compiled record · Read the full review →
Challenge this review Comment (0) Read on arXiv ↗
Cite ▾
arXiv ID1210.7652 (version 2)
Pith IDSPM25OYF, Pith's permanent record of this paper and its review
CitationJuan Garcia-Bellido, Savvas Nesseris. "Is the Jeffreys' scale a reliable tool for Bayesian model comparison in cosmology?." arXiv:1210.7652 (2012). https://pith.science/paper/SPM25OYF
Cite this reviewBibTeX for Pith's review of this paper
Claim paper

Why Pith says this

Highlight any text to comment on it.

Referee report
The paper

The paper claims

The paper argues that a fixed Bayes-factor threshold cannot be a universal rule for choosing between models, because the evidence ratio for linear models depends on parameter covariances and prior widths, not just on how much better one fit

The reading

Cosmologists routinely decide between competing models by computing a Bayes factor and reading off a verdict from Jeffreys' scale, a fixed ladder of thresholds (roughly $1$–$3$ weak, $3$–$20$ definite, $20$–$150$ strong, above $150$ very strong). This paper asks whether that ladder deserves its authority. The authors restrict attention to models that are linear in their parameters, where the evidence integral can be done exactly, and derive closed-form Bayes factors for both Gaussian and flat priors. They find that the log Bayes factor splits into the usual fit-improvement term, $-Delta chi^2_{min}/2$, plus a second term built from the Fisher matrices and prior widths of the two models, so two models that fit equally well can still be separated or conflated depending on their covariances alone. They then fit mock data with polynomials of different degree and show cases where a 4-parameter and a 14-parameter model come out essentially tied, and where polynomials up to degree about 12 are preferred over the model that actually generated the data. If the argument holds, quoting a Jeffreys threshold as if it settled a model comparison is not defensible, and any such verdict has to be reported together with the priors and covariances that produced it.

What carries the argument

The load-bearing object is the exact evidence integral for a linear model with a Gaussian likelihood. With Gaussian priors of inverse covariance $H$ and a Fisher matrix $F$ from the least-squares fit, completing the square gives $B_1 = e^{-chi^2_{min}/2 - c/2}|I_M + H^{-1}F|^{-1/2}$, where $c$ measures the offset between the prior centre and the best fit; with flat priors of width $Delta a_i$ the evidence becomes a product of error functions of $D^{perp}_{ii} Delta a_i / (2sqrt{2})$, with $D^perp$ from the Cholesky decomposition of the rotated Fisher matrix. Both forms make explicit that the evidence ratio is a product of a fit term and a volume term, and the volume term is what the Jeffreys thresholds ignore. The paper pairs this with an analytic Figure of Merit, the inverse volume of the $n$-$\sigma$ confidence ellipsoid, $mathrm{FoM}(M) = |F|^{1/2} Gamma(M/2+1) pi^{-M/2} (delta chi^2)^{-M/2}$, which lets the authors separate the effect of adding parameters from the effect of changing basis (monomials versus Legendre versus Chebyshev) at fixed parameter count.

Extended reading notes

Core claim

The central claim is that Jeffreys' scale is not a 'one size fits all' tool: its thresholds are stated in terms of the Bayes factor alone, but the Bayes factor for linear models is not a function of model complexity alone. The authors derive the exact evidence for Gaussian priors, $B_1 = e^{-chi^2_{min}/2 - c/2}|I_M + H^{-1}F|^{-1/2}$, and for flat priors an error-function product over the prior widths, and both reduce to the schematic form $ln B_{12} = -Delta chi^2_{min}/2 + ln G(M_1,M_2)$ where $G$ carries all the covariance and prior-volume information. The demonstration uses 31 mock points generated from $f(x) = a + (x-b)e^{-c x^2}$ and fits them with polynomials of degree $M_1$ and $M_2$. Pairs such as $(M_1,M_2)=(4,14)$ come out at $ln B_{12} approx 1.2$, which the scale calls indecisive, and $(4,10)$ with wider priors comes out at $ln B_{12} = 0$, which the scale calls a dead heat, even though ten extra parameters are being added. In the reverse direction, comparing the true generating model against high-degree polynomials yields strong support against the true model for degrees up to $M sim 10$–$12$. The authors describe these as Type I (rejecting a true model) and Type II (failing to reject a false one) failures of the scale.

Load-bearing premise

The headline demonstration sets each prior width proportional to the fitted parameter error, $Delta a_i = n sigma_i$ (or $H_{ii} = n sigma_i^2$), with $n=3$ or $7$; because the prior then scales with the posterior, models whose extra parameters are poorly constrained are guaranteed to come out near a tie, so the failure of the scale is partly built into that choice rather than discovered from the data, and no prior fixed independently of the fitted errors is tested.

Editorial extensions

If this is right

  • A Bayes factor quoted without its prior specification and Fisher matrices is not a self-contained piece of evidence, since the same fit improvement can move the verdict across several Jeffreys thresholds when the prior widths change.
  • Adding parameters to a model whose extra directions are poorly constrained by the data need not lower the evidence much, so an Occam penalty read off a fixed scale can understate how little the data actually say.
  • Comparing a candidate physical model against a flexible polynomial expansion can favor the polynomial even when the physical model is the true generator, so such comparisons need a prior on the expansion coefficients rather than a default width.
  • Because the Figure of Merit depends on the polynomial basis through a Jacobian determinant, parameter constraints and the associated FoM rankings are basis-dependent at fixed model dimension, and orthogonal bases such as Chebyshev give tighter contours than monomials for the same data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The failure mode the authors document is structurally the same one that appears whenever an evidence ratio is dominated by the Occam volume rather than the likelihood: the remedy is to report the evidence as a function of prior width, not as a single number compared against a ladder.
  • A natural follow-up is a calibration study: for a fixed data set, sweep the prior width continuously and record where each Jeffreys threshold is crossed, which would convert the scale from a verdict into a sensitivity curve.
  • The same decomposition should apply to any model that is linear in a subset of its parameters after profiling the rest, which suggests the criticism extends well beyond polynomial fitting to, for example, linear perturbation spectra with nuisance amplitudes.
  • If the thresholds are arbitrary in the linear case where everything is computable, the situation can only be less controlled for the non-linear cosmological models the authors mention, where the evidence itself must be estimated numerically.
Original abstract

We are entering an era where progress in cosmology is driven by data, and alternative models will have to be compared and ruled out according to some consistent criterium. The most conservative and widely used approach is Bayesian model comparison. In this paper we explicitly calculate the Bayes factors for all models that are linear with respect to their parameters. We do this in order to test the so called Jeffreys' scale and determine analytically how accurate its predictions are in a simple case where we fully understand and can calculate everything analytically. We also discuss the case of nested models, e.g. one with $M_1$ and another with $M_2\supset M_1$ parameters and we derive analytic expressions for both the Bayes factor and the Figure of Merit, defined as the inverse area of the model parameter's confidence contours. With all this machinery and the use of an explicit example we demonstrate that the threshold nature of Jeffreys' scale is not a "one size fits all" reliable tool for model comparison and that it may lead to biased conclusions. Furthermore, we discuss the importance of choosing the right basis in the context of models that are linear with respect to their parameters and how that basis affects the parameter estimation and the derived constraints.

The compiled record

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Compiled by classifier-1.13-20260917 via OpenRouter, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,737 input tokens, $0.0002 reported.

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.32 · human participants 0.01 · controlled comparison 0.06 · randomized assignment 0.01 · states quantitative result 0.08 · compares with prior results 0.54 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; .

We are entering an era where progress in cosmology is driven by data, and alternative models will have to be compared and ruled out according to some consistent criterium. The most conservative and widely used approach is Bayesian model comparison. In this paper we explicitly calculate the Bayes factors for all models that are linear with respect to their parameters. We do this in order to test the so called Jeffreys' scale and determine analytically how accurate its predictions are in a simple case where we fully understand and can calculate everything analytically. We also discuss the case of nested models, e.g. one with $M_1$ and another with $M_2\supset M_1$ parameters and we derive analytic expressions for both the Bayes factor and the Figure of Merit, defined as the inverse area of the model parameter's confidence contours. With all this machinery and the use of an explicit example we demonstrate that the threshold nature of Jeffreys' scale is not a "one size fits all" reliable tool for model comparison and that it may lead to biased conclusions. Furthermore, we discuss the importance of choosing the right basis in the context of models that are linear with respect to their parameters and how that basis affects the parameter estimation and the derived constraints.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#sentencerolethe authors' own workstates the main conclusionstrength
s0 We are entering an era where progress in cosmology is driven by data, and alternative models will have to be compared and ruled out according to some consistent criterium. background 1.00 0.04 0.02 asserted 0.97
s1 The most conservative and widely used approach is Bayesian model comparison. background 1.00 0.04 0.02 asserted 1.00
s2 In this paper we explicitly calculate the Bayes factors for all models that are linear with respect to their parameters. method 0.58 0.96 0.02 asserted 0.98
s3 We do this in order to test the so called Jeffreys' scale and determine analytically how accurate its predictions are in a simple case where we fully understand and can calculate everything analytically. method 0.66 0.86 0.04 asserted 0.80
s4 We also discuss the case of nested models, e.g. one with $M_1$ and another with $M_2\supset M_1$ parameters and we derive analytic expressions for both the Bayes factor and the Figure of Merit, defined as the inverse area of the model parameter's confidence contours. contribution 0.63 0.95 0.03 asserted 0.98
s5 With all this machinery and the use of an explicit example we demonstrate that the threshold nature of Jeffreys' scale is not a "one size fits all" reliable tool for model comparison and that it may lead to biased conclusions. result 0.74 0.95 0.94 asserted 0.84
s6 Furthermore, we discuss the importance of choosing the right basis in the context of models that are linear with respect to their parameters and how that basis affects the parameter estimation and the derived constraints. contribution 0.90 0.58 0.04 asserted 0.90

What kind of paper the classifier reports this is: theoretical result 0.99 · evidence from proof or derivation 1.00.

What was examined, and what was not

sources read
  • arXiv listing abstract v2
not examined
  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers
levels run
  • P abstract
not run
  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

Figures
Figure 1. FIG. 1: The best-fit
Figure 2. FIG. 2: Contour plots of the Bayes factor log
Figure 3. FIG. 3: Contour plots of the Bayes factor log
Figure 4. FIG. 4: The Bayes factor log
Figure 5. FIG. 5: Contour plots of the Bayes factor log
Figure 6. FIG. 6: Contour plots of the Bayes factor log
Figure 7. FIG. 7: The dependence of the function
Figure 8. FIG. 8: The dependence of Ratio(

Figures reproduced from arXiv:1210.7652 by the authors.

References
How the literature uses it

52 arXiv papers cite this in 68 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 60 sentences without a confident label

Uses its method or tool 4 papers
Cites it as background 1 paper

What it cites · 5 sentences without a confident label

Uses the method or tool of 1 paper
Cites as support 1 paper
Cites as background 11 papers
  • On the other hand, the Bayesian evidence is based on Bayes theorem, see Refs. [ref] , [ref] for in-depth reviews, which expresses the posterior distribution $P(u,M|D)$ for the parameters $u$ of the model $M$ given the data $D$, in terms of the likelihood distribution function $L(D| u,M)$ within a given set of priors $π(u,M)$ [equation] where the likelihood can be obtained from $L(D| u,M)=exp(-χ^2(u)/2)$.

    Statistical methods for cosmological parameter selection and estimation · Introduction · 0.96

  • On the other hand, the Bayesian evidence is based on Bayes theorem, see Refs. [ref] , [ref] for in-depth reviews, which expresses the posterior distribution $P(u,M|D)$ for the parameters $u$ of the model $M$ given the data $D$, in terms of the likelihood distribution function $L(D| u,M)$ within a given set of priors $π(u,M)$ [equation] where the likelihood can be obtained from $L(D| u,M)=exp(-χ^2(u)/2)$.

    Bayes in the sky: Bayesian inference and model selection in cosmology · Introduction · 0.96

  • Several examples of the use of the Jeffreys' scale in cosmology and astronomy can by found in [ref] and references there-in.

    Limitations of Bayesian Evidence Applied to Cosmology · The Jeffreys' scale revisited · 0.95

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.