REVIEW 4 major objections 3 minor 6 references
A relation between log-likelihood and cross-validation log-scores
T0 review · 4 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper proves that the log-likelihood of a hypothesis given finite data is exactly a weighted average of every leave-one-out cross-validation log-score over all non-empty subsets of the data.
desk verdict The intended identity is correct but the paper's central equation is wrong as printed; a corrective revision could make it a citable note, not a major result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by repeated application of the probability product rule, $P(D \mid H I) = P(D_i \mid D_{-i} H I)\,P(D_{-i} \mid H I)$, starting from the tautology that the likelihood is the $d$-th root of its own $d$-th power. Expanding each of the $d$ copies with a different datum $D_i$ splits the likelihood into one column of leave-one-out predictive factors and a second column of smaller likelihoods $P(D_{-i}\mid H I)$; recursing on those smaller likelihoods generates every subset scale. The weights (such as $\binom{d}{k}^{-1}$ and the cyclic average $1/k$) are exactly the counts needed so that each non-empty subset appears once and each data ordering is accounted for.
What would settle it
Take a concrete model and a small dataset (say $d=2$ or $d=3$) with strictly positive predictive probabilities, and evaluate both sides of Eq. (11) numerically; any mismatch beyond rounding refutes the claimed weights. Alternatively, choose a model in which some conditional probability is exactly zero: the right-hand side then contains $\log 0$ while the left-hand side may be finite, showing that the stated identity depends on the positivity convention the paper does not specify.
Extended reading notes
Core claim
The central discovery is that $\log P(D \mid H I)$ is not merely approximated by cross-validation scores; it is identically equal to a sum over all non-empty subsets of the data. In the compact form of Eq. (11), the log-likelihood is $\sum_{k=1}^{d} \binom{d}{k}^{-1}$ times, for each ordered $k$-tuple of data, the average over cyclic permutations of $\log P(D_{i_1} \mid D_{i_2}\cdots D_{i_k} H I)$. Eq. (12) rewrites the same identity as, for each datum, the average over all subsets of the remaining data of predicting that datum from the subset. The paper claims the derivation is fully general: it does not assume exchangeability or a particular sampling model, it works in any logarithmic base, and the same recursion yields generalizations to leave-$k$-out and $k$-fold cross-validation log-scores.
Load-bearing premise
The identity as written requires that every conditional probability appearing inside a logarithm be strictly positive, so all logs are finite; it also presumes the data form a finite list, since no limit or infinite-product convention is supplied.
Editorial extensions
If this is right
- The leave-one-out cross-validation log-score is not an alternative to the log-likelihood but one term in its exact decomposition; comparing them is comparing a piece with the whole.
- Any ranking of models by log-likelihood is implicitly a ranking by the aggregate of all subset log-scores.
- Because the recursion is purely logical, the identity holds for every hypothesis and every finite data set, independent of ordering, exchangeability, or sampling assumptions.
- The same expansion applies to leave-k-out and k-fold cross-validation log-scores, so the log-likelihood can be represented as an average over all such scores, with weights that the paper states for leave-one-out but leaves partly unspecified for general folds.
- The paper's closing remark is a direct corollary: the individual subset scores are expensive to compute, yet their weighted sum is the cheap log-likelihood.
Reading between the lines
- The paper leaves the infinite-data limit open; if the identity extends, the log-likelihood would become an integral over all data subsets of predictive scores rather than a finite sum.
- The identity suggests a practical debugging tool not discussed in the paper: for small $d$, compute the weighted sum over all subset log-scores and compare it with the directly computed log-likelihood; a mismatch would localize errors in cross-validation code.
- The same algebraic pattern may apply to other predictive scores built from conditional probabilities, such as leave-all-but-one-out scores or predictive likelihoods, yielding a family of exact decompositions.
- In model-comparison practice, the result implies that disagreements between leave-one-out cross-validation and log-likelihood rankings must be attributable to the weights given to different subset sizes, a question the paper does not resolve.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript derives an exact identity relating the log-likelihood log P(D | H I) to leave-one-out cross-validation log-scores. The derivation starts from the product rule of probability, expresses P(D|HI) as a d-th root of a d-fold product (Eq. 6), expands each copy using the product rule with a different held-out datum (Eq. 8), and recursively applies the same expansion to the remaining factors. The paper presents an explicit d=3 example (Eq. 9), a general formula (Eq. 10), and two compact forms (Eqs. 11 and 12), and claims the relation extends to arbitrary k-fold cross-validation log-scores. The intended result is that the log-likelihood equals a sum, over all non-empty data subsets, of the average leave-one-out log-score within each subset.
Significance. The claimed identity is elegant and potentially useful: it represents the log-likelihood exactly as a weighted aggregation of all subset-level leave-one-out predictive scores, which connects the likelihood principle to cross-validation diagnostics. The main derivation is self-contained, requires no fitted parameters, and the d=3 expansion is explicit and consistent with the recursive construction; these are genuine strengths. The paper does not provide code or machine-checked proofs, but the combinatorial argument is simple enough to verify by hand once the notation is fixed. However, the two compact displayed formulas, Eqs. (11) and (12), are not correct as printed, and the abstract's k-fold generalization is not derived; these issues are load-bearing for the paper's central claim and require a major revision.
major comments (4)
- [§2, Eq. (11)] Equation (11) is false as written. For a fixed k-subset S and a fixed j in S, summing over all ordered k-tuples (k! orderings of S) and over their cyclic permutations counts log P(D_j | S\{j}) k! times before the factor 1/k, i.e. (k-1)! times after it, whereas the intended coefficient, visible in Eq. (10) and the d=3 example, is 1/k. The d=2 case is a concrete counterexample: Eq. (11) assigns the k=2 contribution log P(D1|D2)+log P(D2|D1), while the correct contribution is half that sum. The identity can be repaired by summing over unordered k-subsets, or by dividing the ordered-tuple sum by k!.
- [§2, Eq. (12)] The same normalization problem affects Eq. (12). For a fixed held-out datum D_j and fixed k, the sum over ordered k-tuples of other data counts each k-subset of D\{j} exactly k! times, so the 'average over the C(d-1,k) subsets' described in the text is not achieved unless each ordered tuple is divided by k! (or the sum is over unordered subsets). The notation 'ordered k-tuples, j excluded' also needs a formal definition.
- [§2, Eq. (10)] The notation D_{-i,j}, D_{-i,-j}, D_{-i,-j,k}, and D_{-i,-j,-k} is never defined. It should be replaced by explicit set notation, e.g. D_A = conjunction over i in A of D_i, or the factors should be written as P(D_j | D_{-i,-j} H I), so that the reader can verify the summation indices.
- [Abstract and §3] The abstract states that the relation 'can be generalized to any k-fold cross-validation log-scores', but Section 3 only sketches this generalization and explicitly says the weights have not been calculated. Either derive the generalization and its weights, or remove the claim from the abstract and state the generalization as a conjecture.
minor comments (3)
- [§2, Eqs. (6)-(8)] The expansion assumes every conditional probability whose logarithm is taken is strictly positive; the paper should state this assumption and specify how zero-probability events are handled (e.g., via extended reals).
- [§1, Eq. (5)] The notation D ≡ V_d i=1 D_i is garbled and should be the logical conjunction, e.g. D ≡ ∧_{i=1}^d D_i; the same applies to other uses of 'V' as a placeholder.
- [Throughout] The manuscript contains many informal and OCR-related artifacts ('/equalx', 'X' for summation, 'D−i,j'), and its casual asides ('You can claim a postcard from me', acknowledgements to Sci-Hub) are not appropriate for a formal journal submission; the published version should use clean mathematical notation and a professional tone.
Circularity Check
No significant circularity: the log-likelihood/log-score identity is derived self-containedly from the product rule.
full rationale
The paper's central claim is a direct mathematical identity obtained by applying the product rule of probability, starting from the root-product identity in eq. (6) and expanding each factor via eq. (7). The log-likelihood log P(D | HI) is not defined in terms of leave-one-out log-scores; rather, the product rule relates conditional probabilities at different subset sizes, and the recursion produces the claimed weighted sum of log-scores. No parameter is fitted to data, no external benchmark is invoked, and no prediction is validated against a fitted subset. The only self-citation is footnote 14 to Porta Mana 2019, which is flagged as future work and is not used in the derivation. Even if eq. (11)'s literal counting of ordered tuples and cyclic permutations overcounts by k! as a matter of normalization, that would be a correctness or typographical issue in a displayed formula, not circularity: the recursive construction shown in eqs. (8)-(10) is self-contained and does not assume the result it purports to prove. The analysis therefore finds no circular step.
Assumptions & free parameters
assumptions (3)
- standard math The product rule of probability P(D|HI) = P(D_i|D_{-i} HI) P(D_{-i}|HI) holds for every datum i.
- domain assumption All conditional probabilities appearing in the log-scores are strictly positive, so their logarithms are finite.
- domain assumption The set of hypotheses {H_h} is mutually exclusive and exhaustive for the Bayesian interpretation (eq. 2).
Cite this review
Pith. "Pith review of A relation between log-likelihood and cross-validation log-scores." pith.science (2026). https://pith.science/paper/DT3I4IYH
@misc{pith2026190808741,
author = {Pith},
title = {Pith review of: A relation between log-likelihood and cross-validation log-scores},
year = {2026},
howpublished = {\url{https://pith.science/paper/DT3I4IYH}},
note = {Machine review of arXiv:1908.08741}
}
abstract
It is shown that the log-likelihood of a hypothesis or model given some data is equivalent to an average of all leave-one-out cross-validation log-scores that can be calculated from all subsets of the data. This relation can be generalized to any $k$-fold cross-validation log-scores.
Reference graph
Works this paper leans on
-
[1]
Bernardo, J. - M., DeGroot, M. H., Lindley, D. V., Smith, A. F. M., eds. (1985): Bayesian Statistics 2 . (Elsevier and Valencia University Press, Amsterdam and Valencia). https ://www.uv.es/~bernardo/valenciam.html . Bernardo, J. - M., Smith, A. F. (2000): Bayesian Theory , reprint. (Wiley, New York). First publ
work page 1985
-
[5]
https://archive.org/detai ls/XQUHIUXHIQUHIQXUIHX2 , http://www-biba.inrialpes.fr/Jaynes/prob.html . Jeffreys, H. (1983): Theory of Probability , third ed. with corrections. (Oxford University Press, London). First publ
work page 1983
-
[13]
— (1975): Explicativity, corroboration, and the relative odds of hypotheses . Synthese 301–2, 39–73. Partly repr. in Good (1983) ch
work page 1975
-
[15]
— (1981): Some logic and history of hypothesis testing . In: Philosophy in economics . Ed. by J. C. Pitt (Reidel), 149–174. Repr. in Good (1983) ch. 14 pp. 129–148. — (1983): Good Thinking: The Foundations of Probability and Its Applications . (University of Minnesota Press, Minneapolis, USA). — (1985): Weight of evidence: a brief survey . In: Bernardo, D...
work page 1981
-
[1939]
Kass, R. E., Raftery, A. E. (1995): Bayes factors . J. Am. Stat. Assoc. 90430, 773–795. https://w ww.stat.washington.edu/raftery/Research/PDF/kass1995.pdf ; https://www.andr ew.cmu.edu/user/kk3n/simplicity/KassRaftery1995.pdf . Krnjajić, M., Draper, D. (2011): Bayesian model specification: some problems related to model choice and calibration . http://hdl...
work page 1995
-
[1994]
Limitations of Bayesian leave-one-out cross-validation for model selection
Chandramouli, S. H., Shiffrin, R. M., Vehtari, A., Simpson, D. P., Yao, Y., Gelman, A., Navarro, D. J., Gronau, Q. F., et al. (2019): Commentary on Gronau and Wagenmakers. Limitations of “Limitations of Bayesian leave-one-out cross-validation for model selection”. Between the devil and the deep blue sea: tensions between scientific judgement and statistic...
work page 2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.