{"id":"fb1467cf-b704-40a7-b795-3850c32bbe1a","arxiv_id":"1908.08741","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The log-likelihood equals the sum, over all subset sizes, of averaged leave-one-out cross-validation log-scores for every subset of the data.","lead":"The paper proves an exact identity: the log-likelihood of a model equals a weighted average of leave-one-out cross-validation log-scores computed over all non-empty subsets of the data. The result formally connects two standard statistical measures, but the proof is a short exercise in probability rules and the practical impact is limited.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (11), the compact form of the central claim, is false as printed: the 'ordered k-tuples times cyclic permutations' sum overcounts each subset's log-score by k!; the intended identity is correct once normalized.","rationale":"Re-deriving the recursion for the coefficients gives a_{d,k} = 1/C(d,k), so the substantive identity is sound; the weakness is in the compact display. The reader's chosen weakest assumption, strict positivity of all conditionals, is a minor technicality: for probabilities the identity extends to the extended reals (all terms are at most 0, so a single -infinity propagates consistently), and for densities one usually assumes positivity almost everywhere. The larger issue is that eqs. (11)-(12), which the strongest claim explicitly quotes, are not normalized correctly: the ordered-tuples plus cyclic-permutations sum overcounts by k!, and eq. (12) has the same problem if 'ordered k-tuples' is read literally. The paper's own eq. (10) and the d=3 example in eq. (9) confirm the intended formula, so the contribution is conditional on fixing the display and on proving or retracting the k-fold generalization. This agrees in direction with the reader's verdict; I do not see a reason to change it.","tokens_in":5916,"tokens_out":28577,"duration_ms":270554,"concrete_test":"Take d=2 with two independent Bernoulli(1/2) data points. Literal evaluation of eq. (11): k=1 gives (1/2)(log(1/2)+log(1/2)) = -log 2; k=2 gives log P(D1|D2)+log P(D2|D1) = 2 log(1/2) = -2 log 2; total RHS = -3 log 2. But log P(D | HI) = log(1/4) = -2 log 2. If eq. (11) is instead normalized by summing over unordered k-subsets only, with one term per j in each subset, it reproduces eq. (10) and the true value -2 log 2. This one-line numerical check settles whether the overcount is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The identity the paper intends is correct: recursive use of P(S | HI) = (1/|S|) sum_{j in S} log P(D_j | S\\{j}, HI) P(S\\{j} | HI) gives log P(D | HI) = sum_{k=1}^d C(d,k)^{-1} sum_{|S|=k} (1/k) sum_{j in S} log P(D_j | S\\{j}, HI). But eq. (11) as written is not this identity. For a fixed k-subset S, the sum over all ordered k-tuples together with the sum over their cyclic permutations counts each conditional log P(D_j | S\\{j}) exactly k! times: for each j, every ordering of the remaining k-1 elements appears as the tail of a cyclic permutation, instead of only once. The formula therefore overcounts by k!. A concrete failure: for d=2, eq. (11) gives RHS = (1/2)(log P(D1)+log P(D2)) + log P(D1|D2)+log P(D2|D1), whereas the correct value is (1/2)(log P(D1)+log P(D2)) + (1/2)[log P(D1|D2)+log P(D2|D1)]; these differ whenever the two conditionals are not both 1. Eq. (12) has the same normalization problem if 'ordered k-tuples' is read literally. The text and eq. (10) convey the correct relation, so the flaw is repairable, but the displayed central equations are not currently correct.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript derives an exact identity relating the log-likelihood log P(D | H I) to leave-one-out cross-validation log-scores. The derivation starts from the product rule of probability, expresses P(D|HI) as a d-th root of a d-fold product (Eq. 6), expands each copy using the product rule with a different held-out datum (Eq. 8), and recursively applies the same expansion to the remaining factors. The paper presents an explicit d=3 example (Eq. 9), a general formula (Eq. 10), and two compact forms (Eqs. 11 and 12), and claims the relation extends to arbitrary k-fold cross-validation log-scores. The intended result is that the log-likelihood equals a sum, over all non-empty data subsets, of the average leave-one-out log-score within each subset.","tokens_in":6274,"tokens_out":14355,"duration_ms":132596,"significance":"The claimed identity is elegant and potentially useful: it represents the log-likelihood exactly as a weighted aggregation of all subset-level leave-one-out predictive scores, which connects the likelihood principle to cross-validation diagnostics. The main derivation is self-contained, requires no fitted parameters, and the d=3 expansion is explicit and consistent with the recursive construction; these are genuine strengths. The paper does not provide code or machine-checked proofs, but the combinatorial argument is simple enough to verify by hand once the notation is fixed. However, the two compact displayed formulas, Eqs. (11) and (12), are not correct as printed, and the abstract's k-fold generalization is not derived; these issues are load-bearing for the paper's central claim and require a major revision.","major_comments":[{"comment":"Equation (11) is false as written. For a fixed k-subset S and a fixed j in S, summing over all ordered k-tuples (k! orderings of S) and over their cyclic permutations counts log P(D_j | S\\{j}) k! times before the factor 1/k, i.e. (k-1)! times after it, whereas the intended coefficient, visible in Eq. (10) and the d=3 example, is 1/k. The d=2 case is a concrete counterexample: Eq. (11) assigns the k=2 contribution log P(D1|D2)+log P(D2|D1), while the correct contribution is half that sum. The identity can be repaired by summing over unordered k-subsets, or by dividing the ordered-tuple sum by k!.","section":"§2, Eq. (11)"},{"comment":"The same normalization problem affects Eq. (12). For a fixed held-out datum D_j and fixed k, the sum over ordered k-tuples of other data counts each k-subset of D\\{j} exactly k! times, so the 'average over the C(d-1,k) subsets' described in the text is not achieved unless each ordered tuple is divided by k! (or the sum is over unordered subsets). The notation 'ordered k-tuples, j excluded' also needs a formal definition.","section":"§2, Eq. (12)"},{"comment":"The notation D_{-i,j}, D_{-i,-j}, D_{-i,-j,k}, and D_{-i,-j,-k} is never defined. It should be replaced by explicit set notation, e.g. D_A = conjunction over i in A of D_i, or the factors should be written as P(D_j | D_{-i,-j} H I), so that the reader can verify the summation indices.","section":"§2, Eq. (10)"},{"comment":"The abstract states that the relation 'can be generalized to any k-fold cross-validation log-scores', but Section 3 only sketches this generalization and explicitly says the weights have not been calculated. Either derive the generalization and its weights, or remove the claim from the abstract and state the generalization as a conjecture.","section":"Abstract and §3"}],"minor_comments":[{"comment":"The expansion assumes every conditional probability whose logarithm is taken is strictly positive; the paper should state this assumption and specify how zero-probability events are handled (e.g., via extended reals).","section":"§2, Eqs. (6)-(8)"},{"comment":"The notation D ≡ V_d i=1 D_i is garbled and should be the logical conjunction, e.g. D ≡ ∧_{i=1}^d D_i; the same applies to other uses of 'V' as a placeholder.","section":"§1, Eq. (5)"},{"comment":"The manuscript contains many informal and OCR-related artifacts ('/equalx', 'X' for summation, 'D−i,j'), and its casual asides ('You can claim a postcard from me', acknowledgements to Sci-Hub) are not appropriate for a formal journal submission; the published version should use clean mathematical notation and a professional tone.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central identity is sound and the d=3 example is correct, but the two compact formulas that carry the main claim are misnormalized as printed. This is repairable within the manuscript's scope, hence major revision rather than rejection. The author's reference to their own in-preparation work is not used in the proof and can be removed or marked as a pointer for future discussion. Editors may also wish to ask the author to revise the informal style before sending the paper to a technical reader."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's main claim—that the log-likelihood equals an average of leave-one-out log-scores over all data subsets—is true, and it's an elementary consequence of the product rule. But the compact equation (11) as printed overcounts by a factor of k!, so the displayed central result is false. The error is repairable, and the informal text and the d=3 example show the intended identity.\n\nWhat's new: the exact subset-weighted identity. It's not a new principle—just repeated applications of the product rule inside a d-th root, averaged over data orderings. But I don't know it from the earlier literature, and it does give a tidy way to see the log-likelihood as a full-scale version of LOO-CV log-scores. That's worth something for conceptual discussions.\n\nSoft spots beyond the typo: eq (10) uses D_{-i,j} without definition; as printed it doesn't match the example at d=3. The abstract promises a generalization to k-fold CV, but the text only sketches it and admits the weights aren't computed. The positivity assumption (all conditionals > 0) is never stated. None of these are fatal to the core identity, but they'd all need to be fixed.\n\nThe paper is short, stylistically quirky, and self-contained. If this came to a journal, I'd send it to a referee, because the corrected identity is a legitimate formal observation and a referee can help clean up the formulas. I would not cite it in its current form; I'd cite it only after those equations are right.","headline":"The intended identity is correct but the paper's central equation is wrong as printed; a corrective revision could make it a citable note, not a major result.","tokens_in":6748,"tokens_out":4276,"would_cite":false,"duration_ms":39645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62A01","62F15"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that the log-likelihood of a hypothesis given finite data is exactly a weighted average of every leave-one-out cross-validation log-score over all non-empty subsets of the data.","keywords":["log-likelihood","leave-one-out cross-validation","log-score","Bayesian probability","product rule","model comparison","predictive scores","data subsets"],"falsifier":"Take a concrete model and a small dataset (say $d=2$ or $d=3$) with strictly positive predictive probabilities, and evaluate both sides of Eq. (11) numerically; any mismatch beyond rounding refutes the claimed weights. Alternatively, choose a model in which some conditional probability is exactly zero: the right-hand side then contains $\\log 0$ while the left-hand side may be finite, showing that the stated identity depends on the positivity convention the paper does not specify.","tokens_in":122,"feed_emoji":"📊","tokens_out":9442,"duration_ms":153990,"temperature":0.7,"pith_summary":"This paper proves an exact identity: for any probabilistic hypothesis or model and any finite list of data, the log-likelihood equals a weighted average of leave-one-out cross-validation log-scores computed within every non-empty subset of the data. The identity follows purely from repeated use of the product rule of probability, so it holds for any hypothesis and any data order. The paper also states an equivalent form in which each datum is predicted conditionally on every subset of the remaining data, and it argues the construction extends to leave-k-out and k-fold cross-validation log-scores. Why it matters: log-likelihood and cross-validation scores are usually treated as rival ways to assess a model, and this result shows the log-likelihood is the complete, all-scales version of the log-score rather than merely an approximation to it.","feed_headline":"Log-likelihood equals the sum of all leave-one-out scores","feed_subtitle":"A new proof rewrites the log-likelihood as averaged predictive scores over every subset of the data.","key_machinery":"The argument is carried by repeated application of the probability product rule, $P(D \\mid H I) = P(D_i \\mid D_{-i} H I)\\,P(D_{-i} \\mid H I)$, starting from the tautology that the likelihood is the $d$-th root of its own $d$-th power. Expanding each of the $d$ copies with a different datum $D_i$ splits the likelihood into one column of leave-one-out predictive factors and a second column of smaller likelihoods $P(D_{-i}\\mid H I)$; recursing on those smaller likelihoods generates every subset scale. The weights (such as $\\binom{d}{k}^{-1}$ and the cyclic average $1/k$) are exactly the counts needed so that each non-empty subset appears once and each data ordering is accounted for.","core_discovery":"The central discovery is that $\\log P(D \\mid H I)$ is not merely approximated by cross-validation scores; it is identically equal to a sum over all non-empty subsets of the data. In the compact form of Eq. (11), the log-likelihood is $\\sum_{k=1}^{d} \\binom{d}{k}^{-1}$ times, for each ordered $k$-tuple of data, the average over cyclic permutations of $\\log P(D_{i_1} \\mid D_{i_2}\\cdots D_{i_k} H I)$. Eq. (12) rewrites the same identity as, for each datum, the average over all subsets of the remaining data of predicting that datum from the subset. The paper claims the derivation is fully general: it does not assume exchangeability or a particular sampling model, it works in any logarithmic base, and the same recursion yields generalizations to leave-$k$-out and $k$-fold cross-validation log-scores.","pith_inferences":["The paper leaves the infinite-data limit open; if the identity extends, the log-likelihood would become an integral over all data subsets of predictive scores rather than a finite sum.","The identity suggests a practical debugging tool not discussed in the paper: for small $d$, compute the weighted sum over all subset log-scores and compare it with the directly computed log-likelihood; a mismatch would localize errors in cross-validation code.","The same algebraic pattern may apply to other predictive scores built from conditional probabilities, such as leave-all-but-one-out scores or predictive likelihoods, yielding a family of exact decompositions.","In model-comparison practice, the result implies that disagreements between leave-one-out cross-validation and log-likelihood rankings must be attributable to the weights given to different subset sizes, a question the paper does not resolve."],"forward_implications":["The leave-one-out cross-validation log-score is not an alternative to the log-likelihood but one term in its exact decomposition; comparing them is comparing a piece with the whole.","Any ranking of models by log-likelihood is implicitly a ranking by the aggregate of all subset log-scores.","Because the recursion is purely logical, the identity holds for every hypothesis and every finite data set, independent of ordering, exchangeability, or sampling assumptions.","The same expansion applies to leave-k-out and k-fold cross-validation log-scores, so the log-likelihood can be represented as an average over all such scores, with weights that the paper states for leave-one-out but leaves partly unspecified for general folds.","The paper's closing remark is a direct corollary: the individual subset scores are expensive to compute, yet their weighted sum is the cheap log-likelihood."],"supporting_citations":[{"why":"Defines and motivates the leave-one-out cross-validation log-score that the identity averages over all data subsets.","marker":"12"},{"why":"Provides the predictive model-selection precedents for cross-validation log-scores that the identity reframes.","marker":"12"},{"why":"Surveys Bayesian predictive comparison methods that use log-scores and cross-validation.","marker":"12"},{"why":"Sets out the product rule of probability used in every recursive step of the expansion.","marker":"1"}],"fun_headline_variants":["Log-likelihood is an average of all leave-one-out scores","CV leave-one-out scores sum to log-likelihood","Exact identity: log-likelihood as CV average","Log-likelihood rewritten as subset score average","Log-likelihood from all subset predictions"],"cache_read_input_tokens":8832,"weakest_assumption_plain":"The identity as written requires that every conditional probability appearing inside a logarithm be strictly positive, so all logs are finite; it also presumes the data form a finite list, since no limit or infinite-product convention is supplied.","fun_headline_variants_meta":{"raw":{"variants":["Log-likelihood is an average of all leave-one-out scores","CV leave-one-out scores sum to log-likelihood","Exact identity: log-likelihood as CV average","Log-likelihood rewritten as subset score average","Log-likelihood from all subset predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3252,"prompt_tokens":772,"completion_tokens":2480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":388,"completion_tokens_details":{"reasoning_tokens":2412}},"tokens_in":388,"tokens_out":2480,"duration_ms":18697,"temperature":1.0,"reasoning_tokens":2412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:31:41.372607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a concrete model and a small dataset (say $d=2$ or $d=3$) with strictly positive predictive probabilities, and evaluate both sides of Eq. (11) numerically; any mismatch beyond rounding refutes the claimed weights. Alternatively, choose a model in which some conditional probability is exactly zero: the right-hand side then contains $\\log 0$ while the left-hand side may be finite, showing that the stated identity depends on the positivity convention the paper does not specify.","supporting_citations":[{"cited_title":"- M., DeGroot, M","cited_arxiv_id":null,"evidence_quote":"Sets out the product rule of probability used in every recursive step of the expansion."}],"review_version":1}