Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative

Koen Oostermeijer

Bayesian accuracy scores multiple-choice answers with a length prior so neither short nor long completions are systematically favored.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 03:29 UTC pith:OCZA23FA

load-bearing objection Useful methods note on length bias in MC log-prob scoring, but abstract-only so the prior and bias claims are still uncheckable. the 3 major comments →

arxiv 2607.12767 v1 pith:OCZA23FA submitted 2026-07-14 cs.AI

Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative

Koen Oostermeijer This is my paper

classification cs.AI

keywords length biasmultiple-choice evaluationBayesian accuracylength normalizationlog-probability scoringlanguage-model benchmarksfew-shot evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multiple-choice language-model benchmarks usually rank answers by the sum of token log-probabilities. Because that sum grows (or shrinks) with the number of tokens, longer candidates are routinely under-scored. Dividing by length is the usual fix, yet the paper shows that this common heuristic often overshoots and begins to reward longer answers instead. The authors characterize exactly when each of the two classical rules is unbiased, then introduce Bayesian accuracy: the posterior probability of each candidate under an explicit prior over answer length. The new score cancels linear length effects, needs no extra model calls, and, on the benchmarks they examine, shows lower residual length bias than both plain and length-normalized accuracy in zero-shot and few-shot regimes.

Core claim

Bayesian accuracy—the posterior probability of each candidate completion under an explicit prior over answer length—is a drop-in replacement for likelihood-based multiple-choice scoring that removes linear length effects and consistently exhibits lower empirical length bias than both standard accuracy and length-normalized accuracy.

What carries the argument

Bayesian accuracy: the posterior P(candidate | length prior) obtained by combining the model’s likelihood with an explicit prior over completion length, thereby cancelling the linear length term that appears in ordinary log-probability sums.

Load-bearing premise

That the chosen prior over answer length correctly cancels linear length effects without introducing new systematic distortions under the length distributions that actually appear in the evaluated benchmarks.

What would settle it

On a held-out multiple-choice suite whose answer-length distribution differs from the training or calibration set, measure the correlation between chosen-answer length and correctness; if Bayesian accuracy still shows a statistically larger absolute correlation than the better of the two classical rules, the claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful methods note on length bias in MC log-prob scoring, but abstract-only so the prior and bias claims are still uncheckable.

read the letter

The one thing worth knowing: this is a clean methods contribution on a real, everyday evaluation artifact—length bias when ranking MC completions by conditional log-prob—and it proposes Bayesian accuracy (posterior under an explicit length prior) as a drop-in fix that needs no extra forward passes. The abstract also claims an analysis of when raw vs length-normalized accuracy are appropriate, and that normalization often over-corrects toward longer answers. That framing is right; the problem and the failure mode of length-norm are known, and a principled prior-based alternative is a reasonable next step.

What looks solid on the page we have: the problem statement is accurate, the motivation is practical (leaderboards, few-shot MC), and the proposed rule is described as zero-extra-compute. If the full paper delivers the analysis of length distributions, a fixed non-circular prior, and consistent bias reduction across benchmarks, it is the kind of small, usable fix people actually cite when they write eval code.

Soft spots, in proportion: we only have the abstract. Prior family, parameters, and how the prior is chosen are unspecified. If that prior is fitted to the same length distribution used to measure bias, the “lower bias” result is partly by construction—the stress-test and the reader both flag this correctly, and it is the load-bearing open question. Bias metric, datasets, and any residual length dependence after the posterior are also invisible. So the drop-in-replacement claim is coherent but not yet verified. That is not a reason to dismiss the idea; it is a reason not to treat the abstract as evidence.

Who it is for: people who run or design LLM multiple-choice evals and care about scoring artifacts. Not a capability paper. I would bring it to a methods reading group once the full text is up, and I would accept it for peer review rather than desk-reject—the problem is real and the proposed fix is concrete enough to deserve a referee. I would not cite it yet on abstract alone. Serious thinking on a narrow, useful problem; verdict conditional on the prior not being circular and the empirics holding up.

Referee Report

3 major / 0 minor

Summary. The manuscript studies length bias in likelihood-based multiple-choice evaluation of language models: ranking by conditional log-probability tends to penalize longer completions, while length-normalized scoring often over-corrects toward longer answers. It analyzes when each rule is appropriate as a function of the completion-length distribution, then proposes Bayesian accuracy—a drop-in scoring rule that ranks candidates by posterior probability under an explicit prior over answer length, thereby removing linear length effects without extra forward passes. The abstract claims this rule consistently shows lower empirical length bias than both standard and length-normalized accuracy across benchmarks and few-shot settings.

Significance. If the analysis and empirical claims hold, Bayesian accuracy would be a practically useful, zero-overhead replacement for the two dominant MC scoring heuristics, with clearer theoretical grounding for when length effects are removed versus merely re-signed. That would matter for fairer leaderboard comparisons and for few-shot evaluation design. The abstract-only evidence, however, does not yet establish that the prior is non-circular or that residual length dependence is actually reduced under a well-defined bias metric, so the significance remains conditional on those details.

major comments (3)

The abstract’s central claim—that an explicit prior over answer length removes linear length effects and yields lower empirical length bias—cannot be assessed without the prior family, its parameters, and the selection procedure. If the prior is estimated from the same completion-length distribution used to measure bias reduction, the reported improvement is partly by construction and may not generalize. The manuscript must state whether the prior is fixed a priori (and on what data) or fitted per benchmark, and must show that residual length dependence does not reappear under the actual length distributions of the evaluated sets.
The abstract asserts that Bayesian accuracy “consistently exhibits lower empirical length bias” than standard and length-normalized accuracy, but neither the bias metric nor any quantitative results (tables, error bars, ablations) are available in the provided material. Without a precise definition of the length-bias measure and the corresponding experimental evidence, the comparative claim is not verifiable and remains load-bearing for the paper’s contribution.
The promised analysis of when standard versus length-normalized accuracy are appropriate, and how their biases depend on the completion-length distribution, is only summarized in the abstract. That characterization is the stated motivation for the Bayesian rule; without the derivations (or at least the key equations and assumptions), it is impossible to confirm that the posterior scoring rule is free of the linear effects it claims to cancel, or that it does not introduce other systematic distortions.

Circularity Check

0 steps flagged

Abstract-only review: no equations or prior-selection procedure available to exhibit circular reduction; claimed Bayesian accuracy is presented as a principled scoring rule, not as a fit to the evaluation length distribution.

full rationale

Only the abstract is available. It states that Bayesian accuracy computes the posterior under an explicit prior over answer length, thereby removing linear length effects, and reports lower empirical length bias than standard and length-normalized accuracy. No equations, no prior family, no fitting procedure, and no self-citations appear in the provided text. Circularity of the kinds enumerated (self-definitional, fitted-input-called-prediction, load-bearing self-citation, uniqueness imported from authors, ansatz smuggled via citation, renaming of a known result) can be asserted only when a specific reduction can be quoted and exhibited. With no such material, the honest finding is no demonstrated circularity. The reader's concern that the prior might have been fitted to the same length distribution used to measure bias is a legitimate correctness/generalization risk, but it is not evidenced circularity under the hard rules; it remains speculation pending the full paper. Score 0 is therefore required.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

Abstract-only review: free parameters and axioms are inferred from the stated method. The central claim rests on an explicit prior over answer length (family and parameters unspecified), on the modeling assumption that length effects are linear/additive in log-probability space, and on empirical length-bias measurements whose metric is not given. No invented physical entities; the 'Bayesian accuracy' score is a derived scoring rule, not a new ontological object.

free parameters (2)

length_prior_parameters
Any parametric prior over answer length (e.g., mean/variance of a length distribution) must be chosen or estimated; the abstract does not state values or estimation procedure.
length_bias_metric_definition
Empirical 'length bias' comparisons depend on how bias is quantified (correlation of score with length, accuracy gap by length bins, etc.); that definition is free until specified.

axioms (3)

domain assumption Conditional log-probability of a completion is the sum of per-token log-probabilities, inducing a systematic dependence on completion length.
Standard in autoregressive LM scoring; stated as the source of length bias in the abstract.
ad hoc to paper An explicit prior over answer length yields a posterior that removes linear length effects for ranking candidates.
Core modeling claim of Bayesian accuracy; correctness depends on prior choice and on length effects being well-captured by that prior.
domain assumption Length-normalized accuracy frequently over-corrects toward longer answers depending on the distribution of completion lengths.
Empirical premise motivating the new rule; abstract asserts it but does not show the supporting analysis.

invented entities (1)

Bayesian accuracy (scoring rule) no independent evidence
purpose: Rank multiple-choice candidates by posterior probability under a length prior so linear length effects are removed.
New named scoring rule introduced in the abstract; independent_evidence is false because no external falsifiable handle (e.g., public code or fixed prior) is provided in the abstract.

pith-pipeline@v1.1.0-grok45 · 6056 in / 2483 out tokens · 17754 ms · 2026-07-15T03:29:50.322886+00:00 · methodology

0 comments

read the original abstract

Multiple-choice benchmarks that rank candidate completions by conditional log-probability suffer from a length bias: because log-probabilities sum over tokens, longer answers tend to be penalized relative to shorter ones in practice. A common mitigation is to normalize scores by completion length, but we show empirically that this heuristic frequently over-corrects, introducing a bias toward longer answers instead. We first analyze these scoring rules, characterizing when standard and length-normalized accuracy are appropriate and how their length biases depend on the distribution of completion lengths. Motivated by this analysis, we introduce \emph{Bayesian accuracy}, a scoring rule that computes the posterior probability of each candidate under an explicit prior over answer length, thereby removing linear length effects. Bayesian accuracy is a drop-in replacement for likelihood-based multiple-choice evaluation, requires no additional forward passes, and consistently exhibits lower empirical length bias than both standard and length-normalized accuracy across benchmarks and few-shot settings.

Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative

Core claim

What carries the argument

Load-bearing premise

What would settle it

discussion (0)