{"id":"52b66536-59ae-4f15-9f1a-87d7847705f6","arxiv_id":"2607.12767","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Bayesian accuracy scores multiple-choice completions with an explicit length prior, reducing empirical length bias versus raw and length-normalized log-probability accuracy.","lead":"Multiple-choice LLM benchmarks that score answers by log-probability are biased by answer length; length-normalization often over-corrects the other way. The paper proposes Bayesian accuracy, which uses an explicit prior over length to remove linear length effects without extra model runs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Abstract-only review leaves the prior family and its selection unspecified, so the claim that Bayesian accuracy removes linear length effects without reintroducing bias cannot be verified.","rationale":"The reader’s weakest-assumption diagnosis is exactly the load-bearing gap: an unspecified length prior whose selection procedure is invisible in the abstract. No stronger technical objection can be raised without equations, tables, or code. The proposed concrete test directly probes whether that prior is non-circular and robust; until it is performed the CONDITIONAL verdict with low confidence remains appropriate. No formal verification or artifacts are present to raise the assessment.","tokens_in":1977,"tokens_out":470,"duration_ms":5475,"concrete_test":"Once the full paper is available, extract the exact prior p(L) and the formula for the posterior score (presumably proportional to p(completion|prompt)·p(L)). Recompute the reported accuracy and length-bias statistics on at least one public benchmark (e.g., MMLU or ARC) after replacing the paper’s prior with a deliberately misspecified alternative (uniform over observed lengths, or a geometric with mean shifted by ±50 %). If the bias ranking reverses or the accuracy gap shrinks by more than a few points, the claimed robustness is prior-dependent and the central claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Bayesian accuracy, by placing an explicit prior over answer length, removes linear length effects and yields lower empirical length bias than both unnormalized and length-normalized accuracy, all without extra forward passes. Because only the abstract is available, the prior family, its parameters, and the procedure used to choose them are never stated. The reader correctly flags this as the weakest assumption: if the prior is fitted to the same length distribution that appears in the evaluation benchmarks, the reported bias reduction is circular and may not generalize. Even a fixed parametric form (e.g., geometric or Poisson) can re-introduce systematic distortions once the true length distribution of the benchmark departs from the prior; without the equations or the bias metric used in the paper, it is impossible to confirm that the posterior scores are free of residual length dependence. Thus the drop-in-replacement claim rests on an uninspectable modeling choice.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript studies length bias in likelihood-based multiple-choice evaluation of language models: ranking by conditional log-probability tends to penalize longer completions, while length-normalized scoring often over-corrects toward longer answers. It analyzes when each rule is appropriate as a function of the completion-length distribution, then proposes Bayesian accuracy—a drop-in scoring rule that ranks candidates by posterior probability under an explicit prior over answer length, thereby removing linear length effects without extra forward passes. The abstract claims this rule consistently shows lower empirical length bias than both standard and length-normalized accuracy across benchmarks and few-shot settings.","tokens_in":2153,"tokens_out":696,"duration_ms":12169,"significance":"If the analysis and empirical claims hold, Bayesian accuracy would be a practically useful, zero-overhead replacement for the two dominant MC scoring heuristics, with clearer theoretical grounding for when length effects are removed versus merely re-signed. That would matter for fairer leaderboard comparisons and for few-shot evaluation design. The abstract-only evidence, however, does not yet establish that the prior is non-circular or that residual length dependence is actually reduced under a well-defined bias metric, so the significance remains conditional on those details.","major_comments":[{"comment":"The abstract’s central claim—that an explicit prior over answer length removes linear length effects and yields lower empirical length bias—cannot be assessed without the prior family, its parameters, and the selection procedure. If the prior is estimated from the same completion-length distribution used to measure bias reduction, the reported improvement is partly by construction and may not generalize. The manuscript must state whether the prior is fixed a priori (and on what data) or fitted per benchmark, and must show that residual length dependence does not reappear under the actual length distributions of the evaluated sets.","section":null},{"comment":"The abstract asserts that Bayesian accuracy “consistently exhibits lower empirical length bias” than standard and length-normalized accuracy, but neither the bias metric nor any quantitative results (tables, error bars, ablations) are available in the provided material. Without a precise definition of the length-bias measure and the corresponding experimental evidence, the comparative claim is not verifiable and remains load-bearing for the paper’s contribution.","section":null},{"comment":"The promised analysis of when standard versus length-normalized accuracy are appropriate, and how their biases depend on the completion-length distribution, is only summarized in the abstract. That characterization is the stated motivation for the Bayesian rule; without the derivations (or at least the key equations and assumptions), it is impossible to confirm that the posterior scoring rule is free of the linear effects it claims to cancel, or that it does not introduce other systematic distortions.","section":null}],"minor_comments":[],"recommendation":"uncertain","confidential_remarks":"This is an abstract-only review: the full text was not available. The recommendation is therefore uncertain rather than a substantive accept/reject. Once the full manuscript is supplied, the load-bearing checks are (1) prior specification and whether it is fitted to evaluation lengths, (2) the formal bias metric and tables, and (3) the length-distribution analysis. If those are sound and non-circular, the contribution looks publishable; if the prior is data-dependent without disclosure, the bias-reduction claim would need major revision or rejection."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this is a clean methods contribution on a real, everyday evaluation artifact—length bias when ranking MC completions by conditional log-prob—and it proposes Bayesian accuracy (posterior under an explicit length prior) as a drop-in fix that needs no extra forward passes. The abstract also claims an analysis of when raw vs length-normalized accuracy are appropriate, and that normalization often over-corrects toward longer answers. That framing is right; the problem and the failure mode of length-norm are known, and a principled prior-based alternative is a reasonable next step.\n\nWhat looks solid on the page we have: the problem statement is accurate, the motivation is practical (leaderboards, few-shot MC), and the proposed rule is described as zero-extra-compute. If the full paper delivers the analysis of length distributions, a fixed non-circular prior, and consistent bias reduction across benchmarks, it is the kind of small, usable fix people actually cite when they write eval code.\n\nSoft spots, in proportion: we only have the abstract. Prior family, parameters, and how the prior is chosen are unspecified. If that prior is fitted to the same length distribution used to measure bias, the “lower bias” result is partly by construction—the stress-test and the reader both flag this correctly, and it is the load-bearing open question. Bias metric, datasets, and any residual length dependence after the posterior are also invisible. So the drop-in-replacement claim is coherent but not yet verified. That is not a reason to dismiss the idea; it is a reason not to treat the abstract as evidence.\n\nWho it is for: people who run or design LLM multiple-choice evals and care about scoring artifacts. Not a capability paper. I would bring it to a methods reading group once the full text is up, and I would accept it for peer review rather than desk-reject—the problem is real and the proposed fix is concrete enough to deserve a referee. I would not cite it yet on abstract alone. Serious thinking on a narrow, useful problem; verdict conditional on the prior not being circular and the empirics holding up.","headline":"Useful methods note on length bias in MC log-prob scoring, but abstract-only so the prior and bias claims are still uncheckable.","tokens_in":2781,"tokens_out":530,"would_cite":false,"duration_ms":5980,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Bayesian accuracy scores multiple-choice answers with a length prior so neither short nor long completions are systematically favored.","keywords":["length bias","multiple-choice evaluation","Bayesian accuracy","length normalization","log-probability scoring","language-model benchmarks","few-shot evaluation"],"falsifier":"On a held-out multiple-choice suite whose answer-length distribution differs from the training or calibration set, measure the correlation between chosen-answer length and correctness; if Bayesian accuracy still shows a statistically larger absolute correlation than the better of the two classical rules, the claim fails.","tokens_in":2809,"feed_emoji":"⚖️","tokens_out":505,"duration_ms":5016,"temperature":0.7,"pith_summary":"Multiple-choice language-model benchmarks usually rank answers by the sum of token log-probabilities. Because that sum grows (or shrinks) with the number of tokens, longer candidates are routinely under-scored. Dividing by length is the usual fix, yet the paper shows that this common heuristic often overshoots and begins to reward longer answers instead. The authors characterize exactly when each of the two classical rules is unbiased, then introduce Bayesian accuracy: the posterior probability of each candidate under an explicit prior over answer length. The new score cancels linear length effects, needs no extra model calls, and, on the benchmarks they examine, shows lower residual length bias than both plain and length-normalized accuracy in zero-shot and few-shot regimes.","feed_headline":"Bayesian score ends length bias in multiple-choice LM tests","feed_subtitle":"A length prior turns log-probabilities into posteriors that favor neither short nor long answers","key_machinery":"Bayesian accuracy: the posterior P(candidate | length prior) obtained by combining the model’s likelihood with an explicit prior over completion length, thereby cancelling the linear length term that appears in ordinary log-probability sums.","core_discovery":"Bayesian accuracy—the posterior probability of each candidate completion under an explicit prior over answer length—is a drop-in replacement for likelihood-based multiple-choice scoring that removes linear length effects and consistently exhibits lower empirical length bias than both standard accuracy and length-normalized accuracy.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Bayesian accuracy removes linear length bias in MC LM scoring","Length prior yields posteriors free of linear bias for candidates","Bayesian score cuts length bias vs standard and normalized accuracy","Drop-in Bayesian accuracy ends length bias in multiple-choice tests","Explicit length prior fixes over-correction from log-prob normalization"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"That the chosen prior over answer length correctly cancels linear length effects without introducing new systematic distortions under the length distributions that actually appear in the evaluated benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian accuracy removes linear length bias in MC LM scoring","Length prior yields posteriors free of linear bias for candidates","Bayesian score cuts length bias vs standard and normalized accuracy","Drop-in Bayesian accuracy ends length bias in multiple-choice tests","Explicit length prior fixes over-correction from log-prob normalization"]},"model":"grok-4.5","effort":"low","cost_usd":0.003882,"raw_usage":{"total_tokens":1173,"prompt_tokens":693,"num_sources_used":0,"completion_tokens":87,"cost_in_usd_ticks":38820000,"prompt_tokens_details":{"text_tokens":693,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":393,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":693,"tokens_out":87,"duration_ms":5254,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-15T03:29:50.322886+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out multiple-choice suite whose answer-length distribution differs from the training or calibration set, measure the correlation between chosen-answer length and correctness; if Bayesian accuracy still shows a statistically larger absolute correlation than the better of the two classical rules, the claim fails.","supporting_citations":[],"review_version":1}