{"id":"bd22fb91-2bf5-4fde-b837-53fe1b292a86","arxiv_id":"2607.10139","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A panel of independent LLMs whose answers agree is a stronger, training-free verifier than self-consistency or process reward models, with a closed-form law predicting its accuracy and its shared-error ceiling.","lead":"This paper tests whether asking several differently trained language models to solve the same problem, then taking their majority answer, can act as a free, label-free verifier for reasoning. Across seven benchmarks this 'LLM-jury' selects correct answers better than self-consistency and matches or beats trained reward models, and a closed-form formula predicts when the method can be trusted.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Law's one-factor difficulty model conflates shared difficulty with shared blind spots; floor prediction may not port to new domains.","rationale":"The reader's conditional verdict already flags the one-factor assumption. I agree; the strongest empirical counterevidence is the paper's own GPQA case study, which reveals a mechanism—shared content-level conventions—that is not obviously equivalent to independent draws from a common attractor distribution. However, the paper's measured floor predictions match the observed values within 0.009, so the concern is not proven; it is a risk that requires a direct test on a larger sample of unanimous-wrong events. The check would settle whether the law's central output, the shared-error floor, is a trustworthy forecast or an artifact of fitting a too-flexible-but-wrong model to a handful of extremes. If the test rejects the model, the paper's 'predictable' and 'quantified ceiling' claims would be invalid, though the empirical selector comparisons would remain valuable.","tokens_in":25561,"tokens_out":7671,"duration_ms":85043,"concrete_test":"On a held-out benchmark with enough unanimous-wrong cases (e.g., MMLU-Pro, n=1000, or a purpose-collected science QA set), estimate the latent difficulty p for each problem (e.g., via the panel's empirical agreement level or an item-response model). Then test the model's key factorization: among problems where all M panel members are wrong, the probability that they all select the same attractor should equal Σ s_j^M, with s_j measured on the same data. A bootstrap test: sample problems, compute the observed conditional unanimous-wrong rate, and compare it to the model's predicted rate Σ s_j^M with uncertainty. Alternatively, perform a chi-square goodness-of-fit on the full multinomial distribution of the panel's answer classes (correct vs. attractor 1..K vs. idiosyncratic) across problems, binned by estimated p. If the observed co-occurrence of identical wrong answers exceeds the predict","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.2's law assumes a single latent difficulty p fully explains error correlation, so conditional on p and on being wrong, panel members independently choose wrong answers from a measured attractor distribution. This yields Floor = Σ s_j^M E[(1-p)^M]. But the paper's own GPQA Appendix D shows unanimous-wrong cases come from shared content-level heuristics (e.g., molar-enthalpy unit). Such shared blind spots imply that conditional on the same problem, a wrong model's choice is not independent of others' choices; the same misconception is triggered across models beyond what a scalar p or marginal s_j captures. On GPQA the unanimous-wrong rate is estimated from 6 problems—too few to distinguish independence from correlation of wrong-answer choices, and the 0.009 floor MAE masks this. If the model understates the conditional co-occurrence of shared wrong answers on science/knowledge benchmarks, the predicted ceiling (the paper's headline 'quantified ceiling' claim) would be too optimistic exactly where the method is promoted as superior to trained verifiers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes cross-model consensus as a training-free verifier for best-of-N selection in LLM reasoning. A panel of independently trained models each solves a problem once, and the modal answer (with the agreement fraction as a confidence signal) selects from a fixed candidate pool. Across seven benchmarks the method is reported to outperform self-consistency and a single-model LLM-verifier, to match or exceed trained process/outcome reward models in-domain and outperform them out-of-domain, and to be governed by a closed-form 'parameter-free law' that predicts consensus accuracy and a shared-error floor from three measured panel statistics (mean accuracy a, pairwise error correlation ρ, and shared-attractor mass s). The experimental design is strong: fixed candidate pools, identical grading, paired bootstraps, generator robustness, an open-weight ablation, and a fairness gate for trained verifiers. The law's closed forms match Monte-Carlo simulation to within 0.002, and the paper is unusually transparent about prompts, limitations, and per-cell intervals.","tokens_in":25815,"tokens_out":7014,"duration_ms":77467,"significance":"If the claims hold, this is a practically important result: a label-free selector that rivals trained verifiers with no training cost, plus an analytic characterization of when agreement can be trusted. The verifier comparison is well controlled (fixed pool, matched grading, generator-robustness, open-weight ablation), and the paper makes several falsifiable predictions (the consensus-accuracy and floor curves) that are tested against empirical data. The law's closed forms are mathematically coherent and validated against simulation. The main risk is that the law's one-factor latent-difficulty model cannot distinguish 'shared difficulty' from 'shared blind spots'; this matters for the headline 'quantified ceiling' claim, though it does not by itself undermine the core verifier comparison.","major_comments":[{"comment":"The law presupposes that conditional on latent difficulty p, model errors — and choices among wrong answers — are independent given the measured attractor mass profile. Appendix D's GPQA case study documents shared content-level conventions (molar-enthalpy unit, absorption-line tracer, stereochemistry descriptors) that induce dependence among wrong choices beyond a scalar p or marginal s. The same observed agreement distribution is compatible with a 'shared difficulty' model (which the law fits) and a 'shared blind spot' model in which the joint probability of all models landing on the same wrong answer is much higher than the product of marginals. The floor is therefore not identified from the measured statistics, and the claim that the law 'exposes the method's ceiling' out of domain is not supported. A concrete test: hold out a content area, measure the conditional co-occurrence of wr","section":"§3.2, Eq. (1)"},{"comment":"The 'parameter-free law' measures (a, ρ, s) on the same ground-truth-labeled data whose consensus behavior is the prediction target. The statement in Appendix B that 'every benchmark is effectively held out' conflates 'no fitted parameters' with 'out-of-sample.' The leave-one-benchmark-out test in Appendix C transfers only the wrong-answer profile, not the conditional-independence assumption, so it does not address the identifiability problem. Please provide a clean calibration/prediction split: estimate the inputs on a random subset of problems and predict consensus behavior on the held-out remaining problems, and also report the prediction for a benchmark whose inputs are not re-measured. This would strengthen the 'predictable in advance' claim considerably.","section":"Appendix B / §5.2"},{"comment":"The shared-error floor on GPQA (0.030) rests on 6 unanimous-wrong problems out of 198. With n=198, a binomial 95% CI for a 0.030 rate is roughly [0.011, 0.064], so the floor is known only to a factor of about 3. The reported floor MAE of 0.009 does not convey this uncertainty, and the paper's abstract and introduction lean on the floor as a 'quantified ceiling.' Please report a confidence interval on the floor estimate (e.g., from the existing bootstrap machinery used for Table 6), and state explicitly that the GPQA floor measurement is a small-sample estimate. The AIME-2024 residual (predicted 0.011 vs empirical 0.000) also deserves a sentence, since it is outside the headline MAE.","section":"§5.2 / Table 2"},{"comment":"The abstract's statement that 'on competition math it closes the entire gap to an oracle selector' is true only for AIME-2024 in Table 1. On the other competition-math benchmarks the jury captures a fraction of the oracle gap: AIME-2025 40.0 vs 43.3 (33%), MATH-500 96.6 vs 98.6 (41%), OlympiadBench 72.6 vs 79.1 (53%). Please qualify the claim to 'on AIME-2024 it closes the entire gap' or report the average fraction of the oracle gap across the competition-math benchmarks. The current wording is a material overstatement of the results.","section":"Abstract / §5.1"}],"minor_comments":[{"comment":"Eq. (1) is written for a single attractor s, but the reported predictions use the measured multi-attractor profile (s_1,...,s_K). The main text should define both notations in one place and state that the scalar form is the interpretable single-misconception special case; the current transition between 'the predictions we report use this measured profile' and Eq. (1) is confusing.","section":"§3.2, Eq. (1)"},{"comment":"The text says grading uses 'math verify' and a 'boxed-answer parser'; for reproducibility, please name the exact package version or provide the parser source in the supplementary material. Also, 'Mathematical Association of America' is listed as the author of the AIME reference; if this is a dataset, the citation should follow the benchmark's preferred format.","section":"Reproducibility / Appendix A"},{"comment":"In the 'Why It Works' panel, the self-consistency example shows four Qwen samples with three '23' and one '17' and the cross-model panel with three '17' and one '23'. The figure is clear, but the self-consistency answer '23' is shown as wrong in a green/red color scheme; consider making the color convention explicit in the caption.","section":"Figure 1"},{"comment":"The term 'parameter-free law' is used for the closed form, but the inputs are benchmark-specific measurements. Consider renaming to 'closed-form law with measured inputs' throughout, to avoid implying the law is universal or requires no calibration. The abstract already says 'derived in closed form'; the phrase 'parameter-free' is misleading.","section":"Terminology"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about many limitations and the central verifier comparison is solid. The main issue is that the law's floor prediction is not identified: the one-factor difficulty model cannot separate shared difficulty from shared blind spots, and the GPQA floor rests on a tiny sample. The abstract's oracle-gap claim overstates the results. Both are fixable without new experiments, though a calibration/prediction split would substantially strengthen the law's 'predictable in advance' claim. The verifier comparison itself does not depend on the law, so I do not see a fatal flaw; major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper with a slightly oversold theoretical wrapper. If you work on test-time scaling and want a label-free selector, cross-model consensus is real: on the fixed candidate pools used here it beats self-consistency and matches or beats four trained verifiers, and the controls are genuinely careful — same pools, same grader, paired bootstraps, generator-robustness, a fully open-weight ablation, and a fairness gate for the PRMs. Credit where due: the closed-form law is neat, the simulation match is clean, and the paper explicitly separates the known claim that voting helps from the stronger claim that the signal is predictable and has a quantified ceiling.\n\nSoft spots are real but don't sink the main result. First, the law is parameter-free only in a narrow sense: no parameter is fit to the consensus accuracy it predicts, but its inputs (a, rho, s) are measured on ground-truth-labeled data from the same benchmark. Appendix B concedes the law is calibrated from a labeled sample, not label-free to calibrate. The leave-one-benchmark-out test helps, but it is still in-family calibration. Second, the shared-error floor — the headline ceiling — is estimated from very few events. On GPQA, the unanimous-wrong count is six problems, two of which the paper itself regrades as grading artifacts. The stress-test concern is fair: the one-factor difficulty model assumes that conditional on difficulty, wrong-answer choices are independent except for a shared attractor. Shared blind spots like the molar-enthalpy convention could couple errors more strongly than a scalar s captures, which would make the predicted floor too optimistic exactly where the method is promoted as superior to trained verifiers. Third, no code or data artifacts are released.\n\nThese are bounded concerns. The verifier comparison does not depend on the law, and the paper honestly discusses the weak single-model self-scoring baseline in Appendix K. The reader's conditional verdict is about right.\n\nI would send this to serious peer review. The empirical core deserves testing against other pools and generators, and the law's assumptions deserve scrutiny. A good revision would add an out-of-domain calibration test — new domain, labeled set, check the floor — and state plainly that the law is calibrated, not self-contained.","headline":"Solid controlled evidence for cross-model consensus as a label-free verifier; the predictive law is neat but in-sample calibrated, and the shared-error floor rests on very few cases.","tokens_in":26282,"tokens_out":3054,"would_cite":true,"duration_ms":37534,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When several independently trained models each solve a problem once, the majority-agreed answer proves a better selector than self-consistency, self-scoring, or trained reward models outside their domain, and a closed-form law predicts its","keywords":["cross-model consensus","LLM-jury","verifier","self-consistency","reward models","error decorrelation","shared-error floor","parameter-free law"],"falsifier":"Take a domain with a measured non-zero floor (e.g. graduate science), stratify problems by estimated difficulty, and within each stratum compute the pairwise error correlation of two independently trained models after removing the shared-attractor mass s. If residual correlation is significantly positive, the one-factor law underpredicts the floor; if it is zero, the independence assumption is supported.","tokens_in":25434,"feed_emoji":"⚖️","tokens_out":7309,"duration_ms":78363,"temperature":0.7,"pith_summary":"The paper tries to establish that agreement among independently trained models—an \"LLM-jury\"—is a verifier for reasoning selection that needs no labels and no training. Each panel member solves a problem once, never seeing another's work; the modal answer class is selected, and the agreement fraction serves as a confidence signal. This is claimed to beat resampling one model (self-consistency) and a model scoring its own candidates, and to match or outperform trained reward models especially outside their training distribution. The paper also derives a parameter-free closed-form law that predicts consensus accuracy and a \"shared-error floor\" from three measured panel statistics, so one can know in advance when to trust the jury and where it cannot help. A sympathetic reader would care because test-time scaling is currently bottlenecked by selection, and this is a free, general, describable selection signal.","feed_headline":"Independent LLM juries beat trained reward models at picking answers","feed_subtitle":"Why care: the signal costs nothing at inference, needs no labels, and a closed-form law says when to trust it.","key_machinery":"The LLM-jury: M independently trained models each solve a problem once, answers are binned into equivalence classes, and the modal class is selected, with agreement fraction g as a confidence signal. The load-bearing object is the closed-form law: per-problem difficulty p ∼ Beta(ak, (1−a)k) with k = 1/ρ − 1 couples errors, and wrong answers fall on a shared attractor with probability s; marginalizing over p yields beta-binomial consensus accuracy and the floor Floor = s^M E[(1−p)^M]. It converts three measured panel statistics into a full prediction of when agreement can be trusted and where it must fail.","core_discovery":"Independently trained models err differently, so on a fixed candidate pool their wrong answers scatter while the correct answer accumulates agreement; this, the paper claims, makes cross-model agreement a more accurate verifier than resampling one model or having a model score its own candidates. The discovery is a parameter-free law: from measured mean member accuracy a, mean pairwise error correlation rho, and shared-misconception rate s, closed forms predict consensus accuracy and the shared-error floor (Eq. 1), the rate at which the whole panel is unanimous and wrong. Empirically the law matches consensus accuracy to mean absolute error 0.03 across seven benchmarks, the jury closes the f","pith_inferences":["Beyond the paper, the law turns panel design into a measurable trade-off: on a small labeled calibration set one could estimate (a, rho, s) for available models and choose the panel that minimizes rho and s rather than maximizing individual accuracy.","If the shared-error floor is a stable property of a domain, it can serve as an audit for model families: a high floor reveals a shared convention or blind spot that no agreement signal can catch, pointing directly at data or training that would diversify errors.","The paper's rejection-sampling result suggests consensus rewards could be used while training, which would change the floor the law treats as fixed; tracking floor over generations would test whether training on peer agreement actually decorrelates errors or just aligns them."],"forward_implications":["On competition math, a four-model jury captured 100% of the achievable selection gain over self-consistency, while a single model scoring its own candidates captured essentially none.","The parameter-free law predicts consensus accuracy and shared-error floor from three measured statistics, to mean absolute error 0.03 and 0.009, and transfers leave-one-benchmark-out.","On an identical candidate pool, the jury matches the strongest trained verifier inside its math domain and is the top selector outside it, at zero training cost.","Consensus doubles as an abstention dial: a unanimous panel is 99.5% correct on MATH-500 while answering 85% of problems, but on science benchmarks the shared-error floor caps unanimous accuracy below one.","Matched-budget experiments show four decorrelated models beat 32 samples of one model; pairwise error correlation falls from 0.68 within-model to 0.47 cross-family, the ordering that predicts selection accuracy."],"fun_headline_variants":["Free LLM jury beats trained verifiers on reasoning","LLM jury: consensus beats self-scoring at zero cost","Why LLM juries beat reward models: error decorrelation","A law predicts when LLM consensus beats any reward model","LLM consensus: a free verifier that beats trained models"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The law assumes that a single latent difficulty factor explains all error correlation among panel members, so that once difficulty is fixed models' mistakes and wrong-answer choices are independent; if independently trained models share a blind spot that survives after controlling for difficulty, the predicted agreement distribution and shared-error floor are miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["Free LLM jury beats trained verifiers on reasoning","LLM jury: consensus beats self-scoring at zero cost","Why LLM juries beat reward models: error decorrelation","A law predicts when LLM consensus beats any reward model","LLM consensus: a free verifier that beats trained models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000976,"raw_usage":{"total_tokens":4028,"prompt_tokens":835,"completion_tokens":3193,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":3110}},"tokens_in":579,"tokens_out":3193,"duration_ms":22781,"temperature":1.0,"reasoning_tokens":3110,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:21:18.607023+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a domain with a measured non-zero floor (e.g. graduate science), stratify problems by estimated difficulty, and within each stratum compute the pairwise error correlation of two independently trained models after removing the shared-attractor mass s. If residual correlation is significantly positive, the one-factor law underpredicts the floor; if it is zero, the independence assumption is supported.","supporting_citations":[],"review_version":2}