LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

Ning Liu

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read When several independently trained models each solve a problem once, the majority-agreed answer proves a better selector than self-consistency, self-scoring, or trained reward models outside their domain, and a closed-form law predicts its

desk verdict Solid controlled evidence for cross-model consensus as a label-free verifier; the predictive law is neat but in-sample calibrated, and the shared-error floor rests on very few cases. read the letter →

arxiv 2607.10139 v3 pith:NHPKXX66 submitted 2026-07-11 cs.LG cs.AI

Ning Liu This is my paper · ORCID

classification cs.LGcs.AI

keywords cross-modelconsensusLLM-juryverifierself-consistencyrewardmodelserrordecorrelationshared-errorfloorparameter-freelaw

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that agreement among independently trained models—an "LLM-jury"—is a verifier for reasoning selection that needs no labels and no training. Each panel member solves a problem once, never seeing another's work; the modal answer class is selected, and the agreement fraction serves as a confidence signal. This is claimed to beat resampling one model (self-consistency) and a model scoring its own candidates, and to match or outperform trained reward models especially outside their training distribution. The paper also derives a parameter-free closed-form law that predicts consensus accuracy and a "shared-error floor" from three measured panel statistics, so one can know in advance when to trust the jury and where it cannot help. A sympathetic reader would care because test-time scaling is currently bottlenecked by selection, and this is a free, general, describable selection signal.

What carries the argument

The LLM-jury: M independently trained models each solve a problem once, answers are binned into equivalence classes, and the modal class is selected, with agreement fraction g as a confidence signal. The load-bearing object is the closed-form law: per-problem difficulty p ∼ Beta(ak, (1−a)k) with k = 1/ρ − 1 couples errors, and wrong answers fall on a shared attractor with probability s; marginalizing over p yields beta-binomial consensus accuracy and the floor Floor = s^M E[(1−p)^M]. It converts three measured panel statistics into a full prediction of when agreement can be trusted and where it must fail.

What would settle it

Take a domain with a measured non-zero floor (e.g. graduate science), stratify problems by estimated difficulty, and within each stratum compute the pairwise error correlation of two independently trained models after removing the shared-attractor mass s. If residual correlation is significantly positive, the one-factor law underpredicts the floor; if it is zero, the independence assumption is supported.

Watch

Extended reading notes

Core claim

Independently trained models err differently, so on a fixed candidate pool their wrong answers scatter while the correct answer accumulates agreement; this, the paper claims, makes cross-model agreement a more accurate verifier than resampling one model or having a model score its own candidates. The discovery is a parameter-free law: from measured mean member accuracy a, mean pairwise error correlation rho, and shared-misconception rate s, closed forms predict consensus accuracy and the shared-error floor (Eq. 1), the rate at which the whole panel is unanimous and wrong. Empirically the law matches consensus accuracy to mean absolute error 0.03 across seven benchmarks, the jury closes the f

Load-bearing premise

The law assumes that a single latent difficulty factor explains all error correlation among panel members, so that once difficulty is fixed models' mistakes and wrong-answer choices are independent; if independently trained models share a blind spot that survives after controlling for difficulty, the predicted agreement distribution and shared-error floor are miscalibrated.

Editorial extensions

If this is right

On competition math, a four-model jury captured 100% of the achievable selection gain over self-consistency, while a single model scoring its own candidates captured essentially none.
The parameter-free law predicts consensus accuracy and shared-error floor from three measured statistics, to mean absolute error 0.03 and 0.009, and transfers leave-one-benchmark-out.
On an identical candidate pool, the jury matches the strongest trained verifier inside its math domain and is the top selector outside it, at zero training cost.
Consensus doubles as an abstention dial: a unanimous panel is 99.5% correct on MATH-500 while answering 85% of problems, but on science benchmarks the shared-error floor caps unanimous accuracy below one.
Matched-budget experiments show four decorrelated models beat 32 samples of one model; pairwise error correlation falls from 0.68 within-model to 0.47 cross-family, the ordering that predicts selection accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the law turns panel design into a measurable trade-off: on a small labeled calibration set one could estimate (a, rho, s) for available models and choose the panel that minimizes rho and s rather than maximizing individual accuracy.
If the shared-error floor is a stable property of a domain, it can serve as an audit for model families: a high floor reveals a shared convention or blind spot that no agreement signal can catch, pointing directly at data or training that would diversify errors.
The paper's rejection-sampling result suggests consensus rewards could be used while training, which would change the floor the law treats as fixed; tracking floor over generations would test whether training on peer agreement actually decorrelates errors or just aligns them.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a solid empirical paper with a slightly oversold theoretical wrapper. If you work on test-time scaling and want a label-free selector, cross-model consensus is real: on the fixed candidate pools used here it beats self-consistency and matches or beats four trained verifiers, and the controls are genuinely careful — same pools, same grader, paired bootstraps, generator-robustness, a fully open-weight ablation, and a fairness gate for the PRMs. Credit where due: the closed-form law is neat, the simulation match is clean, and the paper explicitly separates the known claim that voting helps from the stronger claim that the signal is predictable and has a quantified ceiling.

Soft spots are real but don't sink the main result. First, the law is parameter-free only in a narrow sense: no parameter is fit to the consensus accuracy it predicts, but its inputs (a, rho, s) are measured on ground-truth-labeled data from the same benchmark. Appendix B concedes the law is calibrated from a labeled sample, not label-free to calibrate. The leave-one-benchmark-out test helps, but it is still in-family calibration. Second, the shared-error floor — the headline ceiling — is estimated from very few events. On GPQA, the unanimous-wrong count is six problems, two of which the paper itself regrades as grading artifacts. The stress-test concern is fair: the one-factor difficulty model assumes that conditional on difficulty, wrong-answer choices are independent except for a shared attractor. Shared blind spots like the molar-enthalpy convention could couple errors more strongly than a scalar s captures, which would make the predicted floor too optimistic exactly where the method is promoted as superior to trained verifiers. Third, no code or data artifacts are released.

These are bounded concerns. The verifier comparison does not depend on the law, and the paper honestly discusses the weak single-model self-scoring baseline in Appendix K. The reader's conditional verdict is about right.

I would send this to serious peer review. The empirical core deserves testing against other pools and generators, and the law's assumptions deserve scrutiny. A good revision would add an out-of-domain calibration test — new domain, labeled set, check the floor — and state plainly that the law is calibrated, not self-contained.

Referee Report

4 major / 4 minor

Summary. The paper proposes cross-model consensus as a training-free verifier for best-of-N selection in LLM reasoning. A panel of independently trained models each solves a problem once, and the modal answer (with the agreement fraction as a confidence signal) selects from a fixed candidate pool. Across seven benchmarks the method is reported to outperform self-consistency and a single-model LLM-verifier, to match or exceed trained process/outcome reward models in-domain and outperform them out-of-domain, and to be governed by a closed-form 'parameter-free law' that predicts consensus accuracy and a shared-error floor from three measured panel statistics (mean accuracy a, pairwise error correlation ρ, and shared-attractor mass s). The experimental design is strong: fixed candidate pools, identical grading, paired bootstraps, generator robustness, an open-weight ablation, and a fairness gate for trained verifiers. The law's closed forms match Monte-Carlo simulation to within 0.002, and the paper is unusually transparent about prompts, limitations, and per-cell intervals.

Significance. If the claims hold, this is a practically important result: a label-free selector that rivals trained verifiers with no training cost, plus an analytic characterization of when agreement can be trusted. The verifier comparison is well controlled (fixed pool, matched grading, generator-robustness, open-weight ablation), and the paper makes several falsifiable predictions (the consensus-accuracy and floor curves) that are tested against empirical data. The law's closed forms are mathematically coherent and validated against simulation. The main risk is that the law's one-factor latent-difficulty model cannot distinguish 'shared difficulty' from 'shared blind spots'; this matters for the headline 'quantified ceiling' claim, though it does not by itself undermine the core verifier comparison.

major comments (4)

[§3.2, Eq. (1)] The law presupposes that conditional on latent difficulty p, model errors — and choices among wrong answers — are independent given the measured attractor mass profile. Appendix D's GPQA case study documents shared content-level conventions (molar-enthalpy unit, absorption-line tracer, stereochemistry descriptors) that induce dependence among wrong choices beyond a scalar p or marginal s. The same observed agreement distribution is compatible with a 'shared difficulty' model (which the law fits) and a 'shared blind spot' model in which the joint probability of all models landing on the same wrong answer is much higher than the product of marginals. The floor is therefore not identified from the measured statistics, and the claim that the law 'exposes the method's ceiling' out of domain is not supported. A concrete test: hold out a content area, measure the conditional co-occurrence of wr
[Appendix B / §5.2] The 'parameter-free law' measures (a, ρ, s) on the same ground-truth-labeled data whose consensus behavior is the prediction target. The statement in Appendix B that 'every benchmark is effectively held out' conflates 'no fitted parameters' with 'out-of-sample.' The leave-one-benchmark-out test in Appendix C transfers only the wrong-answer profile, not the conditional-independence assumption, so it does not address the identifiability problem. Please provide a clean calibration/prediction split: estimate the inputs on a random subset of problems and predict consensus behavior on the held-out remaining problems, and also report the prediction for a benchmark whose inputs are not re-measured. This would strengthen the 'predictable in advance' claim considerably.
[§5.2 / Table 2] The shared-error floor on GPQA (0.030) rests on 6 unanimous-wrong problems out of 198. With n=198, a binomial 95% CI for a 0.030 rate is roughly [0.011, 0.064], so the floor is known only to a factor of about 3. The reported floor MAE of 0.009 does not convey this uncertainty, and the paper's abstract and introduction lean on the floor as a 'quantified ceiling.' Please report a confidence interval on the floor estimate (e.g., from the existing bootstrap machinery used for Table 6), and state explicitly that the GPQA floor measurement is a small-sample estimate. The AIME-2024 residual (predicted 0.011 vs empirical 0.000) also deserves a sentence, since it is outside the headline MAE.
[Abstract / §5.1] The abstract's statement that 'on competition math it closes the entire gap to an oracle selector' is true only for AIME-2024 in Table 1. On the other competition-math benchmarks the jury captures a fraction of the oracle gap: AIME-2025 40.0 vs 43.3 (33%), MATH-500 96.6 vs 98.6 (41%), OlympiadBench 72.6 vs 79.1 (53%). Please qualify the claim to 'on AIME-2024 it closes the entire gap' or report the average fraction of the oracle gap across the competition-math benchmarks. The current wording is a material overstatement of the results.

minor comments (4)

[§3.2, Eq. (1)] Eq. (1) is written for a single attractor s, but the reported predictions use the measured multi-attractor profile (s_1,...,s_K). The main text should define both notations in one place and state that the scalar form is the interpretable single-misconception special case; the current transition between 'the predictions we report use this measured profile' and Eq. (1) is confusing.
[Reproducibility / Appendix A] The text says grading uses 'math verify' and a 'boxed-answer parser'; for reproducibility, please name the exact package version or provide the parser source in the supplementary material. Also, 'Mathematical Association of America' is listed as the author of the AIME reference; if this is a dataset, the citation should follow the benchmark's preferred format.
[Figure 1] In the 'Why It Works' panel, the self-consistency example shows four Qwen samples with three '23' and one '17' and the cross-model panel with three '17' and one '23'. The figure is clear, but the self-consistency answer '23' is shown as wrong in a green/red color scheme; consider making the color convention explicit in the caption.
[Terminology] The term 'parameter-free law' is used for the closed form, but the inputs are benchmark-specific measurements. Consider renaming to 'closed-form law with measured inputs' throughout, to avoid implying the law is universal or requires no calibration. The abstract already says 'derived in closed form'; the phrase 'parameter-free' is misleading.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the jury's empirical advantage is independent of the predictive law, and the law's inputs are not defined in terms of its targets.

full rationale

The paper's load-bearing claim—that cross-model consensus selects better than self-consistency and trained verifiers on fixed candidate pools—is an empirical, independent comparison; it does not depend on the closed-form law. The law in Eq. 1 maps measured panel statistics (a, rho, s) to the consensus accuracy and shared-error floor through a Beta-Binomial latent-difficulty model. These inputs are not defined in terms of the prediction targets: s is the marginal fraction of wrong answers in the modal wrong cluster, not the empirical unanimous-wrong rate, and rho is a pairwise error correlation, not consensus accuracy. The floor formula s^M E[(1-p)^M] is therefore not an identity with the measured floor; it is a nontrivial model prediction that can fail (and does for the single-attractor version on MMLU-Pro, forcing the multi-attractor refinement). The main caveat, acknowledged by the paper in Appendix B, is that the law is 'calibrated from a labeled sample, not label-free to calibrate: characterizing a panel on a new domain requires a labeled calibration set from that domain.' Thus the MAE 0.028 on the seven benchmarks is an in-sample goodness-of-fit rather than a fully out-of-sample forecast, and the 'leave-one-benchmark-out' test still uses the held-out benchmark's measured (a, rho). This weakens the 'parameter-free' wording but is not circularity: no equation reduces the predicted quantity to an input by construction, and the verifier advantage itself is a direct measurement on fixed pools. No load-bearing self-citation, imported uniqueness, or ansatz-via-citation appears in the derivation chain.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim stands primarily on the controlled empirical comparison (Tables 1 and 3). The law's predictive power, a secondary claim, rests on measured panel statistics (a, ρ, s) taken from the same ground-truth-labeled benchmarks it predicts; the generative model is calibrated to reproduce ρ exactly (k=1/ρ−1), and the multi-attractor form was adopted after the scalar form failed on MMLU-Pro. No new physical entities are introduced; the 'shared-error floor' is a derived quantity, not an entity.

free parameters (3)

mean member accuracy a = measured per benchmark (e.g., GSM8K ≈ 0.97, AIME-2024 ≈ 0.70)
Law input; estimated from correct/incorrect labels on the same benchmark whose consensus accuracy it predicts.
pairwise error correlation ρ = measured per benchmark; cross-family mean ≈ 0.47, within-model ≈ 0.68
Sets Beta concentration k = 1/ρ − 1, so the generative model is explicitly calibrated to reproduce this measured statistic before 'predicting' consensus accuracy.
shared-misconception rate s (or top-K mass profile) = per-benchmark fraction of wrong answers in modal wrong class (e.g., MATH-500 leading masses (0.78, 0.18, 0.04))
Single scalar s is the modal wrong-class fraction; multi-attractor predictions use a measured profile with K=6 slots. The refinement was adopted because the scalar form predicted floor 0.41 on MMLU-Pro vs empirical 0.14 (Appendix C).

assumptions (3)

domain assumption Per-problem difficulty p ~ Beta(ak, (1−a)k) with k = 1/ρ − 1 is the only coupling of model errors; conditional on p, models err independently.
§3.2: 'This is the minimal model that couples the two forces governing agreement.' If errors are coupled beyond shared difficulty (e.g., shared training data), the closed forms overstate independence and the floor/accuracy predictions shift.
domain assumption Wrong answers land on a shared attractor with probability s (or on measured clusters) and otherwise as unique singletons; the clustering is the same for every model.
§3.2 and Appendix C multi-attractor generalization. Real wrong-answer distributions are measured, but the model assumes a fixed per-member collision probability independent of the other members' choices.
domain assumption A few hundred labeled problems per domain suffice to estimate (a, ρ, s), and the wrong-answer profile transfers across domains (leave-one-benchmark-out).
Appendix B and C: the law is not label-free to calibrate; the stated bounds use n≥198 and transfer of the profile from other benchmarks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning." pith.science (2026). https://pith.science/paper/NHPKXX66

@misc{pith2026260710139,
  author       = {Pith},
  title        = {Pith review of: LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NHPKXX66}},
  note         = {Machine review of arXiv:2607.10139}
}

abstract

Selecting the correct answer from a pool of candidate reasoning chains is the engine of test-time scaling, yet the standard selectors each carry a cost: self-consistency inherits the errors of the single model it resamples, and trained reward models need labeled data and transfer poorly off-distribution. We study a third signal, free at inference time: cross-model consensus, the degree to which independently trained models, each solving the problem once, agree on a final answer. We treat the panel as an LLM-jury, in which the verification signal is the structure of agreement itself, with no model scoring another's work. Across seven benchmarks it selects correct answers better than self-consistency and far better than a model scoring its own candidates: on competition math it closes the entire gap to an oracle selector, while self-scoring closes almost none. The mechanism is error decorrelation: independently trained models err differently, so their wrong answers scatter while the correct one accumulates agreement. We make this precise with a parameter-free law, derived in closed form, that predicts consensus accuracy from three measured panel statistics to a mean absolute error of $0.03$ and exposes the method's ceiling: a shared-error floor where models share a misconception, near zero on math but non-trivial on science. Against four trained verifiers spanning discriminative, outcome, and generative reward models, the free LLM-jury matches the strongest inside their math training domain and is the top selector outside it. Cross-model consensus is thus a verifier we can characterize in advance: a law that says when to trust it, and a floor that marks where it cannot.