Pith. sign in

REVIEW 3 major objections 5 minor 60 references

Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A moderated jury of cheap open-weight language models, deliberating under a trace-blind moderator, catches step-level reasoning defects more accurately than single frontier judges and at a fraction of the cost.

desk verdict A practical open-weight jury for step-level reasoning defect detection with honest baselines and clear limitations; the headline margin may be optimistic because key choices were made on the same 200-record benchmark, but the direction is supported by a smaller second benchmark. read the letter →

arxiv 2608.12585 v1 pith:XBPOS36Y submitted 2026-08-12 cs.AI

Congchao Wang , Diwakar Singh , Qiaozi Gao , Spyros Matsoukas , Yang Liu , Mahdi Namazifar This is my paper · ORCID
classification cs.AI
keywords LLM-as-a-judgereasoningtraceevaluationmulti-modelconsensusdeliberationstep-leveldefectdetectionopen-weightmodelsBalancedF1defect-guidedretry
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reasoning Jury claims that step-level defect detection in long reasoning traces—the signal needed to filter training data, shape reinforcement learning rewards, and evaluate reasoning models—is better done by a panel of independent language-model judges whose verdicts are combined through moderated consensus than by any single model, even a frontier one. On Hard2Verify, a homogeneous jury of three gpt-oss-120b samples reaches 82.3 Balanced F1 in deliberation, against 73.7 for opus-4.6 solo and 69.5–71.2 for the other frontier judges, while costing between 8% and 16% of opus-4.6's dollar price. The same pattern holds on DeltaBench, where the jury at 61.5 Balanced F1 matches gpt-5.4 and beats opus-4.6. A further experiment shows the structured verdicts are usable: when a generating model retries flagged traces, adding the jury's diagnoses and severity raises accuracy from 71.2% with step locations alone to 76.2%.

What carries the argument

The load-bearing object is the [STEP-x] segmented trace: every reasoning trace is split at sentence-boundary double-newlines into numbered steps, and every juror verdict cites specific steps. Phase 1 gives each juror the problem, the segmented trace, and the final answer and asks for JSON defects with step references, a self-contained what_went_wrong, a severity (neutral/minor/major/fatal), and quoted evidence, enforced by a genericness test and specificity self-check. Phase 2 has two modes: Consolidation, where a judge sees the trace plus all Phase 1 verdicts and verifies, merges, and fills gaps; and Deliberation, where a moderator—blind to the problem, trace, and solution—selects speakers, issues process instructions, tracks a claim-level consensus state, detects convergence, and extracts final consensus defects with confidence and dissents. This two-phase structure is what turns a bag of independent opinions into a single grounded, severity-tagged verdict.

What would settle it

Take a fresh, never-seen set of long reasoning traces with human step-level defect labels (or a held-out split of Hard2Verify), freeze the Phase 1 prompt, severity threshold, and propagation rule before reading any labels, then run the 3×gpt-oss-120b jury and an opus-4.6 solo judge on the same records. If the jury's Balanced F1 does not exceed opus-4.6 by at least the margin reported here (roughly 8.6 points), the central claim that an open-weight jury outperforms frontier single judges at reasoning-defect detection fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a panel of open-weight LLM judges with a moderated consensus phase can identify defects in long step-segmented reasoning traces more reliably than a single frontier model judge, and at a small fraction of the cost. In Phase 1 each juror independently emits structured defect findings (step references, explanation, severity, evidence); in Phase 2 either a consolidator merges them or a moderator runs a deliberation in which jurors critique each other and may revise, with the moderator blind to the problem and trace. On Hard2Verify the deliberated 3×gpt-oss-120b jury scores 82.3 Balanced F1 versus 73.7 for opus-4.6 solo, 71.2 for sonnet-4.6, 69.5 for gemini-3.1-pro, and 83.9 for gpt-5.4; the jury costs $12.35 in deliberation mode and $6.50 in consolidation mode for 200 records, against $79.13 for the opus pass. The authors attribute the gain primarily to pooling independent judgments (recall recovery), with deliberation also adjudicating and deduplicating, and they show that homogeneous juries (three samples of one model) retain most of the lift, so model diversity is not the driver.

Load-bearing premise

The headline results assume that the Phase 1 prompt, the 'minor and above' severity threshold, and the scoring rule that counts a defect as propagating to later steps were not tuned against Hard2Verify's gold labels; if those choices were selected on the test set, the Balanced F1 gains would be optimistic and might not transfer to new traces or benchmarks.

Editorial extensions

If this is right

  • On Hard2Verify, an open-weight jury with deliberation reaches 82.3 Balanced F1, surpassing opus-4.6 (73.7), sonnet-4.6 (71.2), and gemini-3.1-pro (69.5) and landing within two points of gpt-5.4 (83.9), at roughly 8–16% of the frontier judge's dollar cost.
  • A homogeneous jury—three independent samples of gpt-oss-120b with the same model as moderator—captures most of the gain (+13.0 Balanced F1 over its best solo), so model diversity is not required; the ceiling is set by the underlying model's capability.
  • Deliberation mostly improves recall, consolidation trades recall for precision, and a simple union of Phase 1 findings captures much of the improvement; majority voting collapses because jurors rarely flag the exact same steps.
  • The pattern generalizes to DeltaBench: the 3×gpt-oss-120b jury's deliberated consensus (61.5 Balanced F1) beats opus-4.6 solo (58.4) and matches gpt-5.4 (61.8).
  • Feeding jury diagnoses and severity back to a model on retry lifts accuracy over step locations alone (76.2% vs 71.2%), and a selective-retry policy reaches 88.5% corpus accuracy versus 80.0% before retry.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the jury's advantage is driven by pooling and debate rather than by the specific models, the same two-phase structure could be applied to other fine-grained evaluation tasks—such as grading code patches, legal or medical reasoning, or detecting hallucinated citations—where single-model judges are known to miss errors.
  • The paper deliberately does not isolate the moderator's trace-blindness; an extension would vary moderator access (full trace vs transcript only) to measure how much of the deliberation gain comes from preventing content bias versus from simple speaker scheduling.
  • The retry experiment suggests a scalable offline loop: use the jury to flag defective traces, have the generator retry only those, and keep original answers otherwise; the reported 8.5-point corpus gain is a lower bound because there is no blind-retry control, so a careful head-to-head against 'retry everything' would sharpen the value of jury-gated retries.
  • Because the jury is cheap and open-weight, the same pipeline could serve as an RL reward and defect-feedback generator for training reasoning models, provided the latency of deliberation is acceptable for off-policy training; the paper notes this is not yet usable for on-policy RL.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces Reasoning Jury, a two-phase pipeline in which several LLM jurors independently audit a step-segmented reasoning trace and a Phase 2 mechanism (consolidation or moderated deliberation) produces a consensus verdict with per-step defect localization, severity, and evidence. The evaluation uses Hard2Verify (200 records) and DeltaBench (1,226 paired records), comparing solo frontier judges with juries of open-weight models. The headline results are that a homogeneous 3× gpt-oss-120b jury reaches 82.3 Balanced F1 on Hard2Verify versus 73.7 for opus-4.6 solo, that the same jury reaches 61.5 on DeltaBench versus 58.4 for opus, and that the jury costs 8–16% of the frontier judge at list prices. The paper also reports exploratory defect profiling on AIME2026 and a defect-guided retry experiment in which rich feedback improves retry accuracy from 71.2% to 76.2%.

Significance. If the reported results hold, the paper makes a practically important contribution: it identifies a configuration in which three samples of a modest open-weight model, moderated by the same model, outperform much more expensive frontier judges on step-level reasoning-defect detection while producing structured, auditable verdicts. The strengths are the explicit comparison against union and majority-vote baselines, record-level bootstrap intervals on the main benchmark, a second benchmark replication, a paired McNemar-based retry analysis, full prompts in the appendices, and unusually candid limitations (the explanation fields and severity labels are not validated against ground truth). The main benchmark result is not yet established as a pre-specified comparison, because key configuration choices are made on the same 200 records and the headline jury score lacks a confidence interval; the DeltaBench replication also lacks uncertainty quantification. With those additions the claim would be convincing.

major comments (3)
  1. [§3.1, §4.5] The main Hard2Verify result is reported on the same 200 records that were used to choose the Phase 1 prompt: §3.1 compares the Hard2Verify prompt (85.3 Balanced F1 with gpt-5.4) against the paper's prompt (83.9) and then uses the latter for all subsequent experiments, with no held-out development split described. The minor+ severity threshold, the specificity self-check cutoff in §B, and the propagation-aware scoring convention in §G are additional configuration choices that affect every score and are not documented as fixed before the test labels were examined. Because the central claim is the 82.3 versus 73.7 gap, the authors should either show that these choices were pre-specified and not selected on Hard2Verify labels, or provide a confirmatory run on a held-out split or an independent benchmark. The direction of any bias is not obvious, since the prompt selection actually chose a lower-scoring prompt for gpt-5.4, but the possibility of label-based selection remains.
  2. [§4.5, Table 7] The headline homogeneous 3× gpt-oss-120b deliberation score of 82.3 is reported in Table 7 without a bootstrap confidence interval, whereas Table 4 reports ± intervals for the other juries. The opus-4.6 solo score is 73.7±3.7, so the 8.6-point headline gap needs an interval around 82.3 and ideally a paired comparison over records to support the word 'significantly' in the abstract. Please add record-level bootstrap intervals for the homogeneous panels and a paired significance test for jury-versus-opus on Hard2Verify.
  3. [§4.6, Table 12] The DeltaBench generalization result is presented as a replication but has no uncertainty quantification: the 61.5 versus 58.4 Balanced F1 gap has no confidence interval or paired significance test, and §M reports that the paired sample drops 10 records where opus could not complete, with the excluded records concentrated in the longest math traces. Given the smaller gap relative to Hard2Verify and the non-random exclusion, the authors should report bootstrap or paired intervals and analyze the sensitivity of the conclusion to inclusion or exclusion of those 10 records.
minor comments (5)
  1. [Abstract and §4.8] The abstract states the cost is '8 to 15%' of the frontier judge, while Table 2 reports 16% for deliberation and the introduction says '8 to 16%'; please make the cost range consistent across the abstract, introduction, and results.
  2. [§4.6 and §M] The main text says the DeltaBench evaluation is on '1,226 records' and §M clarifies that this is the paired subset after removing 10 records where opus failed; please state this exclusion and its rationale directly in Section 4.6.
  3. [§4.8, Table 2] Section 3.2 says open-weight models are served on a local vLLM cluster, but Table 2 prices gpt-oss-120b using AWS Bedrock list rates; please state explicitly whether the cost analysis reflects list prices or actual serving costs, and whether local serving would change the reported ratios.
  4. [§N.1] The sentence '49steps outputs and27 full outputs contain no parseable integer' appears to mean 49 step-anchor outputs and 27 rich-feedback outputs; please make the condition labels unambiguous.
  5. [Figure 1 caption] The caption says the open-weight jury outperforms frontier models 'by up to 12 points', but Table 7 shows a 13.0-point lift for the homogeneous gpt-oss jury over its best solo sample and Table 1 shows lifts up to 15.4 points; please reconcile the stated maximum.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central jury-vs-frontier comparison is measured against external human-labeled benchmarks (Hard2Verify, DeltaBench) with no fitted parameters, no self-derived targets, and no load-bearing self-citations.

full rationale

Reasoning Jury's central claim is an empirical comparison, not a derivation from its own inputs. The headline numbers (82.3 vs 73.7 Balanced F1 on Hard2Verify, and 61.5 vs 58.4 on DeltaBench) are scored against external human gold labels from Hard2Verify [25] and DeltaBench [15], and the same scoring convention (minor+ severity threshold, propagation-aware step-level Balanced F1) is applied uniformly to the jury and every single-model baseline. No parameter is fitted and then renamed as a prediction; no jury output defines the metric that validates it. The Phase 1 prompt variant was indeed selected after comparing variants on Hard2Verify (Section 3.1: 'Using our prompt (Section B) with gpt-5.4 we get the Balanced F1 score of 83.9. Based on these numbers and the additional utilities that our prompt provides, for the rest of the experiments we leverage our prompt'), which is a legitimate benchmark-selection/overfitting risk rather than circularity: the choice did not alter the gold labels or the metric, and it was applied to baselines and jury alike. The paper's own limitation statements further reduce any appearance of self-support: 'These experiments do not evaluate explanation factuality, evidential support, severity calibration, semantic deduplication, or downstream usefulness' (Section 3.1), and 'benchmark scoring ignores explanation text, severity distinctions, evidence, and dissent, so these results do not establish that consolidation or deliberation improves those properties' (Section 4.7). The moderator-blindness passage is also an explicit caveat, not an imported theorem: 'We treat trace withholding as a design heuristic intended to limit direct re-adjudication, not as a structural guarantee of unbiased moderation. Its causal effect is not isolated in our experiments.' No uniqueness theorem is invoked, no ansatz is smuggled in via citation, and the cited prior work on panels ([4, 10, 28]) is contextual background rather than load-bearing evidence for the specific Hard2Verify result. The DeltaBench replication is an independent external check, and the paper transparently attributes part of the advantage to its prompt and propagation convention rather than to deliberation alone. The exploratory defect taxonomy (Section 5) is explicitly non-canonical ('created simply for demonstration purposes, and it does not establish a standard'), so it cannot support a circularity charge.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The system is an empirical pipeline with no fitted mathematical constants. The main unstated premises are that gold labels measure real defects, that the minor+ step-level metric is the right operationalization, that temperature sampling gives decorrelated judgments, and that the regex step segmenter is content-neutral. None of these is fitted to data, but each is a domain assumption that the reader must accept for the headline comparisons to be meaningful.

free parameters (2)
  • Minor+ severity threshold = minor, major, or fatal count as defective; neutral excluded
    Hand-chosen binarization used for all headline metrics. It defines what counts as a detected defect and could alter comparisons if severity calibration differs across models.
  • Specificity self-check inclusion threshold = >= 4 on a 1 to 5 scale
    Prompt rule set by the authors to suppress vague criticisms. Applied uniformly to all jurors and both Phase 2 modes, so it affects the defect lists but not the relative comparison.
assumptions (5)
  • domain assumption Human gold step labels in Hard2Verify and DeltaBench are accurate ground truth for reasoning defects.
    All headline Balanced F1 numbers are scored against these labels; if the labels are noisy or biased, the measured gains are noisy or biased too.
  • domain assumption Step-level Balanced F1 with the minor+ threshold captures the operational value of defect detection.
    The paper explicitly scores only localization, not explanation factuality, severity calibration, or downstream usefulness, so the main metric is a partial proxy for the full intended capability.
  • domain assumption Independent temperature-1.0 samples of the same model provide sufficiently decorrelated judgments for a jury.
    Section 4.5 shows large gains from homogeneous juries; this premise is required for the claim that sampling diversity, not architecture diversity, drives much of the benefit.
  • domain assumption The regex segmentation pattern (?<=[.!?])\s*\n\s*\n+ produces steps that preserve reasoning content fairly across all compared systems.
    Section 2.1 adopts this segmenter because dedicated segmentation models are unavailable; the paper notes that per-step rates are confounded by segmentation fineness.
  • domain assumption List-price API rates are a fair basis for the cost comparison between frontier and open-weight models.
    The cost ratios in Section 4.8 depend on these list prices; enterprise deals, local hosting, or different rate limits could change the monetary comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces." pith.science (2026). https://pith.science/paper/XBPOS36Y

@misc{pith2026260812585,
  author       = {Pith},
  title        = {Pith review of: Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBPOS36Y}},
  note         = {Machine review of arXiv:2608.12585}
}
read the original abstract

Improving reasoning LLMs requires the ability to judge the quality of long reasoning traces for effective reasoning data curation, strong training signals during reinforcement learning, and an in-depth understanding of reasoning behaviors during model performance evaluation. Additionally, surfacing reasoning mistakes that the model makes would enable improving the model's performance at runtime through providing feedback. Due to the difficulty of this complex task on long reasoning traces, single-model judges (even frontier models) do not do well at identifying reasoning defects. Additionally, leveraging frontier models during online training of reasoning LLMs is generally prohibited due to guardrails in terms of use. In this work, we introduce Reasoning Jury, a system that replaces the single judge with a jury of LLMs and a moderated consensus mechanism, to improve the fidelity of judgments for identifying reasoning defects. In reasoning jury, defects of a reasoning trace and their severity are surfaced through a deliberation where a moderator conducts a discussion amongst the jury where the jurors critique each other's judgments and get to modify their initial votes. The moderator derives a consensus through deliberation amongst jurors or consolidation of judgements. We show that Reasoning Jury with a jury of open-weight models (e.g., gpt-oss-120b) is able to significantly outperform frontier models (opus-4.6, sonnet-4.6, and gemini-3.1-pro) at correctly identifying reasoning defects. Besides accuracy performance improvements, the aggregated cost of the jury (initial verdicts, deliberations, consolidation, etc.) is a fraction (8 to 15%) of the cost of running frontier models in LLM-as-a-judge setup. We also show how these judgements can be leveraged to understand failure modes of reasoning LLMs on benchmarks, which allows much deeper understanding of a model's performance.

Figures

Figures reproduced from arXiv: 2608.12585 by the authors.

Figure 1
Figure 1. Comparing Balanced F1 scores of reasoning defect detection on Hard2Verify benchmark with frontier models [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A representative defect emitted by a juror in Phase 1, on an olympiad problem from Hard2Verify. The juror [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Solo jurors (dots) versus deliberated consensus (star) for each jury, with the lift over best juror on the right. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Consensus versus solo jurors as each jury is reduced from five jurors down to two. Exact numbers in Table 5. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Moderator choice changes deliberation by [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Diversity control: homogeneous juries of three independent temperature- [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: DeltaBench generalization: Balanced F1 for a single opus-4.6 Phase 1 pass versus a homogeneous [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: nemotron-3-super: flagged steps for the top three taxonomy leaves within each category, stacked by severity [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Accuracy vs defect severity for nemotron-3-super on [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: nemotron-3-super: flagged reasoning steps by taxonomy category, stacked by severity (regex segmentation, [PITH_FULL_IMAGE:figures/full_fig_p035_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 44 canonical work pages

  1. [1]

    optimize_anything: A Universal API for Optimizing any Text Parameter

    Lakshya Agrawal, Jae Lee, Shawn Tan, Sanjit A. Seshia, Koushik Sen, Dan Klein, Ion Stoica, and Joseph E. Gonzalez. optimize_anything: A universal api for optimizing any text parameter. InProceedings of the ACM Conference on AI Engineering: Software Engineering for AI (CAIS ’26). ACM, May 2026. doi: 10.48550/arXiv.2605.19633

    work page Pith review arXiv doi:10.48550/arxiv.2605.19633 2026

  2. [2]

    Claude opus 4.6 system card

    Anthropic. Claude opus 4.6 system card. Anthropic, February 2026. URL https://www. anthropic.com/claude-opus-4-6-system-card. Published February 5, 2026

    work page 2026

  3. [3]

    Claude sonnet 4.6 system card

    Anthropic. Claude sonnet 4.6 system card. Anthropic, February 2026. URLhttps://www. anthropic.com/claude-sonnet-4-6-system-card. Published February 17, 2026

    work page 2026

  4. [4]

    Multi-agent-as-judge: Aligning LLM-agent-based automated evaluation with multi-dimensional human evaluation.arXiv preprint arXiv:2507.21028, 2025

    Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, and Dakuo Wang. Multi-agent-as-judge: Aligning LLM-agent-based automated evaluation with multi-dimensional human evaluation.arXiv preprint arXiv:2507.21028, 2025

    arXiv 2025

  5. [5]

    Nudging the boundaries of llm reasoning

    Justin Chih-Yao Chen, Xiangyu Peng, Prafulla Kumar Choubey, Kung-Hsiang Huang, Jiaxin Zhang, Mohit Bansal, and Chien-Sheng Wu. Nudging the boundaries of llm reasoning. InThe Fourteenth International Conference on Learning Representations, 2026

    work page 2026

  6. [6]

    Stop summation: Min-form credit assignment is all process reward model 17 needs for reasoning

    Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, and Fei-Yue Wang. Stop summation: Min-form credit assignment is all process reward model 17 needs for reasoning. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2504.15275

    arXiv 2025

  7. [7]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  8. [8]

    DeepSeek-V4-Pro model card

    DeepSeek-AI. DeepSeek-V4-Pro model card. Hugging Face, 2026. URLhttps://huggingface. co/deepseek-ai/DeepSeek-V4-Pro. Accessed July 28, 2026

    work page 2026

Show all 60 references
  1. [9]

    Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms, 2026

    Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms, 2026. URLhttps://arxiv.org/abs/2605.00674

    2026 arXiv

  2. [10]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InInternational Conference on Machine Learning (ICML), 2024

    2024

  3. [11]

    Gemma 4 technical report.arXiv preprint arXiv:2607.02770, 2026

    Gemma Team. Gemma 4 technical report.arXiv preprint arXiv:2607.02770, 2026. URL https://arxiv.org/abs/2607.02770

    2026 arXiv

  4. [12]

    GLM-5: From vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    GLM-5 Team. GLM-5: From vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026. URLhttps://arxiv.org/abs/2602.15763

    2026 arXiv

  5. [13]

    Gemini 3.1 pro model card

    Google DeepMind. Gemini 3.1 pro model card. Google DeepMind, February 2026. URL https://deepmind.google/models/model-cards/gemini-3-1-pro/. Published February 19, 2026

    2026

  6. [14]

    A survey on LLM-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on LLM-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  7. [15]

    Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, and Wenbo Su. Can large language models detect errors in long chain-of-thought reasoning? InProceedings of the 63rd Annual Meeting of the Association for ...

    2025

  8. [16]

    Reinforcement learning via self-distillation

    Jonas Hübotter, Frederike Lübeck, Lejs Deen Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation. InInternational Conference on Learning Represe...

    2026

  9. [17]

    Let’s verify step by step.International Conference on Learning Representations (ICLR), 2024

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.International Conference on Learning Representations (ICLR), 2024

    2024

  10. [18]

    MiniMax-M2.7 model card

    MiniMax AI. MiniMax-M2.7 model card. Hugging Face, 2026. URLhttps://huggingface. co/MiniMaxAI/MiniMax-M2.7. Accessed July 28, 2026. 18

    2026

  11. [19]

    MiniMax-M3 model card

    MiniMax AI. MiniMax-M3 model card. Hugging Face, 2026. URLhttps://huggingface.co/ MiniMaxAI/MiniMax-M3. Accessed July 28, 2026

    2026

  12. [20]

    Kimi K2.6 model card

    Moonshot AI. Kimi K2.6 model card. Hugging Face, 2026. URLhttps://huggingface.co/ moonshotai/Kimi-K2.6. Accessed July 28, 2026

    2026

  13. [21]

    NVIDIA Nemotron 3: Efficient and open intelligence, 2025

    NVIDIA. NVIDIA Nemotron 3: Efficient and open intelligence, 2025. URLhttps://arxiv. org/abs/2512.20856. White paper

    2025 arXiv

  14. [22]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    OpenAI. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025. URL https://arxiv.org/abs/2508.10925

    2025 arXiv

  15. [23]

    Introducing gpt-5.4

    OpenAI. Introducing gpt-5.4. OpenAI, March 2026. URL https://openai.com/index/ introducing-gpt-5-4/. Published March 5, 2026

    2026

  16. [24]

    gpt-5.4 model

    OpenAI. gpt-5.4 model. OpenAI API documentation, 2026. URL https://developers. openai.com/api/docs/models/gpt-5.4. Accessed July 28, 2026

    2026

  17. [25]

    Hard2Verify: A step-level verification benchmark for open-ended frontier math

    Shrey Pandit, Austin Xu, Xuan-Phi Nguyen, Yifei Ming, Caiming Xiong, and Shafiq Joty. Hard2Verify: A step-level verification benchmark for open-ended frontier math. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2026

  18. [26]

    Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026

    Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026. URL https://qwen.ai/blog?id=qwen3.6-27b

    2026

  19. [27]

    PRISM: Pushing the frontier of deep think via process reward model-guided inference.arXiv preprint arXiv:2603.02479, 2026

    Rituraj Sharma, Weiyuan Chen, Noah Provenzano, and Tu Vu. PRISM: Pushing the frontier of deep think via process reward model-guided inference.arXiv preprint arXiv:2603.02479, 2026

    2026

  20. [28]

    Replacing judges with juries: Evaluating LLM generations with a panel of diverse models.arXiv preprint arXiv:2404.18796, 2024

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Nathan White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models.arXiv preprint arXiv:2404.18796, 2024

    2024 arXiv

  21. [29]

    GLM-5.2-FP8modelcard

    Z.ai. GLM-5.2-FP8modelcard. HuggingFace, 2026. URLhttps://huggingface.co/zai-org/ GLM-5.2-FP8. Accessed July 28, 2026

    2026

  22. [30]

    Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning, 2026

    Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, and Jiaya Jia. Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning, 2026. URLhttps://arxiv.org/abs/2510.19807

    2026

  23. [31]

    ProcessBench: Identifying process errors in mathematical reasoning

    Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. ProcessBench: Identifying process errors in mathematical reasoning. InProceedings of the Association for Computational Linguistics (ACL), 2025. arXiv:2412.06559

    2025 arXiv

  24. [32]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena.Advances in Neural Information Processing ...

    2023

  25. [33]

    statement_refs

    Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. ReasonFlux-PRM: Trajectory-aware PRMs for long chain-of-thought reasoning in LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2506.18896. 19 A Defect schema and...

    2025

  26. [34]

    **PROBLEM**: the task statement

  27. [35]

    The trace is divided into segments, each prefixed with [STEP-x] where x is an integer indicating the ordinal position of each segment in the trace

    **REASONING TRACE**: the reasoning produced by a model while attempting the problem. The trace is divided into segments, each prefixed with [STEP-x] where x is an integer indicating the ordinal position of each segment in the trace

  28. [36]

    ## Primary objective 20 Judge the *weaknesses* of the provided reasoning trace by pointing to **specific bad reasoning moves**

    **FINAL Response**: the final response to the problem, based on the reasoning trace. ## Primary objective 20 Judge the *weaknesses* of the provided reasoning trace by pointing to **specific bad reasoning moves**. ## Critical constraints - **Do NOT write a fresh full response**...

  29. [37]

    In [STEP-14], the trace states

    Name the specific step(s): "In [STEP-14], the trace states..."

  30. [38]

    Quote the exact problematic claim from that step

  31. [39]

    Could this comment apply to a totally different problem with no edits?

    Explain WHY it is wrong, including what the correct value/reasoning should be ## Genericness test (must pass) For every point you make, ask: > "Could this comment apply to a totally different problem with no edits?" If yes, rewrite it so it mentions at least one **problem-spec...

  32. [40]

    Adopt it only if the trace itself supports the claim

    **Verify before adopting.** For each candidate, re-read the cited [STEP-x] in the reasoning trace. Adopt it only if the trace itself supports the claim. Reject candidates the trace does not support -- never include a finding merely because one or several auditors raised it. He...

  33. [41]

    Supplement, don’t average: fold in supporting evidence from the other auditors, but never replace a specific claim with a vaguer paraphrase

    **Merge duplicates at maximum specificity.** When several candidates describe the same underlying defect, output ONE finding, using the MOST SPECIFIC description available among them (exact quotes, numbers, formulas, [STEP-x] refs). Supplement, don’t average: fold in supportin...

  34. [42]

    Include genuine defects that NO auditor raised

    **Fill the gaps.** You are simultaneously auditing the trace yourself under the instructions above. Include genuine defects that NO auditor raised. Your own findings are held to the same grounding, genericness, and specificity requirements as everything else

  35. [43]

    **Recalibrate severity yourself.** A candidate’s severity rating is a suggestion, not a constraint: assign severity from your own reading of the severity guidelines above

  36. [44]

    Output format

    **Same bar for everything.** Every finding in your output -- adopted, merged, or newly added -- must pass the genericness test and score >= 4 on the specificity self-check. ## Output format reminder (JSON ONLY -- same schema as specified above) Your final output remains **only...

  37. [45]

    - **Do NOT abstract away specifics.** When multiple judges describe the same issue at different levels of detail, use the MOST SPECIFIC description as the base

    CONSENSUS TRACKING: One of your jobs is to produce **one high-fidelity consolidated judgement** by: - clustering overlapping findings into canonical, problem-specific items - **preserving maximum specificity** from the most detailed source judgement 24 ## Critical constraints ...

  38. [46]

    Use that as the canonical wording

    **Select the best source**: Identify which judge gave the most specific description (most step references, direct quotes, exact numbers/formulas). Use that as the canonical wording

  39. [47]

    **Preserve from the best source**: - Exact numbers, formulas, or calculations - Direct quotes from the trace (keep in quotation marks) - Specific [STEP-x] references - The ‘correct_value‘ if any judge provides it

  40. [48]

    **Supplement, don’t average**: Add supporting evidence from other judges, but never replace a specific claim with a vaguer paraphrase

  41. [49]

    Could this description apply to a totally different problem with no edits?

    **Genericness test**: Each consolidated issue must pass: "Could this description apply to a totally different problem with no edits?" If yes, rewrite using the most specific judge’s language. Example: - jury-2 says: "In [STEP-14], the trace states ’E proportional to 1/sqrt(D)’...

  42. [50]

    - Prioritise panelists involved in unresolved disagreements

    TURN SELECTION: Choose which panelist speaks next based on: - Ensure every panelist speaks at least once per logical round. - Prioritise panelists involved in unresolved disagreements. - Recall silent panelists who have not spoken in 2+ turns

  43. [51]

    - Ask them to clarify, defend, or concede specific points raised by others

    INSTRUCTION GENERATION: Give the selected panelist a specific, process-oriented instruction: - Point them to specific disagreements they should address. - Ask them to clarify, defend, or concede specific points raised by others. - NEVER suggest what the "right" answer is about...

  44. [52]

    should_terminate

    TERMINATION DETECTION: Signal that deliberation should end when: - A full logical round passes with no new substantive arguments. - All panelists have explicitly signalled agreement on all points. - Arguments are cycling without resolution. You MUST respond with ONLY valid JSO...

  45. [53]

    consensus_state

    Update "consensus_state": Pay special attention to the last juror’s response to your question and check whether the current speaker endorsed, contested, or was silent on each consensus item. Update ‘source_ref‘ on consensus_state accordingly

  46. [54]

    Should deliberation terminate? (Has a full round passed with no new substantive arguments? Are panelists repeating themselves?)

  47. [55]

    Additional rules

    If not, who should speak next and what should they address? Do NOT refer to anything from consensus state when you address the panelists, as they don’t have access to it. Listing 5: The moderator prompt (system prompt and per-turn template). C.3 Juror contribution prompt When ...

    2025

  48. [56]

    The per-problem analysis addresses dependence among traces from the same problem, but broader generalization remains untested

    The experiment covers one generator, one benchmark, and one retry sample per condition and trace. The per-problem analysis addresses dependence among traces from the same problem, but broader generalization remains untested

  49. [57]

    Some improvement over the original answers may therefore come from making a second attempt rather than from the feedback itself

    There is no blind-retry condition. Some improvement over the original answers may therefore come from making a second attempt rather than from the feedback itself. The pairedfull- versus-stepscomparison controls for this shared retry opportunity

  50. [58]

    The experiment consequently measures the complete rich-feedback package; it does not isolate the effect of either field or distinguish explanation from a supplied fix

    The fullcondition bundles diagnosis and severity, and some diagnoses contain explicit corrections. The experiment consequently measures the complete rich-feedback package; it does not isolate the effect of either field or distinguish explanation from a supplied fix

  51. [59]

    Failures caused by omissions may also escape detection

    The gate’s96 .9%wrong-answer recall is specific to this corpus and does not show that each flagged defect caused the corresponding wrong answer. Failures caused by omissions may also escape detection

  52. [60]

    The external AIME answer key validates the correctness of retry outcomes, not the factual accuracy or severity of individual defect descriptions

    The jury findings are not human-validated ground truth. The external AIME answer key validates the correctness of retry outcomes, not the factual accuracy or severity of individual defect descriptions. N.5 Artifacts The traces, per-trace deliberated verdicts, and extracted def...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.