Pith. sign in

REVIEW 3 major objections 5 minor 46 references

A reviewer that detects errors precisely can still fail to improve a math agent's answers when the protocol does not act on its own correct critiques.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:29 UTC pith:BDTSDLAP

load-bearing objection Solid, honestly-scoped paper showing reviewer precision and critique uptake can decouple; the causal story is weaker than the descriptive one, but the measurements are careful. the 3 major comments →

arxiv 2607.15388 v1 pith:BDTSDLAP submitted 2026-07-16 cs.AI

Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning

Chih-Hsuan Yang , Jingyan Jiang , Vikram Vasudevan , Cheng-Hau Yang , Huihuo Zheng , Le Chen , Eliu A. Huerta , Venkatram Vishwanath
show 2 more authors
Ian T. Foster Rajeev Thakur
This is my paper
classification cs.AI
keywords multi-agent reasoningreviewer precisioncritique uptakeself-correctionLLM evaluationOmni-MATHverifier-guided repairagent protocols
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Many recent math and science agent systems route critique through a dedicated reviewer, assuming that a specialized review stage converts wrong answers into right ones. This paper tests that assumption on 4,181 olympiad-level math problems and finds a specific failure: the hierarchical planner–executor–reviewer pipeline had the more precise reviewer (0.861 vs. 0.644) yet changed its next candidate after verified-useful critique only 33.6% of the time versus 93.5% for peer discussion, and repaired far fewer problems. The accuracy gap between the two protocols (85.2% vs. 89.2%) is therefore not explained by reviewer precision but is consistent with differences in critique uptake. The authors introduce a detection→uptake→repair measurement chain that makes each step separately visible, so evaluation no longer has to conflate noticing an error with fixing it. A sympathetic reader should care because reviewer-centric evaluation can make a system look strong at spotting errors while it fails to actually solve more problems.

Core claim

On the Omni-MATH 2 benchmark (4,181 verifier-grounded competition problems) with matched gpt-oss-120b actors at temperature 0, the paper shows that a Planner–Executor–Reviewer (PER) pipeline and a broadcast peer-deliberation protocol diverge on hard tiers: broadcast reaches 89.2% final accuracy, PER 85.2%. The surprising decomposition is process-level. PER's reviewer is more precise (0.861 vs. 0.644) and more resistant to misleading review, yet evaluator-verified useful critique changes the next carried-forward candidate only 33.6% of the time in PER versus 93.5% in broadcast, and reviewer-guided repair is 0.051 versus 0.286. In short, the paper's central claim is that reviewer detection qua

What carries the argument

The load-bearing measurement is a three-step decomposition of the inner review loop. NeglectRate measures how often a reviewer-flagged wrong answer is submitted unchanged; CouplingRate (the paper's uptake statistic) measures the fraction of evaluator-verified useful review episodes in which the immediately following candidate answer differs from the rejected one; ReviewerGuidedRepairRate measures the fraction of true-positive review episodes that the next candidate actually fixes. CouplingRate is the central object: it is an answer-transition statistic computed only on evaluator-verified useful critique, not a judgment of whether the solver 'understood' the feedback. Comparing PER and broadc

Load-bearing premise

The PER–broadcast comparison is read as evidence about critique uptake, but the two protocols differ in several architectural dimensions at once (inner-round budgets, approval rules, prompt scaffolds, and memory surfaces), and all 'useful critique' labels come from the same model family as the actors; if those confounds, rather than routing, drive the accuracy gap, the precision–uptake separation would not be a property of reviewer–solver coupling as such.

What would settle it

Re-run the PER protocol with broadcast's approval rule and shared candidate state while keeping a separated reviewer role; if CouplingRate stays near 0.34 rather than moving toward 0.93, the uptake gap is not caused by critique delivery or interface routing. Alternatively, re-audit the main comparisons with a human-grounded outcome label on the hard tiers 7–10; if the 9–10% evaluator disagreement reverses the PER–broadcast accuracy order, the central gap would not be robust.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct critique is not acted on, improving reviewer precision alone will not improve final accuracy; protocols must also make review feedback binding on the next candidate.
  • Evaluation of multi-agent systems should report uptake and repair alongside reviewer quality, because a precise reviewer can coexist with low realized repair.
  • Forced verbal acknowledgment of critique can be a superficial compliance target: in the tested config it raised neglect and lowered accuracy.
  • Presenting reviewer guidance directly in the solver's working context partially improved follow-through, pointing toward interface design as a lever for making critique take effect.
  • The detection–uptake–repair metrics are portable: any verifier-grounded reasoning protocol can be scored on whether its own useful critique changes the candidate it carries forward.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same detection/uptake split likely applies beyond math to any system where feedback must change a downstream artifact—code-review pipelines, document-drafting agents, or human-AI oversight—suggesting that 'critique is correct' and 'critique is acted on' should be measured as separate things.
  • Precision and uptake may trade off mechanistically: a reviewer policy optimized for high precision (few false alarms) may also emit fewer binding imperatives or be overridden more easily; a testable extension is to measure CouplingRate when reviewer verbosity or route permissions are varied.
  • In safety-critical settings, correct-but-ignored critique is functionally equivalent to no critique, so scalable-oversight proposals that only improve judge accuracy may underdeliver unless they also enforce that the model's next action is conditioned on accepted feedback.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies four verifier-grounded protocols on 4,181 Omni-MATH problems using matched gpt-oss-120b actors at temperature 0: one-shot baseline, single-agent iterative, a Planner–Executor–Reviewer pipeline (PER), and a broadcast-style deliberating group. It finds that collaboration gains open sharply from tier 4 onward, and that broadcast outperforms PER in final accuracy (89.2% vs. 85.2%) and Pass@1 (78.6% vs. 72.8%). The central mechanism claim is that this gap is not explained by reviewer detection quality: PER's reviewer has higher precision (0.861 vs. 0.644) but evaluator-verified useful critique changes the next carried-forward candidate only 33.6% of the time versus 93.5% for broadcast, with ReviewerGuidedRepairRate 0.051 vs. 0.286. The paper introduces NeglectRate, CouplingRate, and ReviewerGuidedRepairRate as separable measurement axes and reports within-PER interventions (ACK-required and EMB) as directional evidence that feedback presentation affects follow-through. It concludes that reviewer precision and critique uptake are empirically separable and that reviewer-centric evaluation can overstate system quality.

Significance. If the reported separation is robust, the paper makes a useful measurement contribution: it operationalizes the detection → uptake → repair chain in multi-agent reviewer–solver systems, provides direct tabulations with no fitted parameters, and includes a strict answer-transition audit that is numerically identical for the headline CouplingRate, a cross-evaluator replay on submitted answers, problem-clustered bootstrap intervals, and a reduced cross-family replication. These are real strengths. The practical implication—that a protocol can detect errors well yet fail to solve more problems because correct critique is not bound to the next candidate—is important for evaluation design. However, the interpretative leap from the PER–broadcast contrast to a generalizable reviewer–solver decoupling phenomenon is constrained by the fact that the two protocols differ on many architectural dimensions at once, and the within-PER probes do not cleanly isolate the proposed interface mechanism. The significance would be higher if the central claim were either supported by a matched routing ablation or consistently scoped to whole-protocol comparison.

major comments (3)
  1. [§4.2, Fig. 3, Table 12] The headline precision–uptake separation is demonstrated by comparing two complete protocols that differ simultaneously in inner-round budget (2 vs. 4+2), approval policy (Planner–Executor vs. unanimous), memory surfaces (separable advice vs. shared transcript), and prompt scaffolds (Table 12, I.4). The paper honestly acknowledges this and labels the within-PER probes as directional, but the abstract, title, and conclusion present "reviewer precision and critique uptake are empirically separable" as the main result. Because the claimed practical lesson is about reviewer–solver coupling, not about budgets or approval rules, this central claim needs either a matched routing ablation that holds those other dimensions fixed or a consistently scoped formulation in the title and abstract. The EMB probe does not provide that isolation: it changes the set of useful-review episodes and does not r
  2. [Table 14 vs. §4.2 and Fig. 4] The PER+EMB useful-coupling value is reported as 0.247 in Table 14 but 0.227 in the main text and Fig. 4. This internal inconsistency must be resolved. More importantly, relative to base PER, EMB worsens NeglectRate (0.488 → 0.698) and lowers CouplingRate (0.336 → 0.247/0.227), even though it improves FinalPassRate. The paper reads "Base PER → EMB → Broadcast" as a sequence of progressively more direct feedback, but the mechanism diagnostics do not move monotonically. The claim that EMB "partially restores coupling" is only true relative to ACK-required, not relative to base PER. The mechanism interpretation therefore needs a trajectory-level explanation or a substantially more cautious wording.
  3. [§3.4, Appendix B, Appendix H.4] Precision, Recall, CouplingRate, and ReviewerGuidedRepairRate all depend on the evaluator label for the pre-review internal candidate (z(0)), and that label comes from gpt-oss-120b, the same model family as the actors. Appendix B replays only submitted final answers; it does not replay the internal pre-review candidate labels. The cross-evaluator disagreement rises to roughly 10% on the hard collaborative slice (tiers 7–10), which is exactly where the main PER–broadcast gap concentrates. The paper should either provide a cross-evaluator replay of the internal z(0) labels or explicitly bound how evaluator disagreement on those labels affects the reported CouplingRate and repair-rate differences.
minor comments (5)
  1. [§4.2, Fig. 4] The sentence "EMB reduces neglect (0.698 NeglectRate)" compares to ACK-required (0.792), not to base PER (0.488). Please state the comparison base explicitly to avoid implying EMB improves on base PER's neglect rate.
  2. [Appendix P] The PER and broadcast trace excerpts come from different problems. The main text should state this explicitly at the point of citation; the appendix does disclose it, but a reader of the main text could infer a same-question pairing.
  3. [Appendix I.2] Code and data are listed as pending institutional approval. Given that several headline metrics are defined by a new measurement framework, the lack of a currently accessible artifact weakens independent verification. Please clarify the expected release timeline or provide a reviewer-accessible snapshot.
  4. [Appendix H.4] The aliases LegacyCouplingRate, UsefulCoupling, and CouplingRate are used across tables and text. Unifying these names would reduce the risk of confusion about which quantity is reported in each table.
  5. [Appendix B] The cross-evaluator replay is computed on submission instances, not on deduplicated unique candidates in both blocks. The text notes the unique-candidate replay is slightly harsher; please include those numbers or state why they are omitted.

Circularity Check

0 steps flagged

No circularity: the precision–uptake separation is an empirical tabulation from independently defined trace metrics, not a fit or self-citation.

full rationale

The paper's central claim—that reviewer precision and critique uptake are empirically separable—rests on direct tabulations of trace labels with distinct operational definitions. Precision is TP/(TP+FP); CouplingRate is the fraction of evaluator-verified useful revise episodes in which the immediately following candidate answer changes; ReviewerGuidedRepairRate is C_{0,Revise,1}/TP. These quantities are not algebraically forced to separate; the observed ordering (PER precision 0.861 vs. broadcast 0.644, but CouplingRate 0.336 vs. 0.935 and repair 0.051 vs. 0.286) is an empirical result, not a consequence of the definitions. No parameter is fitted to the outcome it is then used to explain, and no 'prediction' is a renamed fit. The PER–broadcast contrast is a whole-protocol comparison, but the paper explicitly disclaims causal isolation ('The PER–broadcast comparison is evidence about complete protocol designs... not as an isolation of one factor at a time') and labels the within-PER probes as directional. That is a confound/identifiability limitation, not definitional circularity. The same-evaluator dependency (gpt-oss-120b labels both correctness and useful critique) is a shared-source validity concern, mitigated by the cross-evaluator replay, but it does not make any derived quantity equal to an input by construction. No load-bearing argument reduces to a self-citation: the cited related-work results are external and are used for context, not as the paper's evidence. The paper therefore shows no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No numbers are fitted to make the result appear; the central quantities are computed directly from trace labels. The key assumptions are the reliability of the evaluator, the validity of the answer-transition operationalization, and the comparability of the whole-protocol comparison.

axioms (4)
  • domain assumption Evaluator verdicts from gpt-oss-120b correctly identify wrong/correct candidates.
    All coupling/repair metrics condition on evaluator-verified 'useful' episodes. In the hardest shared modes, cross-evaluator disagreement reaches 9-10% (Appendix B), so evaluator error could shift the measured precision-uptake gap.
  • domain assumption Boxed-answer extraction and normalized string comparison faithfully represent whether the next candidate changed.
    CouplingRate and NeglectRate are defined on extracted answers. The strict/equivalence-aware audit in Table 3 shows stability, but semantic reformulations with identical normalized strings are possible.
  • ad hoc to paper Protocol-level configuration is matched enough that PER-broadcast differences reflect critique routing rather than other design dimensions.
    The authors state protocols differ in shared state, approval rules, inner rounds, and prompt surfaces; they frame the comparison as about whole designs, not isolated routing (Sec. 3.4, App. I.4). The central inference leans on this framing.
  • domain assumption Omni-MATH tier labels provide a meaningful difficulty ordering.
    The tier axis is hybrid metadata, mostly AoPS community ratings with GPT-4o fill-in; tiers 3 and 10 are very small (n=20 and n=15) and the paper flags them.

pith-pipeline@v1.3.0-alltime-deepseek · 40862 in / 9856 out tokens · 94022 ms · 2026-08-01T23:29:20.064998+00:00 · methodology

0 comments
read the original abstract

Many math- and science-oriented agent systems use hierarchical designs with specialized reviewer roles, assuming that a dedicated review stage should help turn wrong candidates into correct ones. We test this assumption on 4,181 verifier-grounded Omni-MATH problems using matched gpt-oss-120b actors. Collaboration adds little on the easiest tiers, but from tier 4 onward the gains open sharply; in this harder regime, broadcast-style peer discussion reaches higher final accuracy than a planner-executor-reviewer pipeline (PER). We ask whether this gap is explained by reviewer quality or by whether critique changes the next answer the protocol carries forward. It is not explained by reviewer precision alone: PER's reviewer is more precise than broadcast's (0.861 vs. 0.644), yet evaluator-verified useful critique is much less likely to change the next candidate and produces lower reviewer-guided repair. These results show that reviewer detection quality and critique uptake are empirically separable. Within matched PER interventions, forcing explicit acknowledgment lowers final accuracy, while embedding reviewer guidance directly in the solver's working context partially improves follow-through without closing the gap. Overall, reviewer-centric evaluation can overstate system quality: a protocol may spot errors well yet still fail to solve more problems if it does not act on those critiques.

Figures

Figures reproduced from arXiv: 2607.15388 by Cheng-Hau Yang, Chih-Hsuan Yang, Eliu A. Huerta, Huihuo Zheng, Ian T. Foster, Jingyan Jiang, Le Chen, Rajeev Thakur, Venkatram Vishwanath, Vikram Vasudevan.

Figure 1
Figure 1. Figure 1: Why reviewer quality and realized repair can diverge. In PER, information flow and decision [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Tier-level collaboration gain over Single-Agent Iterative. Single-Agent Iterative is the matched non-collaborative anchor: it already has iterative opportunity and verifier interaction, so gain over Single-Agent more directly reflects what deliberation and review contribute. (a) FinalPassRate gain. Gains are near zero on tiers 1–2 and open sharply from tier 4 onward. (b) Pass@1 gain. The same pattern holds… view at source ↗
Figure 3
Figure 3. Figure 3: What observable process difference tracks the PER–broadcast gap? Not reviewer precision alone, but whether evaluator-verified useful critique changes the next candidate the protocol carries forward. Left: NeglectRate is the dominant inner-loop failure mode for PER (48.8% of wrong-initial-candidate cases neglect available critique), while TryButFailRate is much closer across protocols. Right: the aggregate … view at source ↗
Figure 4
Figure 4. Figure 4: Within-PER intervention evidence for the uptake story. All values are full-benchmark aggregates (N = 4,181); the model family, reviewer setup, and outer-loop budget are held fixed across conditions, varying only how critique reaches the solver. (a) Final outcomes. ACK￾required lowers FinalPassRate (85.2%→82.5%) and Pass@1, while EMB partially recovers both. (b) Detection→uptake→repair. ACK-required sharply… view at source ↗
Figure 5
Figure 5. Figure 5: Cross-evaluator disagreement robustness. (a) Grouped bars compare pairwise disagree￾ment rates on the full benchmark (N = 16,724; light bars) versus the focused hard collaborative slice (PER and Broadcast, tiers 7–10, N = 1,694; dark bars). Cohen’s κ is annotated above each bar. Overall agreement is high (disagreement 3.4–5.8%, κ = 0.850–0.915), but disagreement roughly doubles on the hard collaborative sl… view at source ↗
Figure 6
Figure 6. Figure 6: Deeper-reflection PER stress test: PER-inner6 (brown-orange) increases local reflective [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Cumulative outer-loop pass rate across three outer attempts. (b) Conditional recovery rate at attempt k (CRR@k), meaning the probability of solving the problem at attempt k given failure on all earlier attempts. Broadcast’s advantage concentrates at the first attempt (78.6% first-pass success), while CRR@2 is nearly tied with Single-Agent (37.2% vs. 37.0%). This makes CRR@2 the key quantity when verifi… view at source ↗
Figure 8
Figure 8. Figure 8: FinalPassRate heatmap by protocol and difficulty tier. Each cell shows the percentage of [PITH_FULL_IMAGE:figures/full_fig_p031_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Inner-loop transition rates by difficulty tier for PER and Broadcast. The three rates are [PITH_FULL_IMAGE:figures/full_fig_p032_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Outer-loop recovery curves for Single-Agent Iterative, PER, and Broadcast across three [PITH_FULL_IMAGE:figures/full_fig_p032_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: NeglectRate (left) and RepairRate (right) by difficulty tier for PER and Broadcast. Shaded [PITH_FULL_IMAGE:figures/full_fig_p033_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Reviewer confusion matrix proportions (TP/FP/FN/TN) by difficulty tier for all protocols. [PITH_FULL_IMAGE:figures/full_fig_p033_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Reviewer false alarm rate (FAR) by difficulty tier. FAR measures the fraction of correct [PITH_FULL_IMAGE:figures/full_fig_p034_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: Tiers 1–4 ≈ 1.32 × 105 ≈ 1.05 × 106 On easier problems, broadcast needs a relatively lower verifier￾cost premium than in the global av￾erage to overtake PER, but single￾agent remains hard to dislodge. Tiers 5–6 ≈ 1.92 × 105 ≈ 1.04 × 106 Mid-difficulty problems already move the PER/broadcast crossover upward relative to tiers 1–4. Tiers 7–10 ≈ 2.89 × 105 ≈ 7.18 × 105 Hard tiers shift the PER/broadcast cros… view at source ↗
Figure 14
Figure 14. Figure 14: (a) Raw cost–performance frontier. Generation tokens per problem versus FinalPassRate for all four protocols. Single-Agent achieves the best tokens-per-extra-solved-problem ratio; PER and Broadcast incur substantially higher generation costs for their respective accuracy gains. Notably, PER is not simply intermediate in cost: it uses more verifier calls per problem than Broadcast (2.19 vs. 1.35), as annot… view at source ↗
Figure 15
Figure 15. Figure 15: Verifier-cost crossover analysis under the scalarized objective [PITH_FULL_IMAGE:figures/full_fig_p036_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Harm-aware protocol decision map for tiers 7–10 under the richer scalarized objective [PITH_FULL_IMAGE:figures/full_fig_p037_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Average evaluator (verifier) calls per problem by difficulty tier and protocol. Across tiers [PITH_FULL_IMAGE:figures/full_fig_p038_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Wall-clock time per problem by difficulty tier and protocol. At tier 10, PER averages [PITH_FULL_IMAGE:figures/full_fig_p038_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Multi-metric overview across all tiers and protocols. The panels show FinalPassRate, [PITH_FULL_IMAGE:figures/full_fig_p039_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Five-dimensional detection–coupling–repair profile for PER (dashed orange) and Broad [PITH_FULL_IMAGE:figures/full_fig_p040_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 24 linked inside Pith

  1. [1]

    How many tries does it take? Iterative self-repair in LLM code generation across model scales and benchmarks.arXiv preprint arXiv:2604.10508, 2026

    Johin Johny Arimbur. How many tries does it take? Iterative self-repair in LLM code generation across model scales and benchmarks.arXiv preprint arXiv:2604.10508, 2026. URL https: //arxiv.org/abs/2604.10508

    Pith/arXiv arXiv 2026

  2. [2]

    Benchmarks saturate when the model gets smarter than the judge.arXiv preprint arXiv:2601.19532, 2026

    Marthe Ballon, Andres Algaba, Brecht Verbeken, and Vincent Ginis. Benchmarks saturate when the model gets smarter than the judge.arXiv preprint arXiv:2601.19532, 2026. URL https://arxiv.org/abs/2601.19532

    arXiv 2026

  3. [4]

    xverify: Efficient answer verifier for reasoning model evaluations.arXiv preprint arXiv:2504.10481, 2025

    Ding Chen, Qingchen Yu, Pengyuan Wang, Mengting Hu, Wentao Zhang, Zhengren Wang, Bo Tang, Feiyu Xiong, Xinchi Li, Chao Wang, Minchuan Yang, and Zhiyu Li. xverify: Efficient answer verifier for reasoning model evaluations.arXiv preprint arXiv:2504.10481, 2025. URL https://arxiv.org/abs/2504.10481

    arXiv 2025

  4. [5]

    A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1):37–46, 1960

    Jacob Cohen. A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1):37–46, 1960. doi: 10.1177/001316446002000104

    doi:10.1177/001316446002000104 1960

  5. [6]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate.arXiv preprint arXiv:2305.14325, 2023. URLhttps://arxiv.org/abs/2305.14325

    Pith/arXiv arXiv 2023

  6. [7]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint ar...

    Pith/arXiv arXiv 2024

  7. [8]

    Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2024

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2024. URLhttps://arxiv.org/abs/2310.01798

    Pith/arXiv arXiv 2024

  8. [9]

    Siegel, Nitya Nadgir, and Arvind Narayanan

    Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter.arXiv preprint arXiv:2407.01502, 2024. URL https://arxiv.org/abs/ 2407.01502. 10

    Pith/arXiv arXiv 2024

  9. [11]

    Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, Mark Malhotra, and Paul Pu Liang

    Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A. Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, Mark Malhotra, and Paul Pu Liang. Towards a science of scaling agent systems.arXiv preprint arXiv:2512.08296, 2025. URL https: //arxiv.org/abs/2512.08296

    Pith/arXiv arXiv 2025

  10. [12]

    Richard Landis and Gary G

    J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data.Biometrics, 33(1):159–174, 1977. doi: 10.2307/2529310

    doi:10.2307/2529310 1977

  11. [13]

    CAMEL: Communicative agents for mind exploration of large scale language model society.arXiv preprint arXiv:2303.17760, 2023

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for mind exploration of large scale language model society.arXiv preprint arXiv:2303.17760, 2023. URL https://arxiv.org/abs/2303. 17760

    Pith/arXiv arXiv 2023

  12. [14]

    Decomposing LLM self-correction: The accuracy-correction paradox and error depth hypothesis.arXiv preprint arXiv:2601.00828, 2026

    Yin Li. Decomposing LLM self-correction: The accuracy-correction paradox and error depth hypothesis.arXiv preprint arXiv:2601.00828, 2026. URL https://arxiv.org/abs/2601. 00828

    arXiv 2026

  13. [15]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents. InInternational Conference on Learning Representations,...

    Pith/arXiv arXiv 2024

  14. [16]

    Self- refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback. InAdvances in Neural Information Processing Sy...

    Pith/arXiv arXiv 2023

  15. [17]

    Mary L. McHugh. Interrater reliability: The kappa statistic.Biochemia Medica, 22(3):276–282,

  16. [18]

    Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip H. S. Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent LLM training.arXiv preprint arXiv:2412.01928, 2024. URL https://arxiv.org/abs/2412.01928

    arXiv 2024

  17. [19]

    On evaluating the integration of reasoning and action in LLM agents with database question answering.arXiv preprint arXiv:2311.09721, 2023

    Linyong Nan, Ellen Zhang, Weijin Zou, Yilun Zhao, Wenfei Zhou, and Arman Cohan. On evaluating the integration of reasoning and action in LLM agents with database question answering.arXiv preprint arXiv:2311.09721, 2023. URL https://arxiv.org/abs/2311. 09721

    Pith/arXiv arXiv 2023

  18. [20]

    gpt-oss-120b model, 2025

    OpenAI. gpt-oss-120b model, 2025. URL https://developers.openai.com/api/docs/ models/gpt-oss-120b. Accessed: 2026-05-01

    2025

  19. [21]

    Introducing gpt-oss, 2025

    OpenAI. Introducing gpt-oss, 2025. URL https://openai.com/index/ introducing-gpt-oss/. Accessed: 2026-05-01

    2025

  20. [22]

    Towards scientific intelligence: A survey of llm-based scientific agents.arXiv preprint arXiv:2503.24047, 2025

    Shuo Ren, Can Xie, Pu Jian, Zhenjiang Ren, Chunlin Leng, and Jiajun Zhang. Towards scientific intelligence: A survey of llm-based scientific agents.arXiv preprint arXiv:2503.24047, 2025. URLhttps://arxiv.org/abs/2503.24047

    arXiv 2025

  21. [23]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2303.11366. 11

    Pith/arXiv arXiv 2023

  22. [24]

    Can llms correct them- selves? a benchmark of self-correction in llms.arXiv preprint arXiv:2510.16062, 2025

    Guiyao Tie, Zenghui Yuan, Zeli Zhao, Chaoran Hu, Tianhe Gu, Ruihang Zhang, Sizhe Zhang, Junran Wu, Xiaoyue Tu, Ming Jin, Qingsong Wen, and Lixing Chen. Can llms correct them- selves? a benchmark of self-correction in llms.arXiv preprint arXiv:2510.16062, 2025. URL https://arxiv.org/abs/2510.16062

    arXiv 2025

  23. [25]

    Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D. Nguyen. Multi-agent collaboration mechanisms: A survey of llms.arXiv preprint arXiv:2501.06322, 2025. URLhttps://arxiv.org/abs/2501.06322

    Pith/arXiv arXiv 2025

  24. [26]

    Self-correction bench: Uncovering and addressing the self-correction blind spot in large language models.OpenReview, 2025

    Ken Tsui. Self-correction bench: Uncovering and addressing the self-correction blind spot in large language models.OpenReview, 2025. URL https://openreview.net/forum?id= 7K1kXowjK1

    2025

  25. [27]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations, 2023. URL https:// arxiv.org/abs/2203.11171

    Pith/arXiv arXiv 2023

  26. [28]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, 2022. URL https://arxiv. org/abs/2201.11903

    Pith/arXiv arXiv 2022

  27. [29]

    Can LLM agents really debate? a controlled study of multi-agent debate in logical reasoning.arXiv preprint arXiv:2511.07784, 2025

    Haolun Wu, Zhenkun Li, and Lingyao Li. Can LLM agents really debate? a controlled study of multi-agent debate in logical reasoning.arXiv preprint arXiv:2511.07784, 2025. URL https://arxiv.org/abs/2511.07784

    arXiv 2025

  28. [30]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. URLhttps://arxiv.org/abs/2308.08155

    Pith/arXiv arXiv 2023

  29. [31]

    Con- fidence v.s

    Zhe Yang, Yichang Zhang, Yudong Wang, Ziyao Xu, Junyang Lin, and Zhifang Sui. Con- fidence v.s. critique: A decomposition of self-correction capability for llms.arXiv preprint arXiv:2412.19513, 2024. URLhttps://arxiv.org/abs/2412.19513

    Pith/arXiv arXiv 2024

  30. [32]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/ 2305.10601

    Pith/arXiv arXiv 2023

  31. [33]

    Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416, 2025

    Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416, 2025. URLhttps://arxiv.org/abs/2503.16416

    Pith/arXiv arXiv 2025

  32. [34]

    Reinforce LLM reasoning through multi-agent reflection.arXiv preprint arXiv:2506.08379, 2025

    Yurun Yuan and Tengyang Xie. Reinforce LLM reasoning through multi-agent reflection.arXiv preprint arXiv:2506.08379, 2025. URLhttps://arxiv.org/abs/2506.08379

    Pith/arXiv arXiv 2025

  33. [35]

    A survey on test- time scaling in large language models: What, how, where, and how well?arXiv preprint arXiv:2503.24235, 2025

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, and Xue Liu. A survey on test- time scaling in large language models: What, how, where, and how well?arXiv preprint arXiv:2503.24235, 2025. URLhttps://arxiv.org/abs/2503.24235

    Pith/arXiv arXiv 2025

  34. [36]

    Small language models need strong verifiers to self-correct reasoning

    Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct reasoning. InFindings of the Association for Computational Linguistics: ACL 2024, 2024. URL https://aclanthology.org/2024.findings-acl.924/

    2024

  35. [37]

    Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators.arXiv preprint arXiv:2504.15253, 2025

    Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators.arXiv preprint arXiv:2504.15253, 2025. URLhttps://arxiv.org/abs/2504.15253. 12

    Pith/arXiv arXiv 2025

  36. [38]

    Self-correction blind spot

    Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Percy Liang, and Daniel Kang. ...

    Pith/arXiv arXiv 2025

  37. [42]

    useful” and “misleading

    (single-agent analog) Never measured in multi-agent settings TryButFailRate(new)Among reviewer-flagged wrong answers, fraction where solver revises but answer remains wrong Error depth hypothesis [14] (single-agent) Not separated from neglect—opposite implications for system design Reviewer–Solver Coupling(new) Whether a reviewer’s correct flag reliably p...

  38. [43]

    Filter to labeled inner-loop transitions with both initial and final correctness labels

  39. [44]

    Restrict to transitions whose initial candidate is wrong

  40. [45]

    CountNeglectwhen the final candidate is still wrong and the normalized final answer is unchanged from the initial one

  41. [46]

    CountTryButFailwhen the final candidate is still wrong but the normalized final answer changed

  42. [47]

    wrong → same wrong answer,

    Divide both counts by the number of wrong-initial transitions to obtain NeglectRate and TryBut- FailRate. This corresponds exactly to the implementation of the paper’s inner-loop met- ric module: ReflectiveNeglectRate is “wrong → same wrong answer,” while ReflectiveTryButFailRate is “wrong → changed-but-still-wrong answer.” Readers who want to port these ...

    2021

  43. [48]

    Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...

  44. [2012]

    doi: 10.11613/BM.2012.031

    doi:10.11613/bm.2012.031 2012

  45. [2024]

    URLhttps://arxiv.org/abs/2407.04622

    Pith/arXiv arXiv

  46. [2025]

    URLhttps://arxiv.org/abs/2503.13657

    Pith/arXiv arXiv