Pith. sign in

REVIEW 1 major objections 4 minor 15 references

Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agents

T0 review · 1 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A noisy verify-repair loop should stop or continue according to the sign of the one-step expected gain, and a rule built on this lifted final true validity by 60.6 percentage points.

desk verdict A serious, well-evidenced paper on when to stop verify-repair loops, with a sound core framework and clearly flagged approximation assumptions that can flip the decision in low-discrimination regimes. read the letter →

arxiv 2607.17641 v1 pith:GT34EFSS submitted 2026-07-20 cs.AI cs.SE

Yitao Wu , Si Shen , Rui Yang , Hong Peng , Bin Hu This is my paper
classification cs.AIcs.SE
keywords verify-repairloopsLLMagentsstoppingrulebelieffilteringmarginalgainverifiernoisediscriminationkeep-bestfallback
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that deciding when to stop repairing a plan in an LLM agent loop is an identifiable sign decision, not a heuristic. It models verifier false acceptance, verifier false rejection, repair success, and repair damage with four parameters, filters repeated verification votes into a belief of true validity, and continues repair only while the expected one-step gain G = (1-b)α - bβ is positive, stopping at boundary b* = α/(α+β). A sympathetic reader would care because fixed repair budgets and pass-rate thresholds can silently destroy correct plans — the paper shows a stress setting where reported acceptance rises while true validity collapses, and the new rule lifts final true validity by 60.6 percentage points over fixed five-round repair. The paper is explicit about its boundaries: the model assumes local stationarity and conditional independence (Section 3.2), does not locate interior peaks under abrupt repair changes (Section H.2), and uses a binary validity state (Section H.6). When verifier discrimination approaches zero, calibrated stopping can fail, and a margin-based keep-best fallback restores near-no-repair safety.

What carries the argument

The four-parameter noise model: ρ0 = false acceptance, ρ1 = false rejection, α = repair success rate, β = damage rate, with verifier discrimination J = 1 - ρ0 - ρ1. Carrying the argument is the belief recursion: after M conditionally independent votes, the posterior b_k updates by Bayes' rule, and the prediction step is b^-_{k+1} = (1-β)b_k + α(1-b_k). These combine into the marginal gain G_k = b^-_{k+1} - b_k = (1-b_k)α - b_kβ, whose zero crossing defines the stopping boundary b* = α/(α+β). The sign of G_k is the decision rule. VRR-Guard is the secondary mechanism: a keep-best rule that replaces the incumbent only when the new plan's vote count exceeds the champion's by margin δ, with a con

What would settle it

Run VRR-Stop on a trajectory set where verifier votes are made dependent (e.g., a single judge that sees its own previous answers, or self-consistency voting with shared drafting) and compare its accept/repair decisions against ground-truth validity labels: if the sign-flip rate systematically exceeds the rate predicted by the conditional-independence model, the core premise is falsified. A complementary check: label per-round repair outcomes and test whether the sign of (1-b_t)α_t - b_tβ_t computed from round-lagged estimates disagrees with the round-1-parameter decision on trajectories where

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the optimal myopic stop/continue decision in a noisy verify-repair loop is determined solely by the sign of G_k = (1 - b_k)α - b_kβ, where b_k is the posterior probability that the current plan is valid. The threshold b* = α/(α+β) is the fixed point of the valid-validity dynamics and does not depend on verifier noise; verifier noise enters only through how reliably the belief can be located. The paper proves a stopping-sign consistency bound (Proposition 1), gives an error-radius criterion for when the sign is identifiable, and shows empirically that reliability is governed by the verifier discrimination J and the decision margin Δ, not by the

Load-bearing premise

The load-bearing premise is that repeated verifier votes are conditionally independent given true validity and that the four noise and repair parameters are stable within a local decision window — the paper itself calls this a local approximation (Sections 3.2, 4.1, and H.2) — so if votes are correlated or repair dynamics drift mid-loop, the estimated sign of the marginal gain can flip and the stopping decision is no longer reliable.

Editorial extensions

If this is right

  • If the sign-criterion is correct, fixed repair budgets are an unsafe default: in six of the eight evaluated settings unconditional repair drives true validity down monotonically, with measured damage probabilities up to 0.94.
  • Pass-rate thresholds and majority voting can actively mislead, since reported acceptance can rise while true validity falls; stopping rules should be built on the estimated belief of true validity, not raw acceptance.
  • Stopping can be calibrated from weak supervision: repeated verification votes plus at most 300 labeled repair transitions suffice to locate the sign of the marginal gain when verifier discrimination J and the decision margin are not too small.
  • Near-zero verifier discrimination is a distinct failure mode where more calibration data can make estimates worse; the margin-based keep-best fallback recovers near-no-repair validity instead of collapsing.
  • The cost profile makes the rule practical: one-off calibration, M verifier calls per round, and an average of 0.72 repair rounds on the stress setting — better validity than no-repair and fixed repair at a fraction of the compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the stopping boundary b* varies from 0.954 in the favorable setting to 0.289 in the stress setting, any universal confidence threshold is unsafe; this implies that per-task or even per-instance estimates of α and β, keyed by difficulty, would make the boundary adaptive.
  • Editorial inference: VRR-Guard's concentration-bound argument suggests a model-agnostic 'keep the champion, replace only on a vote margin' protocol that could be layered onto any iterative refinement loop where calibrated parameters are unavailable; testing it as a standalone wrapper is a direct, low-cost extension.
  • Editorial inference: given the paper's Section H.2 admission that it cannot detect abrupt repair-mechanism changes, a natural next step is online monitoring of vote-stream residuals or per-round transition rates to switch from calibrated stopping to the guarded fallback mid-trajectory.
  • Editorial inference: the binary-validity state (Section H.6) is the main structural limit; extending the belief to a structured state of partial correctness would replace the scalar boundary b* with a set of state-dependent thresholds, and long-horizon tool-use plans are a plausible testbed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper studies stopping rules for LLM verify-repair loops when both the verifier and the repairer are noisy. It introduces a four-parameter model (verifier false-acceptance ρ0, false-rejection ρ1, repair success α, repair damage β), a belief-filtering recursion that estimates the probability that the current plan is truly valid, and a myopic stopping rule based on the sign of the one-step marginal gain G_k = (1−b_k)α − b_kβ, with boundary b* = α/(α+β). When calibrated stopping is unreliable, the paper proposes VRR-Guard, a keep-best fallback that replaces the incumbent candidate only if the new plan receives at least δ more verifier votes. Experiments on GSM8K, MATH-500, MBPP, and BFCL claim large gains over fixed-budget repair, with a headline 60.6-point improvement on a GSM8K prompt-mismatch stress setting, and show that VRR-Guard restores validity when calibrated stopping collapses. The paper is transparent about local-stationarity assumptions and the limitation of binary validity, and it releases replay artifacts.

Significance. If the framework holds up, it addresses a genuinely important problem: verify-repair loops are ubiquitous in LLM agents, and existing stopping heuristics can commit corrupted plans. The paper's strengths include a clean myopic-gain decomposition, a recognizably correct derivation of the belief recursion and stopping boundary, five-fold cross-fitting for calibration, paired McNemar/bootstrap tests for headline claims, and an unusually candid discussion of failure modes and the boundaries of the method. The empirical map of decision-flip risk as a function of verifier discrimination J and decision margin Δ is a useful conceptual contribution, and the guarded keep-best fallback is a simple, plausible safety mechanism. However, as detailed below, the central guarantee of the method is conditional on assumptions that the paper's own data show to be violated in the deployed regime, and the deployed rule does not use the certified interval of Proposition 1.

major comments (1)
  1. [§5.4, Table 8] The identifiability-collapse analysis on Llama is presented as the mechanism behind VRR-Stop's failure, but the evidence is partly circular. The claim that calibrated stopping recovers only for ρ̂1 ≳ 0.30–0.35 is obtained from a parameter sweep under the deployed rule, which itself assumes the binomial model. Since the true failure of the deployed rule may be due to model misspecification rather than solely to EM estimation error, the proposed separation between 'estimation error flips the sign' and 'model misspecification flips the sign' is not established. A more direct test would be to evaluate the stopping rule with true parameters on the actual Llama trajectories, which would isolate the misspecification component.
minor comments (4)
  1. [Algorithm 1 vs. §4.2] Algorithm 1 switches mode to Guard when Eq. (8) fails, but the main text says experiments use VRR-Stop acting directly on the sign of Ĝ_k without the interval. Please clarify whether Algorithm 1 describes the intended deployment procedure or the experimental protocol, and reconcile the discrepancy.
  2. [§4.3, Lemma 1] Lemma 1's bound is stated for a single round and then multiplied by K_max using a union bound. It would be helpful to note that this union bound is conservative, especially when erroneous replacements are positively correlated across rounds, as they may be under within-class difficulty heterogeneity.
  3. [§5.4, Fig. 5] The colorbar label 'Pr[sign flip]' is cut off in the figure. Please ensure the axis label is legible in the final version.
  4. [Appendix H.6] The discussion of α+β ≤ 1 is useful, but it is stated as a 'strict condition' without explaining why it should hold for a well-behaved repair operator. Consider adding a brief derivation or reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the stopping rule is derived from explicit model assumptions, and the headline evaluations use five-fold cross-fitting so test-instance labels do not enter parameter estimation.

full rationale

The paper's central derivation chain is self-contained and non-circular. The belief recursion (Eq. 5), the marginal-gain expression (Eq. 6), and the stopping boundary b* = α/(α+β) (Eq. 7) follow from the stated four-parameter noise model and the conditional-independence/local-stationarity assumptions; b* is the fixed point of the assumed affine population dynamics, not an input disguised as an output. Calibration estimates ρ0, ρ1, α, β from labeled transition samples and an unlabeled binomial mixture, and the main VRR-Stop evaluation explicitly uses five-fold cross-fitting (Appendix B), so no instance's stopping decision touches its own labels. This breaks the 'fitted input called prediction' pattern. Proposition 1 and Lemma 1 are conditional probability statements proved from the stated premises, with no imported uniqueness theorem or author-self-citation carrying the argument. The paper itself flags the relevant limitations: Appendix H.2 states that using first-round transition estimates for the whole process 'should therefore be regarded only as a locally stationary approximation'; Appendix H.6 notes that the strict condition α+β≤1 is violated on one trajectory and that conclusions there rest on single-round marginal gains; Appendix A.5 explicitly disclaims an explicit sample-complexity bound for the deployed rule. These are robustness/correctness concerns, not circularity. One minor evaluation-selection concern is that VRR-Guard's retention margin δ=5 was chosen by replay on all seven settings and then reported on those same settings (Appendix I). This is a hyperparameter selection on the evaluation set and could overstate robustness, but it is not a construction-level reduction of the reported validity to a fitted quantity, so it does not constitute circularity under the standards here.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central model has four fitted noise/transition parameters plus a fitted prior; these are not derived from first principles but calibrated per setting. VRR-Guard adds a tuned margin δ. No new physical entities are postulated. Independent grounding comes from five-fold cross-fitted empirical evaluation.

free parameters (6)
  • α (repair success rate) = 0.320 (Qwen-3B PM-stress); 0.423 favorable; 0.051 Llama
    Estimated from labeled before/after repair pairs; directly sets the stopping boundary b* = α/(α+β).
  • β (repair damage rate) = 0.786 (Qwen-3B PM-stress); 0.938 Llama; 0.617 MATH-500
    Estimated from labeled transitions; drives the damage term in G_k = (1-b)α - bβ.
  • ρ0 (verifier false acceptance rate) = 0.364 (Qwen-3B PM-stress); 0.875 BFCL single
    Estimated by binomial-mixture EM; enters the belief update Eq. (5).
  • ρ1 (verifier false rejection rate) = 0.177 (Qwen-3B PM-stress); 0.609 Llama
    Estimated by binomial-mixture EM; when underestimated (Llama), posterior belief is inflated and stopping decisions collapse.
  • π (prior initial-plan validity) = 0.700 in the main stress setting (no-repair baseline)
    Frequency estimate on calibration folds; used as the initial belief b0^-.
  • δ (VRR-Guard retention margin) = 5
    Selected by pure replay sweep over all seven evaluation settings (§I); governs the replacement rule Eq. (9).
assumptions (5)
  • domain assumption M verification queries are conditionally independent and identically distributed given the true validity y_k (binomial observation model).
    Used to derive Eq. (5). If votes are correlated (judge position bias, plan-specific difficulty), the posterior b_k is misspecified. The paper acknowledges this and diagnoses via beta-binomial in Appendix E.
  • domain assumption ρ0, ρ1, α, β are stable within a local decision window (local stationarity).
    Throughout §3.2 and §4.1; parameters estimated from round-1 frozen trajectories are applied to later rounds. The paper calls this a local approximation and tests shifts but does not provide online drift detection.
  • domain assumption Repair transition is conditionally independent of the verification history H_k given y_k.
    Used for the predict step b^-_{k+1} = (1-β)b_k + α(1-b_k) in §4.1. Calibration samples are collected by forcing repair every round, while deployment repairs only under the policy's Repair decisions; selection can make transition rates depend on b_k beyond y_k.
  • domain assumption True plan quality is a binary valid/invalid state.
    The belief/stopping machinery is built on y_k ∈ {0,1}; partial correctness, multiple error types, and staged goals are excluded (§H.6).
  • standard math The two-component binomial mixture is identifiable (component acceptance rates differ, i.e. J ≠ 0).
    Needed for EM calibration; as J→0 the likelihood flattens and the estimator collapses, which the paper documents as the Llama failure mode.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agents." pith.science (2026). https://pith.science/paper/GT34EFSS

@misc{pith2026260717641,
  author       = {Pith},
  title        = {Pith review of: Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GT34EFSS}},
  note         = {Machine review of arXiv:2607.17641}
}
read the original abstract

Verify-repair loops are a standard means for large language model (LLM) agents to correct faulty plans in code generation, mathematical reasoning, and tool use. When both the verifier and the repairer are noisy, repair can damage already-correct plans, and reported acceptance keeps rising while true validity falls, so existing methods lack a principled basis for deciding when repair should stop. We propose VRR-Stop, a robust stopping framework for noisy verify-repair-repeat (VRR) loops. A four-parameter noise model separates verifier false acceptance and false rejection from the repair and damage behavior of the repairer. Belief filtering turns repeated verification votes into an estimate of committed validity, and the loop commits or repairs according to the sign of the true marginal gain, which requires only sign identifiability rather than accurate recovery of all parameters. When verifier discrimination approaches zero, calibration itself fails and estimation error can flip the stopping sign, so we pair VRR-Stop with VRR-Guard, an estimation-free fallback that replaces the incumbent candidate only under a sufficient verification margin. On a GSM8K stress setting, VRR-Stop improves final true validity by 60.6 percentage points over fixed five-round repair at an average cost of 0.72 repair rounds. Across settings, stopping reliability is governed jointly by verifier discrimination and the decision margin rather than by the absolute size of estimation error.

Figures

Figures reproduced from arXiv: 2607.17641 by the authors.

Figure 1
Figure 1. Clean feedback accepts and commits the valid plan [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The population verification rate A¯ t can keep in￾creasing while the true-validity rate Qt peaks and then de￾clines. The shaded interval marks ∆Qt < 0, where continued repair becomes harmful. validity, and stop when it no longer exceeds a minimum-gain threshold, as specified in Eq. (4): Gk = E[yk+1 − yk | Hk, continue] , π(Hk) =  Repair, Gk > τ, Commit, Gk ≤ τ, (4) where τ ≥ 0 converts extra computation into a mini… view at source ↗
Figure 3
Figure 3. Overview of VRR-Stop, which estimates committed validity [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Loop dynamics under the non-stationary stress [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Stopping-sign flip probability over J and ∆ under the deployed rule; markers denote real verifiers. Flip risk concentrates at low J and low ∆, and near-zero J fails even with a large margin (Llama). flip risk concentrates where low J meets low ∆. The flip probability r…
Figure 6
Figure 6. Figure 6: Expected benefit G(b) = (1−b)α−bβ of running one more repair round, where b is the probability that the current plan is already correct, α is the probability that repair fixes an incorrect plan, and β is the probability that repair breaks a correct one; the three lines…
Figure 7
Figure 7. Figure 7: Mean true validity per repair round when every plan [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: End-to-end failure trace of GSM8K instance #292. The two top rails show the fork that drives it: the original problem [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Per-plan judge acceptance on the N=300 probe window (M=16), split by true validity, shown as per-plan rates with class means in panel (a) and as class distributions in panel (b). Solid lines mark the true mean acceptance of each class and dashed lines the label-free EM…
Figure 10
Figure 10. Figure 10: True validity versus mean repair rounds for each [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 8 linked inside Pith

  1. [3]

    InThe Fourteenth International Confer- ence on Learning Representations (ICLR)

    Deep Think with Confidence. InThe Fourteenth International Confer- ence on Learning Representations (ICLR). Gao,L.;Schulman,J.;andHilton,J.2023. ScalingLawsfor RewardModelOveroptimization. InProceedingsofthe40th InternationalConferenceonMachineLearning,volume202 ofProceedings of Machine Learning Research. Gou,Z.;Shao,Z.;Gong,Y.;Shen,Y.;Yang,Y.;Duan,N.;and...

    2023

  2. [5]

    In Vanschoren,J.;andYeung,S.,eds.,ProceedingsoftheNeu- ral Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021)

    Measuring Mathematical Problem Solving With the MATH Dataset. In Vanschoren,J.;andYeung,S.,eds.,ProceedingsoftheNeu- ral Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021). Huang, J.; Chen, X.; Mishra, S.; Zheng, H. S.; Yu, A. W.; Song,X.;andZhou,D.2024. LargeLanguageModelsCan- notSelf-CorrectReasoning...

    2021

  3. [8]

    arXiv:2512.02304

    When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers. arXiv:2512.02304. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P

    arXiv

  4. [9]

    In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds.,Advances in Neural Information Processing Systems, volume 36, 46534–46594

    Self-Refine: Itera- tive Refinement with Self-Feedback. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds.,Advances in Neural Information Processing Systems, volume 36, 46534–46594. Curran Associates, Inc. Pan,A.;Jones,E.;Jagadeesan,M.;andSteinhardt,J.2024a. Feedback Loops With Language Models Drive In-Context Reward Hackin...

    arXiv

  5. [11]

    InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics

    Stop When Enough: Adaptive Early-Stopping for Chain-of- Thought Reasoning. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics. SanDiego, California,USA: AssociationforComputational Linguistics. ArXiv:2510.10103. Wang,X.;Wei,J.;Schuurmans,D.;Le,Q.;Chi,E.;Narang, S.; Chowdhery, A.; and Zhou, D

    arXiv

  6. [12]

    arXiv:2412.15115

    Qwen2.5 Technical Report. arXiv:2412.15115. Yang, Z.; Zhang, Y.; Wang, Y.; Xu, Z.; Lin, J.; and Sui, Z

    arXiv

  7. [13]

    InProceedings of the 2025 Conference on EmpiricalMethodsinNaturalLanguageProcessing,13573– 13587

    A Probabilistic Inference Scaling Theory for LLM Self-Correction. InProceedings of the 2025 Conference on EmpiricalMethodsinNaturalLanguageProcessing,13573– 13587. Suzhou, China: Association for Computational Lin- guistics. Ye, Z.; Melo, L. C.; Kaddar, Y.; Blunsom, P.; Staton, S.; andGal,Y.2025. Uncertainty-AwareStep-wiseVerification with Generative Rewar...

    2025

  8. [14]

    arXiv:2502.00271

    Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning. arXiv:2502.00271. Zhang, L.; Hosseini, A.; Bansal, H.; Kazemi, M.; Kumar, A.; and Agarwal, R. 2025a. Generative Verifiers: Reward ModelingasNext-TokenPrediction. InTheThirteenthInter- national Conference on Learning Representations (ICLR). OpenReview.net. Zhang, Z.; Zheng, C.; Wu, Y.; Zhan...

    arXiv 2025

Show all 15 references
  1. [15]

    arXiv:2505.11966

    Solve-Detect-Verify: Inference-Time Scaling with Flexible Generative Verifier. arXiv:2505.11966. Zhou, Y.; Xu, A.; Wang, P.; Xiong, C.; and Joty, S

    arXiv

  2. [2016]

    InProceedingsofthe33rdInternationalConference onMachineLearning,volume48ofProceedingsofMachine Learning Research

    Es- timating Accuracy from Unlabeled Data: A Bayesian Ap- proach. InProceedingsofthe33rdInternationalConference onMachineLearning,volume48ofProceedingsofMachine Learning Research. Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao,S.2023. Reflexion:LanguageAgentswit...

    2023

  3. [2021]

    arXiv:2108.07732

    Program Synthesis with Large Language Models. arXiv:2108.07732. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse,C.;andSchulman,J.2021. TrainingVerifierstoSolve Math Word Problems. arXiv:2110.14168. C...

    2021 arXiv

  4. [2023]

    Kamoi, R.; Das, S

    Mistral 7B.CoRR, abs/2310.06825. Kamoi, R.; Das, S. S. S.; Lou, R.; Ahn, J. J.; Zhao, Y.; Lu, X.;Zhang,N.;Zhang,Y.;Zhang,R.H.;Vummanthala,S.R.; Dave, S.; Qin, S.; Cohan, A.; Yin, W.; and Zhang, R. 2024a. EvaluatingLLMsatDetectingErrorsinLLMResponses. In First Conference on Lan...

    arXiv

  5. [2024]

    CoRR, abs/2407.21783

    The Llama 3 Herd of Models. CoRR, abs/2407.21783. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J

    arXiv

  6. [2025]

    InAdvancesinNeuralInformation Processing Systems 38 (NeurIPS 2025)

    Inference-Time Reward Hacking in LargeLanguageModels. InAdvancesinNeuralInformation Processing Systems 38 (NeurIPS 2025). Spotlight. Khalifa,M.;Agarwal,R.;Logeswaran,L.;Kim,J.;Peng,H.; Lee,M.;Lee,H.;andWang,L.2026.ProcessRewardModels That Think.Transactions on Machine Learning...

    2025

  7. [2026]

    InMatusevych,Y.;Eryiğit,G.;andAletras,N.,eds., Proceedingsofthe19thConferenceoftheEuropeanChapter oftheAssociationforComputationalLinguistics(Volume5: Industry Track), 927–936

    Balanced Accuracy: The Right Metric for Evaluating LLM Judges - Explained through Youden’s J statistic. InMatusevych,Y.;Eryiğit,G.;andAletras,N.,eds., Proceedingsofthe19thConferenceoftheEuropeanChapter oftheAssociationforComputationalLinguistics(Volume5: Industry Track), 927–9...

    2026

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.