Pith.

Artificial Intelligence · arXiv:2609.35677 v1 · submitted 28 Sep 2026 · 26 references

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

Christian Moya, Elliott Thornley, Guang Lin

The paper says

“In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls.”

more of the abstract

We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves selective control: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 29 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The impossibility half is solid and will survive; the constructive half — audit correction raising correctness — is only proven where the verifier has no false negatives, and the acknowledged fix in A.1 is never operationalized. This line is wrong

3 major objections · 3 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 26 references say.

The paper says, at §2.2, Theorem 5.1(i), Abstract

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The central constructive statement — Theorem 5.1(i)'s dp_H/dt < 0 implies dp_G/dt > 0, and the abstract's claim that the correction 'raises that of correct responses' — rests on the identity p = p_G + p_H, which is exactly the no-false-negatives assumption c(x,y) ≤ R(x,y) stated in Section 2.2 (sentences b38–b39).

more of objection 1

Appendix A.1 shows that without this assumption dp_G/dt = ||g_R||² − dp_H/dt + d/dt Pr(c = 1, R = 0), where the added rate term is typically negative under verifier-optimizing policies and can have large magnitude. Thus dp_H/dt < 0 no longer implies dp_G/dt ≥ 0. The paper is transparent about this in A.1, but the assumption is not restated among Theorem 5.1's hypotheses (A1)–(A3), and the abstract asserts the constructive claim without the qualification. Because real verifiers do reject correct responses (timeouts, format strictness, ambiguous answers), the headline claim as worded is broader than what is proved. The fix is local: restate the abstract and theorem with the qualification explicit (for example, 'the share of correct responses among accepted responses'), or extend the theorem's selective-control condition to incorporate the extra term.

The paper says, at Appendix A.1, §5.1, Appendix C.4.3

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The extension to false negatives that A.1 proposes is acknowledged but not resolved. The required 'lower bound on d/dt Pr_θ(t)(c = 1, R = 0)' is never derived, estimated, or bounded, and the audit protocol cannot supply it: Section 5.1 defines an audit as revealing c(x,y) for an accepted response y, and Appendix C.4.3 states that rejected responses are never audited.

more of objection 2

The rejected-correct mass Pr(c = 1, R = 0) and its time derivative are therefore unobservable under the proposed protocol precisely in the regime where false negatives occur. I suggest either extending the audit protocol to a subsample of rejected responses so that A.1's bound becomes a monitored condition rather than an existence claim, or proving that no protocol auditing only accepted responses can certify the bound, which would justify a narrower statement of the guarantee. As written, A.1 identifies the gap but does not close it.

The paper says, at Appendix D.1, Appendix D.3

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

No experiment leaves the c ≤ R regime, so the empirical support for the constructive claim is confined to exactly the assumption on which the theory depends. In the Gaussian bandit (D.1), response groups are constructed with (R,c) ∈ {(1,1), (1,0), (0,0)}; in the language-model task (D.3), an output with a format error receives both R = 0 and c = 0 by definition, so correctness is defined so that no correct response is ever rejected.

more of objection 3

The experiments therefore cannot exhibit (or even measure) the failure mode — correctness falling while hacks are suppressed — that the theory identifies as possible once false negatives are allowed. I ask for an experiment in which a non-negligible fraction of correct responses are rejected (for example, a verifier that randomly rejects some correct final answers) with a report of p_G and p_H under PAC in that setting, or alternatively an explicit statement in Sections 6.1–6.2 that all reported correctness gains are within the no-false-negatives model. As it stands, the empirical section cannot detect the failure mode that the theory says it cannot rule out.

3 minor objections

The paper says, at Appendix C.1.3

Pith says: minor objection 1 of 3

The expression for the growth rate of J_R is inconsistent: Section 2.3 gives dJ_R/dt = ||g_R||², while Appendix C.1.3 writes ||g_R||²/2. Since the flow (1) is dθ/dt = g_R, the factor 1/2 is an error, and the two statements should agree.

The paper says, at §6.2, D.3.1

Pith says: minor objection 2 of 3

The main text says GRPO 'illustrates reward hacking in the sense of Proposition 3.1', but GRPO with sampled gradients and AdamW does not satisfy the proposition's exact-gradient-flow hypothesis. The caveat appears only in D.3.1; adding a parenthetical in Section 6.2 would keep the main text's claims aligned with the proposition's hypotheses.

show 1 more

The paper says, at Theorem 5.1(ii), D.1.3

Pith says: minor objection 3 of 3

Assumption (A3) and the bound D are global-in-time conditions, and Figure 5(b) constructs the ISS envelope from trajectory-estimated values of κ and D; the appendix correctly calls this a numerical illustration rather than a certification. A brief note on how one would check the uniform corrective-strength condition (A3) prospectively during training would strengthen the practical claims in the Remark after Theorem 5.1.

What would settle it

Re-run the §6.2/D.3 language-model experiment with a verifier that exhibits false negatives: define R'(x,y) = 1 iff the output is valid, the parsed answer's last two digits match the target, and the first digit equals 1, so correct answers whose first digit is 2 are rejected (c=1, R'=0). Keep the SFT checkpoints, PAC with ρ=1, 20 rounds, and five seeds identical. Along the calibration evaluations, record ˙p_G and ˙p_H at rounds 0, 5, 10, and 20, and estimate the liability term Δ = [Pr(c=1,R'=0)](t2→t1)/(elapsed rounds) using the experimenter's full labels; then check whether any evaluation interval has ˙p_H < 0 with ˙p_G < 0, and whether Δ < −(||g_R||² − ˙p_H) per the A.1 criterion.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 29 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 2 min 47 s · 42,542 tokens read, 21,814 written
  3. Editor wrote the verdict above 15 s · 43,470 tokens read, 1,903 written
  4. Referee major revision: 3 major objections, 3 minor 2 min 40 s · 43,096 tokens read, 20,934 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The thing to know: the negative half is genuinely solid. Propositions 4.1–4.3 — that the verifier-only training record cannot detect, identify, or selectively control accepted errors — are clean indistinguishability arguments. Prop 4.3 is a tight exchange: compatible correctness assignments swap the roles of correct responses and hacks, so no controller using only Lt can satisfy both. I checked the algebra by hand (the gR = p(1-p)[(1-q)sG + q sH - sN] identity follows from normalization; the log-odds decomposition is right) and it holds. This is real and it's new — it converts reward hacking from a qualitative phenomenon into checkable gradient conditions and honest information-theoretic limits.

The soft spot is load-bearing, and it's in the constructive half. Theorem 5.1's headline — suppress hacks and raise correctness — uses p = pG + pH, which requires no verifier false negatives (b38–b39). The authors know this; A.1 is upfront that allowing false negatives adds d/dt Pr(c=1,R=0), and the guarantee needs a lower bound on that rate. But that bound is never operationalized, and the audit protocol (C.4.3) audits only accepted responses, so the rejected-correct rate is unobservable in exactly the regime real verifiers live in — timeouts, format checks, ambiguous answers. So the selective-control result is proven precisely where hack suppression equals correctness gain. That's a real narrowing between the abstract's prose and the proof. The experiments don't help: D.3 constructs R so that c ≤ R holds by design. The negative half is untouched — the enlarged assignment class still contains the constructed pairs, so Prop 4.3 stands unmodified.

Minor reservations: the ISS constants κ and D are fit from the trajectory rather than certified, the repository is unverified, and the LM evidence is one task on one 0.5B model with five seeds. None of these are fatal.

Who it's for: anyone working on RLVR safety, verifier design, or reward-hacking mitigation. The impossibility result is the takeaway; the audit correction is a plausible remedy that needs harder guarantees under false negatives. It deserves a serious referee, with the main effort on making the A.1 fix real — either a bounded false-negative rate with an observable estimator, or an honest retreat to a weaker stated guarantee.

The strongest objection

The central constructive claim — Theorem 5.1(i)-(ii) and the abstract's 'raises that of correct responses' — derives ˙p_G = ||g_R||² − ˙p_H from p = p_G + p_H, which holds only under the no-false-negatives assumption c(x,y) ≤ R(x,y) (Section 2.2, b38-b39). Allow false negatives, and A.1's own algebra gives ˙p_G = ||g_R||² − ˙p_H + d/dt Pr(c=1,R=0). Verifier-optimizing policies shift probability from rejected to accepted responses, so the rejected-correct mass Pr(c=1,R=0) typically falls; the added term is usually negative and, being a probability's rate, can be arbitrarily large in magnitude. Thus ˙p_H < 0 no longer implies ˙p_G ≥ 0, and correctness can fall while hacks are suppressed. A.1 acknowledges this and says the guarantee requires a lower bound on that rate. What remains unresolved: (1) the bound is never operationalized, and (2) the paper's audit protocol audits only accepted responses — Section 5.1 defines an audit as revealing c(x,y) for an accepted response, and Appendix C.4.3 states rejected responses are never audited — so even with audits, the relevant rate is unobservable and the selective-control guarantee cannot be certified precisely in the regime where real verifiers (timeouts, format checks, ambiguous answers) reject correct responses. No experiment leaves the regime: D.3 constructs R so that c ≤ R holds by design. Proposition 4.3's impossibility is unaffected, since its two constructed assignments remain in C_R when the class is enlarged, so the negative half stands; the load-bearing fragility is in the positive half — the paper's solution route is proven only where hack suppression equals correctness gain.

I re-checked the algebra the reader verified. Theorem 5.1's Steps 2-4 (C.4.2): ˙z = b − λS/(pq(1−q)) follows from u = −λP⊥h, g_Rᵀu = 0, and ∇p_G = g_R − h; the ISS integration ˙q ≤ D/4 − λκq is correct. Proposition 3.1's mean-value argument and Proposition 4.3's group-exchange argument are sound, and the latter survives dropping c ≤ R because the constructed assignments remain compatible. The paper is honest about its main limitation: A.1 derives the false-negative equation and states the need for a lower bound on the rejected-correct rate. My stress-test finding is that the acknowledgment covers the equation but not the operational gap: under the paper's own audit protocol (accepted responses only; C.4.3 explicitly never audits rejected responses), Pr(c=1,R=0) and its rate are unobservable, so the audit route cannot even certify the condition the theory requires. Moreover, every experiment constructs c ≤ R by design (D.3), so no empirical evidence bears on the regime where the inference fails. This matches the reader's weakest-assumption pick: the no-false-negatives premise is exactly what converts hack reduction into correctness gain. I recommend no verdict change: the conditional mathematics is sound within its stated assumptions, the limitation is partially acknowledged, and the concern is addressable either by re-scoping the headline to the no-false-negative regime or by extending the audit protocol to rejected responses and demonstrating the A.1 bound empirically. The verdict stays CONDITIONAL, so verdict_should_be is UNCHANGED.

The referee's report

If the results hold as stated, the negative results are the more robust contribution: Propositions 4.1–4.3 are clean indistinguishability arguments whose constructed assignments remain in the enlarged class even when false negatives are allowed, so the impossibilities survive the relaxation of the c ≤ R assumption noted in Appendix A.1. The positive result shows that audits supply exactly the correctness information whose absence the negative results identify. I checked the appendix algebra: the identity g_R = p(1−p)[(1−q)s_G + qs_H − s_N] follows from the normalization conditions, the log-odds decomposition (3) is correct, the projection identities in Appendix C.4.1 and the ISS integration in Appendix C.4.2 are correct, and the proofs of Propositions 4.1–4.3 are sound. The code is released, the experimental protocol is detailed enough to reproduce, and Appendix A is unusually candid about scope.

Pith's reading of the paper (a machine's summary, not the authors')

This paper studies reinforcement learning with verifiable rewards (RLVR), the setting where a model is trained on rewards from an automated checker that can accept incorrect responses, and asks when those accepted errors grow and whether training observations can reveal or prevent the growth. Along verifier gradient flow the paper proves that the share of accepted errors grows exactly when a quantity combining hack bias and correctness-to-hack leakage is positive, and that verifier reward rises while correctness falls precisely at times when the correctness lost to a rising hacked share outweighs the correctness gained from rising acceptance. It then proves that the complete RLVR training record carries the same distribution under every correctness assignment compatible with the verifier, so no monitor using only that record can detect hacks, identify which accepted responses are wrong, or guarantee fewer accepted errors without sacrificing correct responses. The escape route is projected audit correction, which uses correctness labels from audits of some responses and, when its strength exceeds the verifier's drift toward errors, lowers the probability of accepted errors while raising that of correct responses, with an explicit bound on the hacked share. Contextual bandit and language-model experiments corroborate the analysis.

What carries the argument. The mechanism is a log-odds identity for the ratio of hacks to correct responses. With $z = \log(p_H/p_G)$, differentiating along the flow and using the normalization $g_R = p(1-p)[(1-q)\bar{s}_G + q\bar{s}_H - \bar{s}_N]$ decomposes the growth rate into $\dot{z} = p(1-p)[(\bar{s}_H-\bar{s}_G)^\top(\bar{s}_G-\bar{s}_N) + q\|\bar{s}_H-\bar{s}_G\|^2]$, separating correctness-to-hack leakage from hack bias. The second load-bearing object is the orthogonal projector $P_\perp = I - g_R g_R^\top/\|g_R\|^2$ applied to the audited-hack gradient; it removes the component along the reward gradient so that acceptance growth is preserved ($\dot{p} = \|g_R\|^2$ at each time) while every unit of hack suppression converts into an equal gain in correctness. Together the decomposition explains when hacking grows and the projection explains how audits turn hack reduction into correctness gain.

Core claim. The central claim is that safe control of RLVR requires information about correctness beyond the verifier. Along verifier gradient flow the probability of accepted errors grows if and only if $((\bar{s}_H-\bar{s}_G)^\top(\bar{s}_G-\bar{s}_N) + q\|\bar{s}_H-\bar{s}_G\|^2) > 0$, running through two mechanisms: hack bias, which always pushes toward errors, and correctness-to-hack leakage, which can push either way (Proposition 3.2); in parallel, verifier reward rises while correctness falls exactly when the correctness lost to a rising hacked share outweighs the correctness gained from rising acceptance (Proposition 3.1). Because the training record $L_t$ has the same distribution under any correctness assignment compatible with the verifier, no monitor using only the record can detect hacks (Proposition 4.1), identify which accepted responses are wrong (Proposition 4.2), or guarantee fewer accepted errors without sacrificing correct responses under every compatible assignment (Proposition 4.3). Audits break the deadlock: projected audit correction preserves the instantaneous acceptance growth rate $\|g_R\|^2$ while opposing the hacked share, and under full coverage and uniform corrective strength it yields selective control plus an ISS-type bound $q(t) \le e^{-\lambda\kappa t}q(0) + \frac{D}{4\lambda\kappa}(1-e^{-\lambda\kappa t})$ on the hacked share (Theorem 5.1). Contextual bandit and language-model experiments support the analysis.

The premise it rests on. The load-bearing premise is that the verifier has no false negatives — it accepts every correct response, $c(x,y) \le R(x,y)$ — because this makes accepted responses exactly the union of correct responses and hacks, so that any reduction in hacks at fixed acceptance must raise correctness, and it is precisely this premise that turns hack suppression into a correctness guarantee.

If the paper is right (Pith's inferences, not the authors')

If correct, RLVR training telemetry alone — however generously logged with prompts, responses, gradients, and policies — can never certify that rising verifier reward reflects progress on the intended task; safer RLVR must import correctness information from outside the verifier.

Accepted errors fall while correct responses rise only when the correction is strong enough to exceed the verifier's drift toward hacks, and the ISS bound shows that the hacked share decays exponentially at rate $\lambda\kappa$ with a floor $D/(4\lambda\kappa)$ that shrinks as corrective strength grows.

Partial audit coverage can retain most of the benefit: the language-model runs reach about 95 percent correctness while auditing only one quarter of accepted responses, provided the projected audit gradient stays positively aligned with the full hack gradient.

The projection is what prevents audit correction from sacrificing correct responses alongside hacks; raw audit correction initially suppresses both, exactly as the negative cross-term analysis predicts.

Detection and identification are impossible in general, but the obstruction vanishes the moment additional knowledge rules out the indistinguishable correctness assignments, so task-specific assumptions can restore guarantees even without identifying every hack.

My reading is that the no-false-negatives assumption is a narrower practical window than the theorems suggest: the authors acknowledge in Appendix A.1 that once the verifier rejects correct responses, correctness becomes $p_G = p - p_H + \Pr(c=1,R=0)$ and the selective-control guarantee needs a lower bound on $\frac{d}{dt}\Pr(c=1,R=0)$ that no training-time procedure seems able to precompute.

I read the impossibility results as statements about uniform guarantees over the full class $C_R$ of verifier-compatible assignments, not about any given task: the authors note that extra task knowledge can exclude the bad assignments, so the right takeaway is that a verifier alone never settles correctness, rather than that monitoring never works.

The guarantees are proved for exact gradient flow with a fixed verifier and prompt distribution, and the authors say so; extending to changing verifiers, curricula, or non-Euclidean optimizer geometry is where the theory would need to grow, though the language-model runs suggest the qualitative pattern survives sampled Adam updates.

The ISS bound suggests a design rule worth testing: measure the product $\lambda\kappa$ (corrective strength per unit hacked share) and the drift bound $D$ during training to set audit budgets, treating $D/(4\lambda\kappa)$ as the tolerated failure floor.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves selective control: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-29

The authors' main claim, as the classifier picked it out 0.67

“This correction achieves \emph{selective control}: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward.”

How the main claim is stated: asserted 0.84

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.82 · human participants 0.06 · controlled comparison 0.24 · randomized assignment 0.02 · states quantitative result 0.06 · compares with prior results 0.20 · mentions code or data availability 0.02

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. background 1.00 0.09 0.06 asserted 0.87
Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. method 0.67 0.92 0.05 asserted 0.98
We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. result 0.78 0.82 0.39 asserted 1.00
To address this limit, we construct a correction using additional feedback about correctness from audits. method 0.61 0.92 0.08 asserted 1.00
This correction achieves \emph{selective control}: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. result 0.73 0.90 0.48 asserted 0.84
Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness. result 0.90 0.95 0.38 asserted 1.00

What kind of paper the classifier reports this is: theoretical result 0.65 · evidence from observation or experiment 0.70

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-29: 1 request, 4,907 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. Reward hacking and selective control in contextual bandits. Panels (a)–(c) use a log
Figure 2 of the paper
Figure 2. Reward hacking and selective control in the language model, starting from an SFT
Figure 3 of the paper
Figure 3. Growth of hacking in the Gaussian contextual bandit. (a) Hack bias outweighs negative
Figure 4 of the paper
Figure 4. Control using only RLVR training observations in the Gaussian contextual bandit. (a)
Figure 5 of the paper
Figure 5. Selective control with correctness feedback in the Gaussian bandit. (a) Correctness
Figure 6 of the paper
Figure 6. Neural contextual bandit experiments. (a) Acceptance
Figure 7 of the paper
Figure 7. Initialization and audit ablations in the language model. (a,b) With SFT demonstration

Figures are the authors', reproduced from the paper.

References (26)
  1. [1] As in Appendix C.3.3, both assignments are compatible with the verifier, and they exchange the two accepted groups: pH,c2 =p G,c1, p G,c2 =p H,c1. Both groups have positive probability initially. By continuity, they remain positive on a sufficiently short initial interval. STE... on Pith
  2. [2] with ˙pG(t⋆) = pG(t2)−p G(t1) t2−t 1 <0. Substituting˙pG = (1−q) ˙p−p˙qand rearranging yields p(t⋆) ˙q(t⋆)>(1−q(t ⋆)) ˙p(t⋆), as required. The criterion distinguishes a growing hacked share from reward hacking in the sense of Skalse et al. (2022). A growing hacked share can co... on Pith
  3. [3] Xin-Qiang Cai, Wei Wang, Feng Liu, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Rein- forcement learning with verifiable yet noisy rewards under imperfect verifiers.arXiv preprint arXiv:2510.00915, on PithPith's reviewarXiv
  4. [5] Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bow- man, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering ... on PithPith's reviewarXiv
  5. [6] Chris Elliott, Einar Urdshals, David Quarel, and Daniel Murfet. Interpreting reinforcement learning agents with susceptibilities.arXiv preprint arXiv:2605.08007, abs/2605.08007, on PithPith's reviewarXiv
  6. [8] Lukas Helff, Quentin Delfosse, David Steinmann, Ruben H ¨arle, Hikaru Shindo, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting, and Felix Friedrich. Llms gaming verifiers: Rlvr can lead to reward hacking.arXiv preprint arXiv:2604.15149, on PithPith's reviewarXiv
  7. [9] Zachary Kenton, Lili Janzer, Rory Greig, Tian Huey Teh, Kirill Tyshchuk, Jonah Brown-Cohen, Harri Edwards, Senthooran Rajamanoharan, Noah Y . Siegel, Natasha Jaques, and Rohin Shah. Debate training reduces reward hacking in RLAIF.arXiv preprint arXiv:2608.17776, on PithPith's reviewarXiv
  8. [10] Muhammad Khalifa, Zohaib Khan, Omer Tafveez, Hao Peng, and Lu Wang. Countdown-code: A testbed for studying the emergence and generalization of reward hacking in RLVR.arXiv preprint arXiv:2603.07084, abs/2603.07084, on PithPith's reviewarXiv
  9. [11] Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jonathan Kutasov, Sara Price, Naia Bouscal, Samuel R. Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vladimir Mikulik, Ethan Perez, Alex Ro- drig... on PitharXiv
  10. [13] George Wang and Daniel Murfet. Patterning: The dual of interpretability.arXiv preprint arXiv:2601.13548, abs/2601.13548, on PitharXiv
  11. [14] Songtao Wang, Quang Hieu Pham, Fangcong Yin, Xinpeng Wang, Jocelyn Qiaochu Chen, Greg Durrett, and Xi Ye. Detecting and suppressing reward hacking with gradient fingerprints. In Third Conference on Language Modeling, 2026a. Xinpeng Wang, Nitish Joshi, Barbara Plank, Rico Angel... on PithPith's reviewarXiv
  12. [15] could help guide audit selection and correction. Adapting these methods to sustain selective control under limited audit and computation budgets remains an open engineering challenge. 14 Preprint. Under review. B RELATEDWORK We position our work within two lines of research on... on Pith
  13. [16] and when automated verifiers accept responses that violate task requirements (Helff et al., 2026; Zhong et al., 2025; Mahmoud et al., 2026). Training can amplify such reward hacking (Khalifa et al., 2026), and learned hacking can generalize to reward tampering or broader misal... on Pith
  14. [17] or by favoring flatter optima, a property that theory links to the accuracy of rewards under additional assumptions (Ackermann et al., 2026). Other methods im- prove the feedback used for optimization by combining reward models (Coste et al., 2024), adding constraints (Helff e... on Pith
  15. [18] or measure how much reasoning it needs to pass the verifier (Wang et al., 2026b). Detection can then guide correc- tion: Grift groups responses by their gradients, labels each group by inspecting examples, and filters responses for fine-tuning (Wang et al., 2026a), while IR 3 ... on Pith
  16. [21] The penalty therefore bounds depar- ture from the reference along the exact flow. 23 Preprint. Under review. STEP2: BOUND CHANGES IN HACK PROBABILITY. Because the prompt distribution is fixed,K also equals the KL divergence between the joint distributions of prompts and respon... on Pith
  17. [22] Thus, every offset equalslog(1/48)andp G(0) =p H(0) = pN(0) = 1/3. We reuse features and initialization across methods within each seed. AUDITS AND EXACT GRADIENTS. A fixed audit setA aud reveals correctness labels for selected accepted responses. Its audited hacks areH A =A a... on Pith
  18. [23] Parameter Value Accepted-group feature means(4,−1,0,0),(4,1,0,0) Rejected-group second-coordinate mean−0.5 Initial hacked shareq(0) 1/2 Initial acceptancep(0) 2/3 Initial parameters; warmupθ(0) = 0; none Independent feature seeds10(seeds0,...,9) Audit correction gainλ6 Gradien... on Pith
  19. [24] The projection ablation evaluates corrections at the initial policy without advancing it. We estimate gR from independent drawsx∼Dandy∼π θ(·|x)by averagingR(x,y)∇ θ logπθ(y|x). Only the projector uses this estimate. The verifier directiong R and the full hack gradient∇ θpH rem... on Pith
  20. [25] All three rules share the same training recordL t, so verifier observations cannot determine which cor- rectness trajectory applies. This illustrates the limits of detection and identification in Proposi- tions 4.1 and 4.2. Becausec 1 andc 2 exchange correct responses and hack... on Pith
show 6 more
  1. [26] Square markers show correctness under c=R, which equals acceptance. (d) Final hack probabilityp H(T)versus the initial audited fraction pHA(0)/pH(0)for fixed audit sets. Greater coverage reduces the probability of hacks. (e) Deviation |˙p−∥gR∥2|versus the number of samples use... on Pith
  2. [2022] Jane Pan, He He, Samuel R. Bowman, and Shi Feng. Spontaneous reward hacking in iterative self-refinement.arXiv preprint arXiv:2407.04549, on PithPith's reviewarXiv
  3. [2023] Etienne Gauthier, Francis R. Bach, and Michael I. Jordan. Explaining and preventing alignment collapse in iterative RLHF.arXiv preprint arXiv:2605.04266, on PithPith's reviewarXiv
  4. [2024] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F... on PithPith's reviewarXiv
  5. [2025] Mohammad Beigi, Ming Jin, Junshan Zhang, Jiaxin Zhang, Qifan Wang, and Lifu Huang. IR 3: Contrastive inverse reinforcement learning for interpretable detection and mitigation of reward hacking.arXiv preprint arXiv:2602.19416, on PitharXiv
  6. [2026] Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y . Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation.arXiv preprint arXiv:2503.11926, on PithPith's reviewarXiv
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment