Machine Learning · arXiv:2610.35716 v1 · submitted 26 Oct 2026 · 28 references
Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO
The paper says
“Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout.”
more of the abstract
We study how to spend a limited strong-verifier budget in GRPO, separating which weakly passing rollouts to audit from what reward a caught hack receives. At a fixed policy state with centered advantages, the expected update depends on a design only through each rollout's marginal audit probability and correction; reflecting a caught hack's reward about the failure reward negates the advantages of an all-hack group; and auditing whole groups with the fewest weak passes first (low-pass budgeted audits, LB) minimizes the advantage mass left on hacks among pass-count designs with a fixed correction when a label's value falls with the pass count. We tested this analysis in actual LLM training: from each run's starting state, the sign of this mass on a frozen panel predicted whether hacking would be suppressed in 33 of 39 runs, including 13 of 16 preregistered ones. In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels. We also find that the randomness of the update affects training: in Countdown-Code, two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, bring hacking down too but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB at the same budget. A full audit, which the group-level correction reproduces in expectation, stays within about two points of LB in three seeds, while LB uses 28% of its labels, and allocating the same correction by pass count recovers most of the loss (one seed). Training outcomes thus depend not only on the expected update but also on the randomness of its estimate.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. LB — audit the lowest-pass groups first and pay caught hacks 2b-a — is a genuinely new recipe that suppresses reward hacking in two real environments, with honestly conditional theory; the main soft spot is a monotonicity condition proven only in special models and only indirectly checked where the experiments actually operate. This line is wrong
2 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 28 references say.
The paper says, at §3.3 (Theorem 1) and §B
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The optimality claim of Theorem 1 is conditional on d_k being non-negative and non-increasing in k. Non-negativity is established generally (Proposition 5), but monotonicity is proved only in Proposition 2 (pure groups) and Proposition 6 (binomial hack rate with centered advantages).
more of objection 1
The training batches in Sections 5 and 6 mix hacks, correct responses, well-formed failures, and malformed responses with reward 0, and the text itself notes in §3.3 that with malformed responses a pass's advantage need not fall with k. The frozen-panel check in §B verifies the conclusion of Theorem 1 (exact optimality in 123 of 144 cases, within 4.2e-4 in the other 21) rather than the hypothesis; it does not report d_k by pass count, give an estimate of gamma for the 2B*gamma bound, or classify the 21 suboptimal cases as boundary-allocation differences versus genuine pass-count order reversals. I ask the authors to add a direct estimate of d_k (or of gamma) on frozen panels from both environments, or to state explicitly that in the mixed-group regime the monotonicity condition is an unvalidated assumption rather than an established property. This is a strengthening request, not a claim that Theorem 1 is wrong: the theorem is valid under its stated hypotheses.
This review says the paper's mathematics is wrong. Pith did not check that against the paper's source.
The paper says, at §5.4 and §I
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The sign-prediction claim is central to the abstract and conclusion, but the headline 33-of-39 count combines 16 preregistered runs with 23 runs whose designs were scored after the outcomes were known. The paper is transparent about this split and reports 13 of 16 for the preregistered portion, but the confirmatory and exploratory parts should be separated more sharply in the …
more of objection 2
… main text and abstract, with the 13-of-16 result reported with a binomial test and the 33-of-39 figure presented as exploratory evidence rather than as a single confirmatory predictive accuracy.
5 minor objections
The paper says, at §3.3, before Theorem 1
Pith says: minor objection 1 of 5
The sentence 'an uncaught hack's advantage is largest when few siblings pass, so a label spent on a low-pass group buys the most correction' is stated unconditionally; it should be qualified by the d_k monotonicity condition that Theorem 1 requires, since the text immediately afterward notes that the condition can fail for malformed responses.
The paper says, at §5.3
Pith says: minor objection 2 of 5
The 3.1 multiplier used in the scaled-LB ablation is derived from the frozen-panel ratio; please state explicitly in the main text whether this value was fixed before the training run or selected from the frozen panel, and that this arm is an exploratory fit rather than a test of a prespecified design.
show 3 more
The paper says, at §2
Pith says: minor objection 3 of 5
The phrases 'Fullaudits every weak pass' and 'Noneaudits nothing' appear to be missing spaces and should read 'Full audits every weak pass' and 'None audits nothing'.
The paper says, at §8
Pith says: minor objection 4 of 5
The sentence reporting 'p=0.055 in training and p=0.040 on held-out problems at the seed level' could be misread as audit rates; it should say these are seed-level p-values from the t-tests.
The paper says, at §6
Pith says: minor objection 5 of 5
The allocated-by-pass-count correction is a one-seed result, and the conclusion that pass-count allocation 'recovers most of the gap' should be flagged as a single-seed pilot in the main text, not only in the appendix and limitations.
What would settle it
Directly estimate the Theorem 1 hypothesis on the 72 existing frozen panels: compute d̂_k = (1/n_k) Σ over k-pass groups of (Φ^0 − Φ^1)/k with paired bootstrap intervals, test non-increasingness in k, and classify each of the 21 suboptimal panel-objective cases as (i) boundary-fraction-only, meaning the same low-pass ordering with a different randomization of the last partially audited group, or (ii) a genuine ordering reversal, meaning a higher-k group with a larger per-label value.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should know two things. First, this paper has a genuinely new allocation rule — audit whole groups with the fewest weak passes first and pay a caught hack 2b-a — plus a conditional optimality theorem for it, and the rule works in real training: in both LeetCode and Countdown-Code, LB drives hacking to near zero while random audits at the same budget and same penalty fail. Second, the theoretical guarantee is explicitly conditional on a monotonicity condition that is proven in pure-group and binomial models but only checked indirectly in the mixed-group regime the experiments actually exercise. That is the paper's real soft spot, and it is smaller than it sounds.
The novelty is real. Prior budgeted-audit work targets high-pass groups for variance reduction in unbiased estimation; this paper targets low-pass groups because an uncaught hack's advantage is largest there. The reflected penalty 2b-a is a clean observation (Proposition 1) and it explains a concrete empirical fact: paying caught hacks the strong-verdict reward b is no better than random ordering, while the reflected penalty removes hacking with half the labels.
The paper does several things well. The theory is carefully scoped — Lemma 1, Proposition 1 and Theorem 1 have explicit hypotheses and the appendix proofs look right. The empirics are a cut above typical LLM training papers: three seeds for the main comparisons, shared prompts, paired bootstrap intervals, seed-level t-tests, and 16 preregistered runs. The sign-prediction counts (33 of 39, 13 of 16 preregistered) are reported honestly, including the retrospective runs and the cases where prediction failed. The Countdown-Code finding that unbiased corrections suppress hacking but lose 6–12 accuracy points is a real result about estimator variance, not just direction.
The soft spots, in proportion: the load-bearing hypothesis of Theorem 1, that the per-label value dk is non-increasing in k, is proven in two special models but not in the mixed-group setting with malformed responses, where the paper itself says a pass's advantage need not fall with k. The frozen-panel check in Section B tests the theorem's conclusion, not the condition: LB attains the optimum in 123 of 144 cases and stays within 4.2e-4 in the rest, but that does not establish monotonicity, and the 21 suboptimal cases are not classified. The theorem is honestly conditional — the abstract states the condition — so this is a gap in coverage, not an internal contradiction, and the empirical suppression results do not depend on exact optimality. No code or data is released, several supporting arms are single-seed, and the 3.1 scaling factor is fit to frozen panels; all of these are acknowledged in the paper.
This paper deserves a serious referee. The referee should ask for a direct empirical check of monotonicity (estimated dk by k with uncertainty on frozen panels) and ideally code release. I would bring it to a reading group.
The strongest objection
The load-bearing step of the central argument is the hypothesis of Theorem 1 (Section 3.3): the per-label value d_k = (μ0_k − μ1_k)/k must be non-negative and non-increasing in k for the fractional-knapsack argument to make LB optimal among whole-group pass-count designs with a fixed correction. Non-negativity is proven in general (Proposition 5, since the correction c ≤ a in both environments). Monotonicity is proven only in special models: Proposition 2 (pure groups, non-passing responses at b, std normalization with ε = 0) and Proposition 6 (centered advantages, binomial hack rate). Both environments train with group-std normalization, and their batches mix hacks, correct responses, well-formed failures and malformed responses (reward 0); the paper states in Section 3.3 that with malformed responses a pass's advantage need not fall with k. The frozen-panel check in Section B addresses the conclusion rather than the condition: LB attains the optimum over Theorem 1's design class in 123 of 144 panel-objective cases and is within 4.2e-4 in the other 21, but the authors do not classify those 21 as boundary-allocation differences or genuine pass-count order reversals, nor do they report an estimated γ for the approximate-case bound 2Bγ of Theorem 1. In the mixed-group regime the experiments actually exercise, the needed monotonicity is therefore supported neither by proof nor by a direct test, and 21 of 144 cases show LB is not exactly optimal. This is not an internal inconsistency — the theorem and abstract state the condition explicitly, and the empirical suppression results do not depend on exact optimality — but the headline 'minimizes' claim is a conditional guarantee whose key hypothesis has only indirect empirical coverage.
Read the paper and its appendices in full, then checked the derivations the central claim touches. Lemma 1 is exact for centered advantages; Proposition 1's reflection calculation (λ = (b−c)/(a−b), λ = 1 at c = 2b−a) checks out under both advantage forms; Proposition 5's monotonicity of hack mass in the hack reward holds for ε ≥ 0, giving d_k ≥ 0 generally; Theorem 1's summation-by-parts knapsack argument is sound given its monotonicity hypothesis. The one place where the claim's reach exceeds its proof is that hypothesis for mixed groups under std normalization. The reader's weakest_assumption identifies the same spot and I agree with it: the condition is proven for pure and binomial compositions, checked in Section B only through the conclusion (123 of 144 panel-objective cases optimal, 21 within 4.2e-4), and the 21 violations are unclassified, so a direct test of d̂_k monotonicity is what would settle the matter. This concern is load-bearing for the 'minimizes' wording but bounded in consequence: the empirical core — LB suppressing hacking in both environments while random audits do not, with paired seeds and seed-level t-tests, held-out generalization, the reflected-penalty control against c = b, and the 0.1/0.2 budget discrimination — does not depend on exact knapsack optimality. Independent support worth crediting: preregistered predictions with failures honestly reported, the full-audit and pass-count-allocation controls localizing the IPW accuracy gap to estimator randomness, and the explicit statement of the condition. No new ground was found beyond the reader's analysis, so the verdict stays CONDITIONAL and unchanged, with the condition being direct verification of d_k monotonicity (or a tight bound on its violations) in the mixed-group regime.
The referee's report
If the results hold, the paper makes a substantial contribution: a clean decomposition of audit selection from correction, an explicit conditional optimality theorem for LB, and a rare preregistered empirical comparison in two reward-hacking environments. The strengths are real and should be credited. The proofs in Appendix A have explicit hypotheses and I found no load-bearing derivation error. The experiments use shared prompts, paired bootstrap intervals, seed-level t-tests, and a preregistration log, and the paper is unusually honest about single-seed ablations, failed predictions, fitted components (the 3.1 scaling factor, the variance-reduction weights), and domains where the frozen-push analysis stops predicting. The LB result is practically meaningful: low-pass whole-group audits with the reflected penalty suppress hacking at a fraction of the labels used by full audits or random audits, and the full-audit controls support the randomness-based explanation for the unbiased corrections' accuracy loss.
Pith's reading of the paper (a machine's summary, not the authors')
This paper claims that a small budget of stronger checks can suppress reward hacking in group-relative policy optimization (GRPO), provided the checks are aimed where an uncaught hack is reinforced most and a caught hack is pushed away rather than merely zeroed. The central recipe has two parts: audit whole groups in increasing order of their number of weak passes (low-pass budgeted audits, LB), and give a caught hack the reflected reward $c = 2b - a$, the weak-pass reward $a$ mirrored about the failure reward $b$. At a frozen policy state, the expected update depends on a design only through each rollout's marginal audit probability and correction, the reflected reward exactly negates the advantages of an all-hack group, and LB minimizes the expected advantage mass left on hacks among pass-count designs whenever a label's value does not increase with the pass count (Theorem 1). In LeetCode and Countdown-Code training runs, LB drove hacking to near zero within about ten steps, while random audits at the same budget and penalty failed and used more labels. The sign of the frozen-state hack mass predicted which runs would suppress hacking in 33 of 39 cases, including 13 of 16 preregistered ones, making the analysis usable as a cheap pre-training screen.
What carries the argument. Three pieces carry the argument. The reflected penalty $c = 2b - a$, a caught hack's reward obtained by mirroring the weak-pass reward $a$ about the failure reward $b$, is the only fixed correction that exactly negates the advantages of a fully audited all-hack group under both centered and std-normalized advantages (Proposition 1). The value per label $d_k = (\mu_{0k} - \mu_{1k})/k$ measures the expected reduction in hack advantage mass per strong-verifier label spent on a group with $k$ weak passes, and the design problem becomes a fractional knapsack in which LB spends labels in decreasing order of $d_k$, which is the same as auditing whole groups with the fewest weak passes first whenever $d_k$ falls with $k$ (Theorem 1). The objective being minimized is the expected hack mass $J$, the sum over groups of the mean normalized advantage of hacks within each group, and the paper predicts training outcomes from the sign of $J$ evaluated on a frozen panel at the starting checkpoint.
Core claim. The central claim is that selection and correction jointly, not the audit budget alone, determine how a GRPO update responds to hacks. With centered advantages, the expected raw policy gradient depends on an audit design only through each hack's marginal audit probability multiplied by its correction, so coupling audits changes only the noise of an update (Lemma 1). Giving a caught hack the reflected reward $2b - a$ makes post-audit rewards satisfy $r'_i = 2b - r_i$ and thereby exactly negates the advantages of a group whose weak passes are all hacks, whereas the strong-verdict reward $b$ only zeroes them (Proposition 1). Because an uncaught hack's advantage grows as its group has fewer weak passes, the expected hack advantage mass falls by $d_k = (\mu_{0k} - \mu_{1k})/k$ per label spent on a $k$-pass group, which turns the audit design problem into a fractional knapsack; auditing whole groups with the fewest weak passes first maximizes the mass removed among pass-count designs with a fixed correction whenever $d_k$ is non-increasing in $k$ (Theorem 1). Empirically, this design suppressed hacking to near zero in two code-training environments while random audit orders did not, the sign of the frozen hack mass predicted suppression in 33 of 39 runs, and unbiased corrections that match a full audit in expectation still lost 6-12 points of held-out accuracy to LB in Countdown-Code, showing that the randomness of the update estimate matters as well.
The premise it rests on. The load-bearing premise is that the value per label $d_k$, the expected reduction in hack advantage mass per strong-verifier label spent on a group with $k$ weak passes, never increases with $k$, a condition the paper proves only for pure all-hack or all-correct groups (Proposition 2), verifies on frozen panels where LB reached the optimum in 123 of 144 cases, and needs for Theorem 1's guarantee that auditing the fewest-pass groups first is optimal among whole-group fixed-correction designs.
If the paper is right (Pith's inferences, not the authors')
A fixed budget near a third of weak passes suffices to drive hacking to near zero within about ten GRPO steps in LeetCode and Countdown-Code, while random whole-group or per-item audits at the same penalty and budget leave late hacking at 36-56% and use roughly twice the labels in LeetCode.
The selection rule and the correction value matter as much as the budget: with the strong-verdict reward $c = b$, targeting the lowest-pass groups is no better than targeting random groups, whereas with $c = 2b - a$ the same groups remove hacking with half the labels.
Unbiased corrections that match a full audit in expectation, per-item reward-level IPW and the group-level advantage correction, suppress hacking in Countdown-Code but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB, while a full audit stays within about two points, so a low-noise biased update can beat an unbiased noisier one at a fixed budget.
The sign of the frozen hack mass $J$ at the starting checkpoint predicts whether hacking will be suppressed, holding in 33 of 39 runs (13 of 16 preregistered), although its magnitude does not rank designs, as the paper's failed online rule showed.
Setting the budget as a fraction of weak passes makes label spending fall automatically as hacking is suppressed, so the strongest designs are also the cheapest: LB used 28% of a full audit's labels in Countdown-Code and about half the labels of random audits.
My inference: the value-per-label knapsack view suggests the ordering principle should transfer to finer-grained decisions, such as pass-count-dependent audit probabilities tuned to a frozen-panel estimate of $d_k$, but the paper's own preregistered online rule that improved $J$ trained worse than LB, so such an extension would have to control estimator noise before claiming a win.
My inference: the reflection identity is testable outside code domains, because any two-valued cheap verifier (format check, rubric, model-based monitor) admits a flip penalty $2b - a$, and the recipe should generalize wherever the failure reward is approximately homogeneous; where it is not, Proposition 4's group-adaptive reflection is the natural repair.
My inference: the 33-of-39 sign prediction is cheap to reuse as a screening rule, namely to score a proposed audit design's hack mass on a frozen panel at the starting checkpoint and read its sign before spending training compute, and it should be benchmarked against a simpler baseline such as predicted hack concentration in low-pass groups, which the paper notes it has not been.
My inference: because Lemma 1 fails under group std normalization and Theorem 1 is proven only for whole-group pass-count designs with a fixed correction under a reliable strong verifier, the sharpest open question is an extension of the ordering argument to normalized advantages or heterogeneous failure rewards; until then LB's guarantees are conditional on exactly that design class.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout. We study how to spend a limited strong-verifier budget in GRPO, separating which weakly passing rollouts to audit from what reward a caught hack receives. At a fixed policy state with centered advantages, the expected update depends on a design only through each rollout's marginal audit probability and correction; reflecting a caught hack's reward about the failure reward negates the advantages of an all-hack group; and auditing whole groups with the fewest weak passes first (low-pass budgeted audits, LB) minimizes the advantage mass left on hacks among pass-count designs with a fixed correction when a label's value falls with the pass count. We tested this analysis in actual LLM training: from each run's starting state, the sign of this mass on a frozen panel predicted whether hacking would be suppressed in 33 of 39 runs, including 13 of 16 preregistered ones. In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels. We also find that the randomness of the update affects training: in Countdown-Code, two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, bring hacking down too but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB at the same budget. A full audit, which the group-level correction reproduces in expectation, stays within about two points of LB in three seeds, while LB uses 28% of its labels, and allocating the same correction by pass count recovers most of the loss (one seed). Training outcomes thus depend not only on the expected update but also on the randomness of its estimate.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-27
The authors' main claim, as the classifier picked it out 0.47
“In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.65 · human participants 0.04 · controlled comparison 0.93 · randomized assignment 0.19 · states quantitative result 0.98 · compares with prior results 0.95 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout. | background 0.99 | 0.08 | 0.03 | asserted 1.00 |
| We study how to spend a limited strong-verifier budget in GRPO, separating which weakly passing rollouts to audit from what reward a caught hack receives. | contribution 1.00 | 0.82 | 0.09 | asserted 0.75 |
| At a fixed policy state with centered advantages, the expected update depends on a design only through each rollout's marginal audit probability and correction; reflecting a caught hack's reward about the failure reward negates the advantages of an all-hack group; and auditing whole groups with the fewest weak passes first (low-pass budgeted audits, LB) minimizes the advantage mass left on hacks among pass-count designs with a fixed correction when a label's value falls with the pass count. | contribution 0.59 | 0.84 | 0.36 | asserted 1.00 |
| We tested this analysis in actual LLM training: from each run's starting state, the sign of this mass on a frozen panel predicted whether hacking would be suppressed in 33 of 39 runs, including 13 of 16 preregistered ones. | result 0.99 | 0.96 | 0.21 | asserted 1.00 |
| In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels. | result 1.00 | 0.92 | 0.31 | asserted 1.00 |
| We also find that the randomness of the update affects training: in Countdown-Code, two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, bring hacking down too but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB at the same budget. | result 1.00 | 0.96 | 0.10 | asserted 1.00 |
| A full audit, which the group-level correction reproduces in expectation, stays within about two points of LB in three seeds, while LB uses 28% of its labels, and allocating the same correction by pass count recovers most of the loss (one seed). | result 1.00 | 0.94 | 0.09 | asserted 0.99 |
| Training outcomes thus depend not only on the expected update but also on the randomness of its estimate. | interpretation 0.98 | 0.86 | 0.28 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.69 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 6,739 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (28)
- [1] Anonymous. HackFree-GRPO: Detecting and mitigating reward hacking in RLVR-trained code agents. Submitted to ICLR 2027.https://openreview.net/forum?id=IplYh5Agup, on Pith
- [3] Then|E[Φ𝑝]−Φ 𝑤|≤2𝐶 𝐾,ℎ(1−(1−𝑝) ℎ)≤2𝐶 𝐾,ℎℎ𝑝, so the expected hack mass tends to its unaudited value as𝑝→0. Proof. Since the normalized advantages sum to zero,Φ= 1𝐾 ˝𝑖𝐴𝑖(1[𝑖∈𝐻]−ℎ/𝐾) , and by the Cauchy– Schwarz inequality|Φ|≤ 1𝐾√𝐾−1 √︁ℎ(𝐾−ℎ)/𝐾=𝐶 𝐾,ℎ, using˝𝑖𝐴2𝑖≤𝐾−1 . If no hack ... on Pith
- [4] arXiv:2605.02909. 13 Omar El Mansouri, Fathinah Asma Izzati, Mohamed El Amine Seddik, and Salem Lahlou. Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients, on PithPith's reviewarXiv
- [5] arXiv:2510.18924. Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization, on PithPith's reviewarXiv
- [8] arXiv:2609.26866. Daniel G. Horvitz and Donovan J. Thompson. A generalization of sampling without replacement from a finite universe.Journal of the American Statistical Association, 47(260):663–685, on PithPith's reviewarXiv
- [9] arXiv:2603.07084. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-Like Training: A Critical Perspective, on PithPith's reviewarXiv
- [10] arXiv:2503.20783. Christian Moya, Elliott Thornley, and Guang Lin. Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control, on PithPith's reviewarXiv
- [11] arXiv:2609.35677. Pixel Nomand, Elena Voss, Marcus Hale, and Sofia Reyes. PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR, on PithPith's reviewarXiv
- [12] arXiv:2608.11368. Benjamin Pikus, Pratyush Ranjan Tiwari, and Burton Ye. Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets, on PithPith's reviewarXiv
- [13] arXiv:2508.14094. Qwen Team. Qwen2.5 Technical Report, on PitharXiv
- [16] arXiv:2402.03300. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A Flexible and Efficient RLHF Framework, on PithPith's reviewarXiv
- [17] arXiv:2409.19256. Wan Tian, Zhongyi Li, Xiang Xu, Minhao Zou, Yijie Peng, and Fuzhen Zhuang. STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking, on PithPith's reviewarXiv
- [18] arXiv:2609.36900. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer reinforcement learning. https://github.com/huggingface/trl, on PithPith's reviewarXiv
- [19] arXiv:2605.25864. You Wang, Michael Pradel, and Zhongxin Liu. Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study, on PithPith's reviewarXiv
- [20] arXiv:2503.15223. Jake Ward, Shawn Hu, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, and Alex Cloud. Gradient Routing Localizes and Removes Unintended Behaviors in RL, on PithPith's reviewarXiv
- [21] arXiv:2610.17062. Aria Wong, Joshua Engels, and Neel Nanda. Designing effective monitor-based interventions for mitigating reward hacking during RL. InICML 2026 Workshop on Mechanistic Interpretability, 2026.https: //openreview.net/forum?id=uFTnN6fUgW. Esther Xin. Are Verifier... on PithPith's reviewarXiv
- [22] arXiv:2609.06386. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang,... on PithPith's reviewarXiv
- [23] arXiv:2505.09388. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze C... on PithPith's reviewarXiv
- [24] arXiv:2503.14476. Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ Propensity of Exploiting Test Cases, on PithPith's reviewarXiv
- [25] arXiv:2510.20270. A Proofs Throughout, the batch (prompts, responses, weak rewards, classes and response lengths) is fixed and the audit design is the only source of randomness, except in Theorem 1 and Proposition 2, where compositions are random given pass counts. All random ... on PitharXiv
show 8 more
- [27] keeps the weak reward and adds a correction along an inverse-propensity estimate of the hack-probability gradient, projected to be orthogonal to the reward gradient. The original Euclidean rule collapsed the policy within one step under full-parameter training, so we use an ad... on Pith
- [28] keeps the weak reward, so each GRPO step still reinforces hacks, and the projected correction is locally orthogonal to the weak-reward gradient in the chosen metric. With per-item audits at the same rate, hacking rose to 64.80% in the late window, and the unprojected correctio... on Pith
- [2017] arXiv:1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, on PithPith's reviewarXiv
- [2022] arXiv:2210.10760. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI Control: Improving Safety Despite Intentional Subversion, on PithPith's reviewarXiv
- [2023] arXiv:2312.06942. Shivam Gupta. Marginally Correct Tool Caches Can Reverse Group-Normalized Policy Updates, on PithPith's reviewarXiv
- [2024] arXiv:2412.15115. Nazmus Sakib, Sadif Ahmed, and Nafiu Rahman. Audit the group, not the patch: full-suite rewards for coding agents at a fraction of the cost. Blog post, 23 October on PithPith's reviewarXiv
- [2025] arXiv:2503.11926. JunweiDeng,AlexBie,andNataliaPonomareva. SimHack: Predictingtheonsetofdetectablerewardhacking before RL training, on PithPith's reviewarXiv
- [2026] arXiv:2610.28707. Kazuki Egashira, Mark Vero, Jasper Dekoninck, Florian E. Dorner, Robin Staab, and Martin Vechev. Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR, on PithPith's reviewarXiv
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment