REVIEW 3 major objections 7 minor 45 references
Natural false positives in code RLVR rewards pay for real bugs, yet at small scale the held-out damage stays under a preregistered 1.5-point margin and is predicted by a cheap pre-training audit.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 07:35 UTC pith:XXVYUGCE
load-bearing objection Preregistered causal contrast of natural suite FPs in RLVR: non-inferiority holds, static audit predicts leak mass, and the measurement layer is the real contribution. the 3 major comments →
When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
At 1–1.5B parameters and 400 GRPO steps on MBPP, training against the original suite’s natural (persistent, per-task, asymmetric) false positives is non-inferior to a hardened extra-tests-only reward under a preregistered 1.5-pt held-out margin (gap 0.20 pt, one-sided 95% upper bound 0.75 pt across estimators and three model families). Rewarded false-positive mass tracks a pre-training static leakiness audit (Spearman ~0.80) and concentrates on leak-stratum tasks (+43.8 pt). Roughly half of the leaked reward pays for verified wrong code, and the mechanism is consistent with selection of pre-existing error modes rather than learned exploitation.
What carries the argument
The preregistered two-arm causal contrast: identical model, tasks, seeds, and compute, differing only in which suite pays the reward (original MBPP versus MBPP+ extra tests only), paired with a static leakiness audit that scores each task from k=16 base-model samples before training. This isolates what natural verifier false positives do and predicts where reward will leak.
Load-bearing premise
The hardened extra-test suite is treated as a good enough oracle for defining both false positives and held-out ability, even though those extra tests sometimes reject correct code and scoring uses extra tests alone rather than the official combined metric.
What would settle it
A longer-horizon or larger-scale preregistered run in which the leaky arm’s rewarded false-positive share or FP-only group share rises while the held-out gap exceeds the 1.5-point margin, or a re-audit showing that most of the residual “verified-wrong” mass is plus-suite error rather than model error.
If this is right
- A cheap static audit of base-model samples against both suites locates where rewarded false-positive mass will sit before any RLVR run.
- Hardening the reward removes measurement inflation (about 8 points of train reward) without buying much held-out capability at this scale.
- Train-task reward is an accounting of what was paid, not of what was learned; reward QA must be computed on tasks the model has not trained on.
- About half of rewarded false positives are real bugs, so suite failures need a taxonomy before they are treated as model failures.
- The same wrong outputs appear across three families and in base models, so a suite hole aligned with a shared prior taxes every model trained on it.
Where Pith is reading between the lines
- If selection dominates at small scale, active reward hacking may stay latent until capability or horizon grows enough for policies to climb the channel rather than merely sample it.
- The exploratory finding that frontier judges read their own false positives only weakly suggests oversight of stronger generators may need heterogeneous readers and mechanical anchors rather than a single stronger judge.
- Static leakiness could serve as a release gate for any new unit-test reward suite, analogous to a coverage or thoroughness metric before training.
- The flat FP incidence through 800 steps leaves open whether scale converts selection into exploitation; that is the natural next causal contrast on the same instrument.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper runs a preregistered two-arm GRPO contrast on MBPP that isolates natural (persistent, per-task, asymmetric) verifier false positives: identical models, tasks, seeds, and compute, with reward paid by original MBPP tests (leaky) versus MBPP+ extra tests only (hardened). Across three 1–1.5B families, the confirmatory results are that held-out extra-test pass rate is non-inferior under a frozen 1.5-pt margin (gap 0.20 pt; one-sided 95% upper bound 0.75 pt under six estimators), that pre-training static leakiness tracks train-time rewarded-FP mass (Spearman ≈0.80), and that the leak-stratum FP share is +43.8 pt above clean tasks. Exploratory work reports that roughly half of rewarded FPs are human-adjudicated genuinely wrong code, that mechanism evidence favors selection of pre-existing base-model error modes over learned exploitation within the measured horizon, and a carefully fenced meta-audit of two frontier judges on their own FPs.
Significance. If the confirmatory results hold within the stated box, the paper supplies the missing causal measurement that suite audits and noise-robustness theory have both called for: what natural, non-resampled verifier FPs do under actual RLVR. Strengths that raise the evidential bar include a full preregistration chain with published failed gates, seed-as-unit inference, multi-estimator non-inferiority, two replication families frozen before their data, classifier-free FP mass quantities invariant to later audit revisions, a self-penalizing verified-wrong trail, an 800-step continuation, and a released audit/adjudication/reproducibility package. The practical instruments—a cheap static leakiness audit and a base-policy control—are immediately usable for reward QA. The contribution is measurement, not a new algorithm; that is appropriate and valuable for RLVR reliability work.
major comments (3)
- [§4.1 Average held-out effect; Abstract; §6] §4.1 and the abstract/discussion takeaway that hardening “buys little capability”: non-inferiority is correctly executed (margin frozen; six estimators; three families), but with 5 seeds the design cannot reliably detect true gaps between roughly 0.2 and 1.5 pt (half-width 0.58–0.80 pt). The bounded-effect reading is sound; the stronger practitioner sentence that hardening is mainly for measurement, not capability, should be tied more tightly to this power floor wherever it appears outside §4.1/§7, so readers do not treat a non-detection band as near-zero damage.
- [§3.1 Arms, suites; §3.4 Verified-wrong audit; §4.3; §7 Suite operationalization] §3.1 / §7 suite operationalization: both the hardened reward and all held-out scoring use extra-tests-only, not the official EvalPlus base∧plus union, and the paper’s own audit finds plus-suite false negatives, contract-invalid inputs, and convention mismatches. The primary [C] gap is still well-identified because both arms are scored by the same extra suite, and the B/C union re-score leaves cells on the same side of the margin. For the FP definition itself (base-pass ∧ extra-fail) and for any claim that “~half the leak pays for real bugs,” a load-bearing sensitivity is still missing: recompute tracking and verified-wrong residual after excluding tasks with signed plus-suite defect predicates (or report the share of FP mass on defect-free tasks only). Without that, residual suite error remains a partial confounder of the [E] mechanism and residual-share claims even though it does not ov
- [§4.5 Mechanism; §4.7 800-step continuation; Abstract] §4.5–§4.7 mechanism (selection vs learning): the base-model control and flat/declining FP trajectories are the strongest mechanism evidence, and the 800-step continuation is a real strength. Family C’s continuation is underpowered (MDE +1.0 pp/100 steps), and parameter-level sharpening is only bounded above a small nats/token floor on B/C. The abstract’s unqualified “FP incidence does not grow” should carry the same powered-refutation / underpowered-non-detection split already stated in §4.7, and “selection rather than learned exploitation” should remain explicitly horizon- and scale-scoped so it is not read as a general RLVR law.
minor comments (7)
- [§1 Introduction; Figure 1] Define “hardened = extra-tests-only (not EvalPlus union)” in the first paragraph of the introduction and in Figure 1’s caption, not only in footnote 1 and §7. Many readers will assume MBPP+ means official union scoring.
- [Table 1; §4.6; Abstract] Table 1 / §4.6: Family C’s 62.78% verified-wrong share is flagged as least-audited and composition-sensitive (9.3% syntax-invalid). Keep that caveat adjacent to every table appearance, including the abstract’s “both replication families reproduce a large share,” so “large” is not read as a precise cross-family ranking.
- [§4.4; Abstract] §4.4 eval-side exposed-stratum association: the non-replication (B reversed candidate, C underpowered) is disclosed, but the abstract omits it while including other [E] items. Either drop the eval-side association from high-level summaries or state the non-replication in one clause.
- [§5; App. F] §5 meta-audit is carefully fenced and exploratory, but it is long relative to the paper’s confirmatory contribution. Consider moving more of the per-study receipts to the appendix (already partly done) and keeping only the capability-relative motivation paragraph in the main text to protect focus on the causal contrast.
- [Figure 2] Figure 2 encoding appears as Unicode private-use glyphs in the manuscript source dump; ensure the production PDF has clean axis labels and that zero-leakiness jitter is documented in the caption (it is mentioned in text).
- [§3.5; §4.1] §3.5 honesty notes are excellent; a short pointer from §4.1 to the unconditional vs base-conditional plus-pass trap would help readers who only skim results.
- [Title; Abstract] Minor typography: title spacing (“WHEN THEREWARDSUITEISLEAKY”), occasional missing spaces in headings, and consistent capitalization of [C]/[E]/[I] tags in the abstract vs body.
Circularity Check
No significant circularity: confirmatory claims are empirical two-arm contrasts and pre-training audits validated against independent train-time quantities under frozen preregistration.
full rationale
The paper’s load-bearing [C] results do not reduce to their inputs by construction. Non-inferiority is a measured hardened-minus-leaky held-out gap under a single fixed extra-test oracle applied to both arms, with margin and estimators frozen before outcome data; the gap is not defined from the leaky reward. Static leakiness is computed from base-model samples before any training and is then correlated with a separate mechanical train-time quantity (rewarded FP mass = base-pass ∧ extra-fail during GRPO); rank tracking (Spearman ~0.80) and the train-side leak-stratum gap (+43.8 pt) are empirical validations of that pre-training map, not fits renamed as predictions. Verified-wrong is labeled [E], human-adjudicated under signed predicates that can only shrink the exploitation claim, and is not used in the primary non-inferiority or classifier-free FP-mass tests. There is no self-citation chain, uniqueness theorem from the same author, or ansatz smuggled via prior work that forces the central bound. Residual suite defects (plus false negatives, extra-only vs union) are quantified and do not make the contrast tautological. Honest finding: self-contained empirical causal design.
Axiom & Free-Parameter Ledger
free parameters (4)
- non_inferiority_margin_1.5pt =
1.5 percentage points
- static_audit_k_samples =
16
- training_horizon_and_scale =
400 steps primary; 1–1.5B params
- tracking_min_rewarded_rollouts =
10
axioms (5)
- domain assumption Extra-tests-only MBPP+/HumanEval+ scoring is a valid held-out and FP-defining oracle relative to original MBPP base tests (even though base and extra are often disjoint and plus suite has defects).
- standard math Seed is the unit of inference (n=5; exact sign-flip floor p=0.031); intervals are two-sided 90% / one-sided 95% against margin unless stated.
- domain assumption GRPO with identical prompts, tasks, seeds, and compute isolates the reward suite as the sole causal difference between arms.
- ad hoc to paper Human-adjudicated verified-wrong predicates correctly separate model bugs from suite artifacts for mechanism claims.
- domain assumption Absence of rising FP-only share / rewarded-FP mass within measured horizon is evidence favoring selection over learned exploitation (not proof of zero at larger scale).
invented entities (2)
-
static leakiness score
independent evidence
-
verified-wrong residual (taxonomy)
no independent evidence
read the original abstract
The test suites used as RLVR rewards for code have natural false positives: per-task, persistent, asymmetric errors that accept the same wrong programs every time they appear, unlike the symmetric or resampled noise assumed by existing noise-robustness analyses. We run a preregistered two-arm causal contrast on a deployed suite: GRPO on identical MBPP tasks, seeds, and compute, rewarded by the original MBPP tests (leaky) versus the MBPP+ extra tests (hardened). Two further families replicate the design under a preregistration frozen before their data existed. [C] The average held-out effect is bounded: non-inferior under a preregistered 1.5-pt margin (gap 0.20 pt, one-sided 95% upper bound 0.75 pt). [C] Rewarded false-positive mass tracks a cheap static leakiness audit computed before training (Spearman 0.80), and the registered train-side test puts the leak-stratum FP share +43.8 pt above clean tasks. [E] Auditing every rewarded FP under signed, human-adjudicated rules finds a large residual of verified genuinely wrong code: 47.57% record-weighted; both replication families reproduce a large share. The reward paid for real bugs, not merely suite artifacts. [E] Mechanism evidence is consistent with selection of pre-existing error modes rather than learned exploitation: FP incidence does not grow within our horizon, and untrained base models already produce the same wrong outputs under the leaky filter. We then turn the same instrument on the frontier judges themselves: on their own false positives they self-assess only weakly, a same-author test is unresolved, and even the highest-scoring reader we probe stays far below its score on a weaker policy's errors -- two subjects on MBPP, licensing nothing about frontier models in general. A cheap static audit locates exposure before training; hardening the reward removes the measurement inflation, though here it buys little capability.
Figures
Reference graph
Works this paper leans on
-
[1]
Program synthesis with large language models.arXiv preprint arXiv:2108.07732,
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,
-
[2]
Sambaran Bandyopadhyay. Can LLMs judge better than they generate? evaluating task asym- metry, mechanistic interpretability and transferability for in-context QA.arXiv preprint arXiv:2606.28050,
-
[3]
Mohammad Beigi, Ming Jin, and Lifu Huang. Proxy reward internalization and mechanistic exploitation: A learned precursor to reward hacking and its generalization.arXiv preprint arXiv:2606.09711,
-
[4]
Tuan-Dung Bui, Thanh Trong Vu, Thu-Trang Nguyen, Son Nguyen, and Hieu Dinh V o. Correctness assessment of code generated by large language models using internal representations.arXiv preprint arXiv:2501.12934,
-
[5]
Xin-Qiang Cai, Wei Wang, Feng Liu, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Rein- forcement learning with verifiable yet noisy rewards under imperfect verifiers.arXiv preprint arXiv:2510.00915,
-
[6]
Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond binary rewards: Training LMs to reason about their uncertainty.arXiv preprint arXiv:2507.16806,
-
[7]
Mehul Damani, Isha Puri, Idan Shenfeld, and Jacob Andreas. Right in the right way: LM training with verifiable rewards and human demonstrations.arXiv preprint arXiv:2607.01181,
-
[8]
Darshan Deshpande, Anand Kannappan, and Rebecca Qian
doi: 10.1038/s41586-025-09422-z. Darshan Deshpande, Anand Kannappan, and Rebecca Qian. Benchmarking reward hack detection in code environments via contrastive analysis.arXiv preprint arXiv:2601.20103,
-
[9]
Florian E. Dorner, Vivian Y . Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data.arXiv preprint arXiv:2410.13341,
-
[10]
Dorner, Robin Staab, and Martin Vechev
Kazuki Egashira, Mark Vero, Jasper Dekoninck, Florian E. Dorner, Robin Staab, and Martin Vechev. Delay, plateau, or collapse: Evaluating the impact of systematic verification error on RLVR.arXiv preprint arXiv:2605.02909,
-
[11]
EvilGenie: A reward hacking benchmark
Jonathan Gabor, Jayson Lynch, and Jonathan Rosenfeld. EvilGenie: A reward hacking benchmark. arXiv preprint arXiv:2511.21654,
-
[12]
William Guey and Pierrick Bougault. Self-preference is weak or absent in verifiable instruction-following revision: A four-model test under genuine authorship.arXiv preprint arXiv:2606.20093,
-
[13]
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y . Wu, Y . K. Li, et al. DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence.arXiv preprint arXiv:2401.14196,
-
[14]
URLhttps: //transformer-circuits.pub/2026/workspace/index.html. 20 Lukas Helff, Quentin Delfosse, David Steinmann, Ruben H ¨arle, Hikaru Shindo, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting, and Felix Friedrich. LLMs gaming veri- fiers: RLVR can lead to reward hacking.arXiv preprint arXiv:2604.15149,
Pith/arXiv arXiv 2026
-
[15]
Qwen2.5-Coder technical report.arXiv preprint arXiv:2409.12186,
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-Coder technical report.arXiv preprint arXiv:2409.12186,
-
[16]
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate.arXiv preprint arXiv:2509.04664,
-
[17]
Muhammad Khalifa, Zohaib Khan, Omer Tafveez, Hao Peng, and Lu Wang. Countdown-code: A testbed for studying the emergence and generalization of reward hacking in RLVR.arXiv preprint arXiv:2603.07084,
-
[18]
Self-attribution bias: When AI monitors go easy on themselves.arXiv preprint arXiv:2603.04582,
Dipika Khullar, Jack Hopkins, Rowan Wang, and Fabien Roger. Self-attribution bias: When AI monitors go easy on themselves.arXiv preprint arXiv:2603.04582,
-
[19]
Miseong Shawn Kim. Compete then collaborate: Frontier AI teachers build a verifiable curriculum to improve a coding student beyond imitation.arXiv preprint arXiv:2607.08255,
-
[20]
Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishi...
-
[21]
Rethinking verification for LLM code generation: From generation to testing
Zihan Ma, Taolin Zhang, Maosong Cao, Junnan Liu, Wenwei Zhang, Minnan Luo, Songyang Zhang, and Kai Chen. Rethinking verification for LLM code generation: From generation to testing. arXiv preprint arXiv:2507.06920,
-
[22]
Natural emergent misalignment from reward hacking in production RL.arXiv preprint arXiv:2511.18397,
Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, et al. Natural emergent misalignment from reward hacking in production RL.arXiv preprint arXiv:2511.18397,
-
[23]
Mitigating self-preference by authorship obfuscation.arXiv preprint arXiv:2512.05379,
Taslim Mahbub and Shi Feng. Mitigating self-preference by authorship obfuscation.arXiv preprint arXiv:2512.05379,
-
[24]
Reward hacking in rubric-based reinforcement learning.arXiv preprint arXiv:2605.12474,
Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu, and Yunzhong He. Reward hacking in rubric-based reinforcement learning.arXiv preprint arXiv:2605.12474,
-
[25]
Andreas Plesner, Francisco Guzm ´an, and Anish Athalye
URLhttps://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/. Andreas Plesner, Francisco Guzm ´an, and Anish Athalye. An imperfect verifier is good enough: Learning with noisy rewards.arXiv preprint arXiv:2604.07666,
Pith/arXiv arXiv 2024
-
[26]
Ali Rad, Khashayar Filom, Darioush Keivan, Peyman Mohajerin Esfahani, and Ehsan Kamalinejad. Rate or fate? RLVεR: Reinforcement learning with verifiable noisy rewards.arXiv preprint arXiv:2601.04411,
-
[27]
Auditing reward hackability in code RL training environments.arXiv preprint arXiv:2606.16062,
21 Shreshth Rajan. Auditing reward hackability in code RL training environments.arXiv preprint arXiv:2606.16062,
-
[28]
Before the model learns the bug: Fuzzing RLVR verifiers.arXiv preprint arXiv:2606.01066,
Jaideep Ray. Before the model learns the bug: Fuzzing RLVR verifiers.arXiv preprint arXiv:2606.01066,
-
[29]
Spurious rewards: Rethinking training signals in RLVR.arXiv preprint arXiv:2506.10947,
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in RLVR.arXiv preprint arXiv:2506.10947,
-
[30]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[31]
Jingwei Shi, Xinxiang Yin, Jing Huang, Jinman Zhao, and Shengyu Tao. CodeHacker: Automated test case generation for detecting vulnerabilities in competitive programming solutions.arXiv preprint arXiv:2602.20213,
-
[32]
The limits of inference scaling through resampling.arXiv preprint arXiv:2411.17501,
Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. The limits of inference scaling through resampling.arXiv preprint arXiv:2411.17501,
-
[33]
Mohammad Taufeeque, Stefan Heimersheim, Adam Gleave, and Chris Cundy. The obfusca- tion atlas: Mapping where honesty emerges in RLVR with deception probes.arXiv preprint arXiv:2602.15515,
-
[34]
Preregistering NLP research
Emiel van Miltenburg, Chris van der Lee, and Emiel Krahmer. Preregistering NLP research. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 613–623,
2021
-
[35]
The verification horizon: No silver bullet for coding agent rewards
Binghai Wang, Chenlong Zhang, Dayiheng Liu, Jiajun Zhang, Jiawei Chen, Mingze Li, Mouxiang Chen, Rongyao Fang, et al. The verification horizon: No silver bullet for coding agent rewards. arXiv preprint arXiv:2606.26300,
-
[36]
Patrick Wilhelm, Thorsten Wittkopp, and Odej Kao. Monitoring emergent reward hacking during generation via internal activations.arXiv preprint arXiv:2603.04069,
-
[37]
Fang Wu, Aaron Tu, Weihao Xuan, Heli Qi, Xu Huang, Qingcheng Zeng, Shayan Talaei, Yijia Xiao, Peng Xia, Xiangru Tang, Yuchen Zhuang, Yinxi Li, Bing Hu, Hanqun Cao, Wenqi Shi, Rui Yang, Nan Liu, Huaxiu Yao, Ge Liu, Li Erran Li, Amin Saberi, Naoto Yokoya, Jure Leskovec, and Yejin Choi. Position: The hidden costs and measurement gaps of reinforcement learnin...
-
[38]
Rui Wu and Ruixiang Tang. When reward hacking rebounds: Understanding and mitigating it with representation-level signals.arXiv preprint arXiv:2604.01476,
-
[39]
Xing Zhang, Yanwei Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, and Peiyang He. The blind curator: How a biased judge silently disables skill retirement in self-evolving agents.arXiv preprint arXiv:2607.07436,
-
[40]
SpecBench: Measuring reward hacking in long-horizon coding agents.arXiv preprint arXiv:2605.21384,
Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu, and Zhengyao Jiang. SpecBench: Measuring reward hacking in long-horizon coding agents.arXiv preprint arXiv:2605.21384,
-
[41]
Hardening agent benchmarks with adversarial hacker-fixer loops.arXiv preprint arXiv:2606.08960,
Ziqian Zhong, Ivgeni Segal, Ivan Bercovich, Shashwat Saxena, Kexun Zhang, and Aditi Raghu- nathan. Hardening agent benchmarks with adversarial hacker-fixer loops.arXiv preprint arXiv:2606.08960,
-
[42]
Chenyu Zhou. More convincing, not more correct: Self-play reward hacking of reference-free LLM judges.arXiv preprint arXiv:2607.05904,
-
[43]
22 Yefan Zhou, Austin Xu, Yilun Zhou, Janvijay Singh, Jiang Gui, and Shafiq Joty. Variation in verification: Understanding verification dynamics in large language models.arXiv preprint arXiv:2509.17995,
-
[44]
the hardened suite
23 A GATE LEDGER The two failed launch gates of §3.3, in full. Gate b (blind hand-audit, 24 samples): VOID — 8/24 verdicts hinged on suite-side properties; its diagnosis established 0 scorer-vs-evalplus divergences on the same 24 samples. Gate b ′ (evidence-grounded, one-shot, 20 verified-wrong samples): FAIL — 6/20 overturned on evidence, with a post-mor...
2048
-
[45]
another model’s
lower them, while single-instance variance remains unquantifiable in both directions). The .959-versus-.74/.76 contrast is adescriptive comparison across different item pools and differ- ent protocol stages(400-row fresh-session multi-shard vs2680-row single-session batch),not a same-instrument measurement. With that grading, the three-reader picture is d...
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.