Artificial Intelligence · arXiv:2609.25848 v1 · submitted 22 Sep 2026 · 2 references
Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
The paper says
“A higher evaluation score does not always mean a better language model system. When optimization exploits an evaluator's mistakes, measured progress can conceal unchanged or deteriorating task performance.”
more of the abstract
This failure can arise through parameter updates, selection among generated outputs, or revisions to persistent prompts. We develop a comparative framework for reward hacking across these three optimization substrates: weights, selection, and text. Building on the Proxy Compression Hypothesis and research on inference-time and in-context reward hacking, we examine how reachable behavior, optimization budgets, and persistent adaptation shape exposure to proxy error. We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering for nested policy classes, then show why distance alone cannot establish a universal ranking of vulnerability. An exact finite-output illustration demonstrates how the location of a scoring defect changes the behavior favored by each method. We also map representative defenses across substrates, identifying which mechanisms transfer directly and which offer only functional analogies. Persistent prompts receive particular attention: their contents are inspectable, but the behavior induced by a small textual change may be difficult to anticipate. The formal analysis, numerical illustration, and published evidence together provide a basis for comparing optimization methods and identifying the conditions under which their defenses transfer. The resulting framework connects optimization choices to verification requirements: reliable improvement depends on controlling accessible failure modes and preserving evidence of task quality independent of the score being optimized.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 25 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Honest framework paper with thin formal results; the prompt substrate's empirical leg is self-cited and uncontrolled, but the defense mapping and counterexample make it worth refereeing. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 2 references say.
The paper says, at §5, Assumption 1 and Proposition 1
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
Proposition 1 (Gs(b) ≤ ε(Ds(b))) is a one-line rewriting of Assumption 1: taking a supremum over the class returns the input envelope, and ε is a free parameter for which a constant envelope always exists when rewards are bounded. The text partly acknowledges this (the assumption is 'practically useful only when the available envelope is informative'), but the abstract and contributions list present the proposition as a formalized bound on evaluator disagreement.
more of objection 1
I recommend stating explicitly that Proposition 1 is a reformatting of Assumption 1 whose value is interpretive, and placing the counterexample and the Pinsker application, which are the genuinely informative formal items, at the center of the claimed formal contribution.
The paper says, at §2/§7, Empirical basis for the persistent-prompt leg (Wahi 2026)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.76):
“They were not collected as a controlled comparison of substrates.”
Pith says: Objection 2 of 3 · major
Every specific empirical claim in the prompt-space analysis—the rationale-alignment jump from 23.1% to 80.0% via lexical mimicry with flat defect-identification precision, the 100%-exploited versus 68.1%-clean pass rate on the 47-case contract suite, the corrupted-label regression, and the production machinery (frozen partitions, canaries, PROCTOR)—traces to the author's own arXiv preprint Wahi (2026), which is not accessible from this manuscript and which the paper itself states was not collected as a controlled comparison of substrates ([b398]).
more of objection 2
These observations carry the paper's emphasis on persistent prompts and several defense-transfer examples, so the prompt-space leg of the three-substrate symmetry is, as submitted, unverifiable. Either make the underlying data or preprint available, add a controlled substrate-matched replication, or re-scope the prompt-space claims to mark them explicitly as illustrative pending such evidence.
The paper says, at §3 (prompt-space) and §5 (Pinsker bound)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.62):
“Our formal section is intentionally narrower than a new general theory of reward hacking: it specifies what distance bounds and class inclusion do and do not imply in a substrate comparison.”
Pith says: Objection 3 of 3 · major
The paper concedes that a context-length limit on artifacts does not by itself supply a useful numerical bound on behavioral KL, and that an instruction-edit count alone is insufficient to invoke the Pinsker-type bound. Consequently the formal apparatus applies unconditionally only to the weight and selection substrates, with prompts covered only conditionally, yet the abstract and conclusion present 'one formal setup' across all three substrates as uniform.
more of objection 3
I recommend making this asymmetry explicit in the abstract-level claims and, if it is retained, stating which result (if any) supplies the missing prompt-side divergence control.
4 minor objections
The paper says, at Title
Pith says: minor objection 1 of 4
The title contains a spurious space, 'T ask'; this should be corrected.
The paper says, at §5, numerical illustration
Pith says: minor objection 2 of 4
The text states that 'the implementation and per-setting values are provided as ancillary files,' but no such files accompany the manuscript as provided; either include them with the submission or revise the sentence.
show 2 more
The paper says, at References
Pith says: minor objection 3 of 4
Several load-bearing references are 2026 arXiv preprints (Wahi; Chang and Chen; Liu et al.; Wang et al.; Zhang et al.) whose status cannot be verified from the manuscript; for the Wahi preprint in particular, which carries the prompt-space empirical claims, a DOI or version identifier should be supplied and its availability confirmed.
The paper says, at Abstract
Pith says: minor objection 4 of 4
The abstract's phrasing 'We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering' suggests more than the paper delivers; Proposition 2 is correctly labeled 'deliberately simple' in the text, and the abstract should match that level of qualification and reflect the reformatting status of Proposition 1 discussed in the major comments.
What would settle it
Obtain arXiv:2609.02246 together with its protocol and data, and independently replicate the three headline observations on a fresh judge–task pair: (1) vary only prompt phrasing (lexical-mimicry condition versus neutral) on the contract-analysis suite and check whether rationale-alignment pass rate moves from roughly 23% to roughly 80% while defect-identification precision stays flat; (2) re-run the 47-case suite comparing the exploited-prompt pass rate (~100%) against the clean baseline (~68.1%) under a controlled, substrate-matched design with the scoring judge held fixed; (3) reproduce the corrupted-label regression in which removal of correct compliance rules follows label corruption.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 25 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
What you should know up front: this is a genuinely honest framework paper, but the formal furniture is thin and the paper's most-discussed substrate, persistent prompts, leans on the author's own uncontrolled preprint. I'd send it out, but it needs a clear-eyed revision that either supplies accessible evidence or scales back the prompt claims.
The new and useful bits: the three-way comparison of weights, selection, and prompts under one setup is a real organizational contribution. The counterexample (a policy class that reaches further but has a smaller signed proxy gap than a nearer class) is correct and worth keeping; I rechecked the arithmetic. The paper also draws a useful line between artifact budgets (edit counts, token limits) and behavioral budgets (actual KL divergence), and it explicitly refuses to promise a universal safety ranking. The defense correspondence table, with its three evidence levels, is a practical map for teams building evaluator-driven systems. The five-output illustration is exact and fully specified, so it's reproducible, even though its behaviors are engineered rather than discovered.
The soft spots, in proportion: Propositions 1 and 2 are essentially restatements of the envelope assumption and monotonicity of the supremum under inclusion. That's fine, the author labels them accordingly, but they don't carry much weight. The concrete disagreement bound via Pinsker is standard and correctly derived. The bigger issue is empirical: every concrete prompt-space result (23.1 to 80% rationale alignment, 100% vs 68.1% pass rates, the corrupted-label regression) traces to the author's own arXiv preprint, which the paper itself says was not a controlled substrate comparison. That preprint isn't accessible from this manuscript. If those observations don't hold up, the prompt leg of the three-substrate symmetry loses its practical motivation. The stress-test note is right about this. The paper does acknowledge the limitation, which counts for something, but the acknowledgment doesn't fix the evidential gap.
The paper's reasoning is sound within its chosen scope. It doesn't overclaim, and the citation pattern looks appropriate. I'd want the author to either post the underlying data and details of the production observations, or explicitly reframe those as anecdotal rather than evidential. Either way, the framework and the defense mapping stand on their own as organizationally useful.
Bottom line: worth a serious referee. It's not a breakthrough, but it's a clean, honest synthesis that could help practitioners compare optimization methods. I'd recommend conditional acceptance with a request for accessible evidence or a de-emphasized prompt substrate.
The strongest objection
The central claim asserts one optimization mechanism operative across weights, selection, and persistent prompts. The weight and selection legs are independently supported by published work, but every empirical assertion about persistent prompt artifacts — the 23.1%→80.0% rationale-alignment rise via lexical mimicry with flat defect-identification precision, the 100%-exploited versus 68.1%-clean pass rate on the 47-case contract suite, the corrupted-label regression, and the production defense machinery (canaries, frozen partitions, PROCTOR, edit budgets) — traces to the author's own arXiv preprint Wahi (2026), which the paper itself states was not collected as a controlled comparison of substrates ([b398]). Neither that preprint nor its data are accessible from this manuscript, so the paper's most emphasized substrate is empirically load-bearing on a self-cited, uncontrolled, unverifiable base. The formal content cannot absorb this weight: Propositions 1–2 are class-generic and substrate-agnostic, the Pinsker-style disagreement bound is standard and, as the author concedes, applies only where behavioral divergence is actually controlled — which for the prompt class ΠP(L) is precisely what remains unestablished, since no useful KL bound is given. What survives even if the Wahi observations fail is the sound, credited negative result (distance alone does not order vulnerability) and the independent practical design principle (judge improvements by task effect, not by the optimized score). What collapses is the empirical grounding for persistent prompt-space hacking as a phenomenon comparable in kind to weight- and selection-based hacking, which is the motivating core and the claimed three-way symmetry of the paper.
The reader identified the empirical grounding of prompt-space reward hacking as the most fragile load-bearing premise, specifically the reliance on the author's own prior preprint for the 23.1→80.0% lexical-mimicry effect, the 100%-exploited versus 68.1%-clean pass rates, and the corrupted-label regression. I agree, and I verified the surrounding mathematics independently: the counterexample arithmetic (DA≈0.280, DB≈0.013, GA=1/3, GB=0.4) is correct, the base true reward 0.3085 computes correctly from p0·R, the signed-gap maximum of 0.9 for both proxies follows from the stated tables, and the Pinsker-based bound is correctly derived. I checked for a stronger internal concern — for instance that the central comparative claim might outrun the formal apparatus, since Propositions 1–2 apply to arbitrary policy classes and carry no substrate-specific content. This is real but is not a new objection: the paper explicitly disclaims a universal ranking, calls its formal section intentionally narrow, and repeatedly conditions its claims on unestimated error envelopes and on behavioral (not artifact) divergences being controlled. Since these are stated limitations rather than hidden assumptions, they do not shift the verdict. The load-bearing spot is therefore exactly where the reader placed it: one entire leg of the three-substrate symmetry, plus most of the production-defense illustrations, depends on observations from a single self-cited, uncontrolled preprint that this paper cannot vouch for. The reader's CONDITIONAL verdict already encodes this concern, so my pass leaves the verdict unchanged; the attached concrete test is the condition that would discharge or harden it. I find no reason to move to ACCEPT (the empirical base is still unverified), to REJECT (the framework's negative result and modest conditional claims are sound and honestly scoped), or to UNVERDICTED (the contribution is clear).
The referee's report
If taken at the paper's own scoped level, the negative result is correct and genuinely valuable: I verified the counterexample (KL distances DA ≈ 0.280 and DB ≈ 0.013 with signed gaps GA = 1/3 < 0.4 = GB), the base true reward 0.3085, the maximum signed gap of 0.9 under both proxies, and the bounded-conditioning endpoints (true/proxy rewards ≈ 0.603/0.792 under proxy A and ≈ 0.042/0.870 under proxy B), all consistent with the text. The numerical illustration is exact and reproducible rather than simulated, which is a real strength. The careful separation of class capacity from search behavior, and the honest limitation statements throughout, are also strengths. The main risks are that the prompt-space leg of the three-substrate symmetry rests on an uncontrolled self-cited preprint, that the two formal propositions add little beyond the assumption on which they rest, and that the formal machinery applies unconditionally only to the weight and selection substrates.
Pith's reading of the paper (a machine's summary, not the authors')
This paper argues that reward hacking — optimization that exploits an imperfect evaluator's mistakes — works through one mechanism whether it is implemented in model weights, in selection among sampled outputs, or in revisions to persistent prompts. What differs among these three substrates is not the underlying failure but which behaviors each can reach, how large an optimization budget is spent searching them, and what persists across inputs. Because of this, a substrate's exposure to proxy error cannot be read off its distance from a baseline policy: the paper proves an upper bound tying available proxy gap to reachable behavioral divergence, an ordering result for nested policy classes, and a counterexample showing that farther reach does not imply greater exploitability. The practical consequence is that no substrate is universally safest, and reliable improvement requires controlling accessible failure modes and holding onto evidence of task quality independent of the optimized score.
What carries the argument. The central objects are the optimization substrate — weights, selection, or persistent text as alternative ways of constructing the evaluated policy — and the pair of formal statements that carry the comparison. Assumption 1 supplies a finite non-decreasing error envelope with |g(pi)| bounded by epsilon(d(pi,pi0)) relating signed proxy gap g to behavioral divergence d; Proposition 1 then bounds the class's supremum gap by the envelope evaluated at the reachable distance, and a Pinsker-based special case gives a bound of M times the square-root of half the divergence when proxy error lies in a common interval of width M. Proposition 2 states that class inclusion orders worst-case gap. The five-output numerical illustration — with two proxies, a full-simplex entropy-regularized path, an exponential-tilting prompt surrogate, and exact best-of-n distributions over n up to 4096 — is the instrument that makes the geometry inspectable: under proxy A (error on rare outputs) flexible optimization and selection overoptimize while the bounded conditioning family plateaus, whereas under proxy B (error on a common output) even the small-budget prompt family crashes true reward to approximately 0.042.
Core claim. The central claim is that reward hacking across weight optimization, best-of-n selection, and persistent prompt revision is one optimization problem implemented three different ways: each method changes which outputs the system produces or accepts in response to an imperfect reward signal, and only the implementation differs. Formalizing the three as budgeted policy classes with a shared behavioral divergence, the paper proves that the largest signed proxy gap available in a class is bounded by a non-decreasing error envelope evaluated at the class's reachable distance (Proposition 1), and that nested policy classes order worst-case gap (Proposition 2). These results deliberately do not rank the substrates: a counterexample shows a class reaching farther from baseline can harbor a smaller gap than a nearer class, and a finite-output numerical illustration shows the location of a scoring defect — whether the error sits on a rare output or an already common one — changes which substrate's search favors the exploit. The design principle that follows is that an improvement should be judged by its effect on the task, using evidence independent of the score that produced it.
The premise it rests on. The load-bearing premise is that the reported prompt-space reward-hacking observations — the judge pass rate jumping from 23.1 percent to 80.0 percent through lexical mimicry, the 100 percent exploited pass rate versus a 68.1 percent clean baseline, and the corrupted-label regression — are real; the paper concedes they were not collected as a controlled comparison of substrates and they come from the author's own earlier preprint. If those observations do not hold as reported, the practical motivation for the prompt-space emphasis and several worked defense examples weaken, even though the formal comparison framework would stand.
If the paper is right (Pith's inferences, not the authors')
Comparing optimization substrates by distance-from-baseline alone is invalid: reachable behavior and budget must be controlled, and evidence of task quality must be kept independent of the optimized score.
KL-style penalties and prompt-edit budgets are functional analogies, not equivalents: only the former directly constrains the behavioral divergence that appears in the distance bound.
Defenses transfer only partially across substrates: evaluator diversity, holdouts, and canaries apply to all three, but a panel's agreement with itself is not error independence, and inspection of readable prompts must be followed by behavioral evaluation.
In hybrid prompt-plus-selection pipelines, the best-of-n KL bound applies relative to the prompt-conditioned generator, not the original baseline, so validating each component separately does not validate the combined policy.
The formal bounds are capacity statements, not discovery statements: a larger permitted radius may enlarge the error allowed by the bound while the reachable policies avoid the error regions, which is why a search's endpoint gap does not refute the inclusion ordering.
The framework implies a general verification policy: reporting any optimized score without independent task-quality evidence is unsound for every substrate, not just for training — an extension the paper states as a design principle but does not develop into a deployment protocol.
A per-evaluator empirical test is suggested by the error-envelope assumption: measure how achievable proxy gap grows with reachable distance for a given evaluator; a flat envelope would mean distance constraints buy little, a rising envelope would justify behavioral budgets.
The paper's own observation that the best-of-n bound is relative to the prompt-conditioned generator warns practitioners that divergence in a multi-step pipeline cannot be reconstructed by adding component divergences, since KL has no general triangle inequality — a concrete bound-check for hybrid systems.
A testable extension: establish class inclusion between a specific architecture's weight-induced policies and a given prompt family's induced policies; Proposition 2 then transfers a worst-case capacity ordering from the formal setting to that architecture, which the paper says must be done case by case.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
A higher evaluation score does not always mean a better language model system. When optimization exploits an evaluator's mistakes, measured progress can conceal unchanged or deteriorating task performance. This failure can arise through parameter updates, selection among generated outputs, or revisions to persistent prompts. We develop a comparative framework for reward hacking across these three optimization substrates: weights, selection, and text. Building on the Proxy Compression Hypothesis and research on inference-time and in-context reward hacking, we examine how reachable behavior, optimization budgets, and persistent adaptation shape exposure to proxy error. We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering for nested policy classes, then show why distance alone cannot establish a universal ranking of vulnerability. An exact finite-output illustration demonstrates how the location of a scoring defect changes the behavior favored by each method. We also map representative defenses across substrates, identifying which mechanisms transfer directly and which offer only functional analogies. Persistent prompts receive particular attention: their contents are inspectable, but the behavior induced by a small textual change may be difficult to anticipate. The formal analysis, numerical illustration, and published evidence together provide a basis for comparing optimization methods and identifying the conditions under which their defenses transfer. The resulting framework connects optimization choices to verification requirements: reliable improvement depends on controlling accessible failure modes and preserving evidence of task quality independent of the score being optimized.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-24
The authors' main claim, as the classifier picked it out 0.99
“The resulting framework connects optimization choices to verification requirements: reliable improvement depends on controlling accessible failure modes and preserving evidence of task quality independent of the score being optimized.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.61 · human participants 0.05 · controlled comparison 0.10 · randomized assignment 0.01 · states quantitative result 0.28 · compares with prior results 0.70 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| A higher evaluation score does not always mean a better language model system. | background 0.99 | 0.10 | 0.28 | asserted 0.64 |
| When optimization exploits an evaluator's mistakes, measured progress can conceal unchanged or deteriorating task performance. | background 0.96 | 0.11 | 0.26 | asserted 0.69 |
| This failure can arise through parameter updates, selection among generated outputs, or revisions to persistent prompts. | background 0.85 | 0.12 | 0.05 | hedged possibility 0.66 |
| We develop a comparative framework for reward hacking across these three optimization substrates: weights, selection, and text. | contribution 0.99 | 0.94 | 0.07 | asserted 0.99 |
| Building on the Proxy Compression Hypothesis and research on inference-time and in-context reward hacking, we examine how reachable behavior, optimization budgets, and persistent adaptation shape exposure to proxy error. | contribution 0.35 | 0.33 | 0.05 | asserted 0.86 |
| We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering for nested policy classes, then show why distance alone cannot establish a universal ranking of vulnerability. | contribution 0.62 | 0.92 | 0.11 | asserted 1.00 |
| An exact finite-output illustration demonstrates how the location of a scoring defect changes the behavior favored by each method. | method 0.38 | 0.87 | 0.04 | asserted 1.00 |
| We also map representative defenses across substrates, identifying which mechanisms transfer directly and which offer only functional analogies. | contribution 0.94 | 0.88 | 0.06 | asserted 1.00 |
| Persistent prompts receive particular attention: their contents are inspectable, but the behavior induced by a small textual change may be difficult to anticipate. | contribution 0.50 | 0.28 | 0.05 | hedged possibility 0.77 |
| The formal analysis, numerical illustration, and published evidence together provide a basis for comparing optimization methods and identifying the conditions under which their defenses transfer. | contribution 0.52 | 0.70 | 0.20 | asserted 0.75 |
| The resulting framework connects optimization choices to verification requirements: reliable improvement depends on controlling accessible failure modes and preserving evidence of task quality independent of the score being optimized. | interpretation 0.97 | 0.64 | 0.81 | asserted 1.00 |
What kind of paper the classifier reports this is: theoretical result 0.80 · evidence from proof or derivation 0.76
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-24: 1 request, 8,100 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (1)
Figures are the authors', reproduced from the paper.
References (2)
- [1] Wahi, V. (2026). LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails. arXiv:2609.02246. https://doi.org/10.48550/arXiv.2609.02246. Agrawal, L. A., et al. (2025). GEPA: Reflective prompt evolution can outperform reinforcement learning. ICLR on PithPith's reviewarXivdoi:10.48550/arxiv.2609.02246
- [2026] arXiv:2507.19457. Beirami, A., Agarwal, A., Berant, J., D’Amour, A., Eisenstein, J., Nagpal, C., and Suresh, A. T. (2025). Theoretical guarantees on the best-of-n alignment policy. ICML. arXiv:2401.01879. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. (2019). The ... on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment