REVIEW 7 cited by
Spontaneous Reward Hacking in Iterative Self-Refinement
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Language models are capable of iteratively improving their outputs based on natural language feedback, thus enabling in-context optimization of user preference. In place of human users, a second language model can be used as an evaluator, providing feedback along with numerical ratings which the generator attempts to optimize. However, because the evaluator is an imperfect proxy of user preference, this optimization can lead to reward hacking, where the evaluator's ratings improve while the generation quality remains stagnant or even decreases as judged by actual user preference. The concern of reward hacking is heightened in iterative self-refinement where the generator and the evaluator use the same underlying language model, in which case the optimization pressure can drive them to exploit shared vulnerabilities. Using an essay editing task, we show that iterative self-refinement leads to deviation between the language model evaluator and human judgment, demonstrating that reward hacking can occur spontaneously in-context with the use of iterative self-refinement. In addition, we study conditions under which reward hacking occurs and observe two factors that affect reward hacking severity: model size and context sharing between the generator and the evaluator.
Forward citations
Cited by 7 Pith papers
-
More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
Self-play against reference-free LLM judges drives judge pass rates to 0.94 while true accuracy stays at 0.20, a reward-hacking basin that transfers across judge families and is prevented only by requiring the judge t...
-
When Do Agent Loops Mistake Stagnation for Progress? Self-Evaluation Bias and Externally Grounded Verification in Long-Running Autonomous LLM Agent Loops
For open-ended agent goals whose success lives outside the transcript, even a strong in-band judge fails; out-of-band world-state gating is structurally required to stop the progress mirage.
-
Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agents
In noisy verify-repair loops, repair only while the estimated expected gain (1-b)α - bβ stays positive; stop when the belief crosses b* = α/(α+β), and fall back to keep-best when calibration is unreliable.
-
Memory Reward Inflation in Self-Improving LLM Agents
Self-graded memory in LLM agents systematically overvalues wrong episodes, compounding through reuse, and can be corrected by a de-correlated, answer-free signal.
-
L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training
A distilled 99M-parameter CLIP gives L-CLIPScore, a lightweight caption metric that matches CLIPScore on human correlation and best improves captioning models when mixed with CIDEr.
-
LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition
Reference-free LLM judge scores failed to select better table-extraction outputs over eight regeneration iterations on FinTabNet and OmniDocBench; keeping the first output was safest.
-
Language Games as the Pathway to Artificial Superhuman Intelligence
A position paper arguing that open-ended language games with fluid roles, varied rewards, and evolving rules can drive expanded data reproduction and thus a path to artificial superhuman intelligence.
Discussion (0). Continue with ORCID to comment.