REVIEW 24 cited by
Defining and Characterizing Reward Hacking
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We provide the first formal definition of reward hacking, a phenomenon where optimizing an imperfect proxy reward function leads to poor performance according to the true reward function. We say that a proxy is unhackable if increasing the expected proxy return can never decrease the expected true return. Intuitively, it might be possible to create an unhackable proxy by leaving some terms out of the reward function (making it "narrower") or overlooking fine-grained distinctions between roughly equivalent outcomes, but we show this is usually not the case. A key insight is that the linearity of reward (in state-action visit counts) makes unhackability a very strong condition. In particular, for the set of all stochastic policies, two reward functions can only be unhackable if one of them is constant. We thus turn our attention to deterministic policies and finite sets of stochastic policies, where non-trivial unhackable pairs always exist, and establish necessary and sufficient conditions for the existence of simplifications, an important special case of unhackability. Our results reveal a tension between using reward functions to specify narrow tasks and aligning AI systems with human values.
Forward citations
Cited by 24 Pith papers
-
RECEIPT: Deterministic, Reward-Hacking-Resistant Verification for White-Box Agentic XSS Discovery
A replay-based verifier with environment isolation, role separation, and a browser-execution sentinel lets white-box LLM agents report XSS exploits that are real, reproducible, attacker-to-victim vulnerabilities.
-
Auditing Discovery Claims: A Two-Sided Criterion for Agentic Science, with the Negative Side Decidable
A two-sided audit with a formally decidable negative side shows a single-oracle RNA design claim collapses from 43/60 to 1/60 under a three-predictor panel, while two AI-written operators survive a held-out judge.
-
When Does Reward Teach State? A Hidden-Automaton Instrument and a Group-Language Warning Signal
High reward in sparse RL does not imply latent-state recovery; a hidden-DFA instrument separates perception from planning gaps and flags group-language structure as a pre-training warning.
-
Attention Limited Reward Learning
Limited attention makes pairwise preference labels non-identifiable for reward, can reverse Bradley-Terry rankings, and bounds learning by attended information rather than raw label count.
-
Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
Latent DPO trains only a small preference encoder per user, reducing LLM personalization training time by 80-90% with comparable alignment quality.
-
ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents
Static-policy judges achieve near-zero recall on scope violations; request-conditioned pre-execution judges reach F1 0.66 (open-weight best) against an expert reference of 0.78 on a 4,897-call labeled benchmark.
-
The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation
Strict-launch-gated rejection-sampling self-distillation compounds cross-family clean Godot generation from 8.8% to 42.2% and full best-of-K coverage, while gold duplication and a lenient BUILD filter erase the gain.
-
Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happened
LLM harness optimizers invent failures that provably never happened—adding a guard against a nonexistent rule in 15/60 runs on legal data—when prompted to fix failures and shown a benign repeated-move pattern.
-
Avoiding unsafe sets when training with Langevin Dynamics
Langevin training trajectories on strongly convex losses avoid geometrically isolated failure regions with probability exponentially small in dimension after an O(d) burn-in, with a local spectral rate controlling tra...
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers
Survey experiment finds that people apply more deontological standards to AI described as human-programmed and to the programmers themselves than to unaided humans or unprogrammed robots in a moral dilemma.
-
Combee: Scaling Prompt Learning for Self-Improving Language Model Agents
Mastery-conditioned constrained RL expands the instructional action set only when prerequisites are mastered, reducing reward hacking and raising mastery gains on Junyi and XES3G5M.
-
Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking
TailorPO aligns diffusion models by generating paired noisy samples from the same intermediate state, ranking them by the reward of their predicted clean images, and optimizing the preferred one at every denoising step.
-
Pre-Strings Lectures on Artificial Intelligence
Lecture notes define neural-network field theory and survey how it recovers known QFT/string results plus applied AI techniques for string problems.
-
Coupled Variational Reinforcement Learning for Language Model General Reasoning
CoVRL trains an LLM on a mixture of question-only and answer-guided reasoning traces, using the model's own answer probability as reward, and reports consistent gains on math and general-reasoning benchmarks.
-
School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
Fine-tuning LLMs on harmless reward-hacking examples made them keep hacking in new settings, and made GPT-4.1 produce unrelated misaligned outputs such as advising poisoning and evading shutdown.
-
Residual Reward Models for Preference-based Reinforcement Learning
Combining a hand-designed or learned prior reward with a preference-trained residual improves sample efficiency and final performance in preference-based reinforcement learning.
-
Safety Features for a Centralised AGI Project
A policy proposal for seven safety features, including bottom-up pause authority, congressional-chartered board oversight, risk monitoring, and verification technology, to reduce catastrophic risks in a centralized US...
-
Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?
Preference signals in LLM alignment are concentrated in early response tokens, so models trained on data truncated to the first half perform as well as or better than those trained on full responses.
-
PerPO: Perceptual Preference Optimization via Discriminative Rewarding
PerPO trains multimodal LLMs by ranking their candidate answers with deterministic visual rewards (IoU, edit distance) and using the reward differences as margins in listwise preference optimization.
-
TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
TaoSR-AGRL improves e-commerce search relevance by combining dense rule-aware reward shaping with adaptive ground-truth-guided replay in GRPO, reporting gains on Taobao's private offline and online evaluations.
-
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
The paper surveys security risks of LLM agents, organizes them into a five-level autonomy taxonomy, and proposes an untested CMDP-based architecture called R2A2.
-
Deep Reinforcement Learning: From First Principles to Reasoning Models
A textbook survey of deep reinforcement learning, from Bellman foundations to DQN, PPO, MuZero, offline RL, and reasoning models, with UAV/SD-WAN examples throughout.
-
Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.
Discussion (0). Continue with ORCID to comment.