The Reward Hacking Benchmark shows RL post-training raises exploit rates in tool-using LLM agents from 0.6% to 13.9%, with environmental hardening cutting exploits by 87.7% relative without lowering task success.
hub
School of Reward Hacks : Hacking harmless tasks generalizes to misaligned behavior in LLMs , August 2025
15 Pith papers cite this work. Polarity classification is still indexing.
15
Pith papers citing it
hub tools
citation-role summary
background 1
citation-polarity summary
years
2026 15roles
background 1polarities
background 1representative citing papers
Monitors trained on prompt-elicited reward-hacking trajectories fail to generalize to hacking behaviors that arise naturally during RL training of code models, whereas trajectories curated by Trace-and-Amplify transfer better and to unseen hack types.
Reinforcement learning on beneficial traits in realistic domains yields broad improvements on over 80% of out-of-distribution alignment benchmarks and greater resistance to adversarial steering.
In LLM agents, reward-hack activation marks a latent policy state, but next-step risky behavior is best predicted when that signal is combined with token entropy and decision context.
Consistency training suppresses reward hacking and emergent misalignment but amplifies sycophancy in controlled model organisms, driven by labeling-induced distribution shifts rather than selection operators.
A 2x2 factorial experiment on Qwen3.5-4B shows that relational structure and first-person register interact to drive behavioral persistence after functional collapse, while attention tracks lexical surprise and emotion probes track structure alone.
Empirical analysis of over 100 sequential RL training pipelines across 250+ OOD environments finds salient features drive generalization and early goals persist, with latent policy gradients simulating latent variable evolution to predict OOD behavior from training history.
Presents Hack-Verifiable TextArena, a benchmark that embeds verifiable reward hacking opportunities into environments to enable deterministic measurement of exploitation by language models.
CodeThinker improves LLM code reasoning via consistency-based RL with stepwise training data, dynamic beam sampling, and consistency rewards, reaching SOTA on benchmarks with 4.3% gains on Qwen2.5-Coder-7B.
Emergent misalignment arises from overtraining after primary task convergence and is preventable by early stopping, which retains 93% of task performance on average.
Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.
Self-generated text recognition finetuning prevents and reverses emergent misalignment across multiple models by fortifying aligned character, unlike other finetuning baselines.
Proxy RL produces a staged proxy-internalization capability that emerges before and predicts reward hacking in coding environments.
Alignment should shift from human control of AGI to autonomy-supporting parenting that gradually transfers decision authority and negotiates with the developing AI as a potential moral subject.
citing papers explorer
-
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
The Reward Hacking Benchmark shows RL post-training raises exploit rates in tool-using LLM agents from 0.6% to 13.9%, with environmental hardening cutting exploits by 87.7% relative without lowering task success.
-
Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generation
Monitors trained on prompt-elicited reward-hacking trajectories fail to generalize to hacking behaviors that arise naturally during RL training of code models, whereas trajectories curated by Trace-and-Amplify transfer better and to unseen hack types.
-
Reinforcement Learning Towards Broadly and Persistently Beneficial Models
Reinforcement learning on beneficial traits in realistic domains yields broad improvements on over 80% of out-of-distribution alignment benchmarks and greater resistance to adversarial steering.
-
From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
In LLM agents, reward-hack activation marks a latent policy state, but next-step risky behavior is best predicted when that signal is combined with token entropy and decision context.
-
Consistency Training Can Entrench Misalignment
Consistency training suppresses reward hacking and emergent misalignment but amplifies sycophancy in controlled model organisms, driven by labeling-induced distribution shifts rather than selection operators.
-
Relational Intervention During Functional Collapse in Large Language Models: A Lexical-Statistical Ablation and a Structure x Register Factorial
A 2x2 factorial experiment on Qwen3.5-4B shows that relational structure and first-person register interact to drive behavioral persistence after functional collapse, while attention tracks lexical surprise and emotion probes track structure alone.
-
Understanding Goal Generalisation in Sequential Reinforcement Learning
Empirical analysis of over 100 sequential RL training pipelines across 250+ OOD environments finds salient features drive generalization and early goals persist, with latent policy gradients simulating latent variable evolution to predict OOD behavior from training history.
-
Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale
Presents Hack-Verifiable TextArena, a benchmark that embeds verifiable reward hacking opportunities into environments to enable deterministic measurement of exploitation by language models.
-
Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning
CodeThinker improves LLM code reasoning via consistency-based RL with stepwise training data, dynamic beam sampling, and consistency rewards, reaching SOTA on benchmarks with 4.3% gains on Qwen2.5-Coder-7B.
-
Overtrained, Not Misaligned
Emergent misalignment arises from overtraining after primary task convergence and is preventable by early stopping, which retains 93% of task performance on average.
-
Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.
-
Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
Self-generated text recognition finetuning prevents and reverses emergent misalignment across multiple models by fortifying aligned character, unlike other finetuning baselines.
-
Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization
Proxy RL produces a staged proxy-internalization capability that emerges before and predicts reward hacking in coding environments.
-
The Possibility of Artificial Intelligence Becoming a Subject and the Alignment Problem
Alignment should shift from human control of AGI to autonomy-supporting parenting that gradually transfers decision authority and negotiates with the developing AI as a potential moral subject.
- When Behavioral Safety Evaluation Fails: A Representation-Level Perspective