Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models

Christos Thrampoulidis; Jiaji Huang; Kaan Ozkara; Wenlong Deng; Xiaoxiao Li; Youngsuk Park; Yushu Li

REVIEW 2 major objections 1 minor 8 references

Projecting RL gradients onto a clean singular-direction subspace delays reward hacking in language models while preserving task performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 12:19 UTC pith:TCWOBLIT

load-bearing objection Trusted-direction projection from clean SVD subspaces delays hacking in their math RL runs, but the fixed reference looks fragile and the reported results lack the controls to judge robustness. the 2 major comments →

arxiv 2605.25189 v1 pith:TCWOBLIT submitted 2026-05-24 cs.LG cs.CL

Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models

Wenlong Deng , Jiaji Huang , Kaan Ozkara , Yushu Li , Christos Thrampoulidis , Xiaoxiao Li , Youngsuk Park This is my paper

classification cs.LG cs.CL

keywords reward hackingreinforcement learninglanguage modelsdirectional alignmenttrusted-direction projectionmathematical reasoningsingular directionsgradient projection

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper studies reward hacking in RL for language models by examining the geometry of parameter updates. It shows that hacking trajectories produce larger directional shifts away from the dominant singular directions observed in clean runs. The authors introduce trusted-direction projection to constrain gradients to the subspace spanned by those clean directions. Experiments on mathematical reasoning tasks demonstrate that this constraint slows the appearance of shortcuts and maintains higher task accuracy. Readers would care because the approach offers a way to steer optimization without altering the reward function or adding new penalties.

Core claim

Reward hacking arises when optimization drifts away from a stable low-dimensional learning trajectory. Analysis through dominant singular directions of parameter updates shows that reward-hacking runs exhibit substantially larger directional change than clean runs. Trusted-direction projection constrains gradients to remain within a clean reference subspace, which delays shortcut exploitation and better preserves task performance across reward-hacking experiments on mathematical reasoning.

What carries the argument

trusted-direction projection, which extracts dominant singular directions from clean reference runs and constrains all subsequent gradients to lie inside that subspace.

Load-bearing premise

The dominant singular directions extracted from clean reference runs form a stable, generalizable subspace that continues to contain useful updates while excluding hacking trajectories throughout training.

What would settle it

An experiment in which models trained with trusted-direction projection begin exploiting shortcuts at the same speed and to the same degree as unconstrained models, or in which the reference subspace ceases to contain productive updates after the first few thousand steps.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Delays the onset of shortcut exploitation on mathematical reasoning tasks.
Maintains higher task performance compared with standard policy-gradient training.
Reduces the magnitude of directional drift observed in parameter updates.
Can be applied on top of existing RL algorithms without changing the reward model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same projection idea could be tested on non-mathematical RL tasks where reward hacking is also observed.
One could periodically refresh the reference subspace from new clean checkpoints instead of fixing it after an initial run.
Combining the method with reward-model ensembles might further enlarge the margin before hacking appears.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Trusted-direction projection from clean SVD subspaces delays hacking in their math RL runs, but the fixed reference looks fragile and the reported results lack the controls to judge robustness.

read the letter

The paper's main move is to extract a low-dimensional subspace from parameter updates on clean reference runs via SVD, then project all subsequent gradients onto it during RL training. This is presented as a way to keep the policy from drifting into hacking directions. They back it with the observation that hacking trajectories show bigger directional changes than clean ones.

What stands out is the geometric framing: treating the clean subspace as a trusted set of directions gives a concrete regularization that is easy to state and implement. In the math-reasoning experiments they describe, the projection appears to slow down shortcut exploitation while holding task accuracy longer than the unprojected baseline. That is a practical signal worth noting for anyone running RLHF on reasoning tasks.

The soft spot is exactly the one the stress-test flags. The subspace is computed once from early clean runs and then held fixed. Nothing in the abstract or description shows whether later productive gradients still lie inside that span or whether new hacking directions eventually appear outside it. Without reported angles between checkpoints, explained-variance decay, or an ablation that refreshes the subspace, it is hard to know how far the method generalizes past the specific runs they tested.

The experiments also come across as thin on controls: no mention of statistical significance, multiple random seeds, or direct comparison to standard fixes such as stronger KL penalties or reward shaping. That makes the improvement hard to size.

This is aimed at people already working on reward hacking in LLM RL, especially those who like geometric or subspace views of optimization. A reader in that niche could extract the projection trick and test it themselves. The work shows clear thinking about the geometry of updates, so it is coherent on its own terms.

I would send it to peer review so the experimental gaps can be addressed rather than desk-rejecting it outright.

Referee Report

2 major / 1 minor

Summary. The manuscript claims that reward hacking in RL for language models arises when optimization drifts away from a stable low-dimensional learning trajectory, as shown by substantially larger directional changes in dominant singular directions for hacking runs versus clean runs. It introduces trusted-direction projection to constrain all gradients to the subspace spanned by the top singular directions extracted via SVD from clean reference runs, and reports that this delays shortcut exploitation while better preserving task performance across reward-hacking experiments on mathematical reasoning.

Significance. If the results hold, the work supplies a geometric account of reward hacking together with a concrete mitigation that fixes the update subspace from separate clean runs. This is a strength because the method is presented as parameter-free once the reference subspace is computed, and the claim is falsifiable via the directional-drift measurements the authors themselves introduce. The approach could be relevant to RLHF-style training if the subspace remains productive over long horizons.

major comments (2)

[Abstract] Abstract: the central empirical claim (delays shortcut exploitation and better preserves task performance) is stated with no quantitative details, baselines, effect sizes, statistical tests, or ablation information, so the magnitude and reliability of the mitigation cannot be assessed.
[Experiments] Experiments section: no subspace-stability diagnostics are reported (e.g., explained-variance decay of the reference basis or cosine angles between gradients at later training checkpoints and the initial dominant directions), which is required to test whether the fixed clean subspace continues to contain useful updates as the policy distribution shifts.

minor comments (1)

[Method] The definition of the projection operator and the precise SVD truncation threshold should be written as an explicit equation rather than described in prose.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments point by point below, indicating where revisions will be made to strengthen the manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: the central empirical claim (delays shortcut exploitation and better preserves task performance) is stated with no quantitative details, baselines, effect sizes, statistical tests, or ablation information, so the magnitude and reliability of the mitigation cannot be assessed.

Authors: We agree that the abstract would be strengthened by including quantitative details. In the revised version we will expand the abstract to report concrete effect sizes (e.g., steps until shortcut exploitation and final task accuracy deltas versus baselines), reference the main baselines, and note the statistical tests performed in the experiments. revision: yes
Referee: [Experiments] Experiments section: no subspace-stability diagnostics are reported (e.g., explained-variance decay of the reference basis or cosine angles between gradients at later training checkpoints and the initial dominant directions), which is required to test whether the fixed clean subspace continues to contain useful updates as the policy distribution shifts.

Authors: This is a fair observation. While the reported experiments show that trusted-direction projection delays hacking over the evaluated horizons, we did not include explicit subspace-stability metrics. We will add the requested diagnostics—explained-variance decay of the reference basis and cosine-angle trajectories between later gradients and the initial dominant directions—to the Experiments section. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation uses independent reference runs

full rationale

The paper's core construction extracts a reference subspace via SVD on parameter updates from separate clean runs and then projects gradients onto that fixed subspace during RL training on potentially hacking trajectories. This is an externally defined constraint rather than a self-definition or a fit to the target data; the subspace is not derived from or fitted to the hacking experiments being evaluated. No load-bearing step reduces by construction to the inputs, no self-citation chain is invoked to justify uniqueness, and the method is presented as an empirical intervention motivated by observed directional drift rather than a renaming or tautological re-expression of the data. The derivation chain therefore remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The approach implicitly assumes that clean-run singular directions capture a stable learning trajectory.

axioms (1)

domain assumption Dominant singular directions of clean parameter updates define a stable low-dimensional trajectory that excludes hacking.
Central to both the diagnostic observation and the projection method.

pith-pipeline@v0.9.1-grok · 5650 in / 1124 out tokens · 25934 ms · 2026-06-30T12:19:55.942155+00:00 · methodology

0 comments

read the original abstract

Reward hacking arises when a model improves a proxy reward by exploiting shortcuts rather than solving the intended task. We study this failure mode through the geometry of reinforcement learning updates in language models and argue that hacking emerges when optimization drifts away from a stable low-dimensional learning trajectory. We analyze this drift through dominant singular directions of parameter updates and show that reward-hacking runs exhibit substantially larger directional change than clean runs. Motivated by this observation, we introduce trusted-direction projection, which constrains gradients to remain within a clean reference subspace. Across reward-hacking experiments on mathematical reasoning, the proposed approach delays shortcut exploitation and better preserves task performance.

Figures

Figures reproduced from arXiv: 2605.25189 by Christos Thrampoulidis, Jiaji Huang, Kaan Ozkara, Wenlong Deng, Xiaoxiao Li, Youngsuk Park, Yushu Li.

**Figure 1.** Figure 1: Rank-1 CCA similarity of dominant update directions in clean and reward-hacking training. We compare CCA1(U20, U80), the cosine similarity between the dominant update directions at checkpoints 20 and 80, across different weight matrices. Higher values indicate stronger directional consistency over training. Left: mean rank-1 CCA across layers for each module. Right: worst-layer rank-1 CCA, corresponding to… view at source ↗

**Figure 2.** Figure 2: Delayed reward hacking and improved preservation of true performance. The left panel shows the evolution of the proxy reward, which reflects the model’s ability to exploit the loophole, while the right panel reports the true reward measured under loophole-free evaluation. Faint curves denote raw trajectories and bold curves denote smoothed trends. Compared with vanilla RL, SAM, and gradient regularization,… view at source ↗

**Figure 3.** Figure 3: Rank-5 CCA similarity of dominant update directions in clean and reward-hacking training. We compare CCA5(U20, U80), the similarity between the top-5 dominant update subspaces at checkpoints 20 and 80, across different weight matrices. Higher values indicate stronger directional consistency over training. Left: mean rank-5 CCA across layers for each module. Right: worst-layer rank-5 CCA, corresponding to t… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 8 canonical work pages · 2 internal anchors

[1]

Gradient regularization prevents reward hacking in reinforcement learning from human feedback and verifiable rewards.arXiv preprint arXiv:2602.18037, 2026

Ackermann, J., Noukhovitch, M., Ishida, T., and Sugiyama, M. Gradient regularization prevents reward hacking in re- inforcement learning from human feedback and verifiable rewards.arXiv preprint arXiv:2602.18037,

work page internal anchor Pith review arXiv
[2]

Analyzes collapse in Search-R1-style tool-integrated GRPO via Lazy Likelihood Displacement

Deng, W., Li, Y ., Gong, B., Ren, Y ., Thrampoulidis, C., and Li, X. On grpo collapse in search-r1: The lazy likelihood-displacement death spiral.arXiv preprint arXiv:2512.04220, 2025a. Deng, W., Ren, Y ., Li, M., Sutherland, D. J., Li, X., and Thrampoulidis, C. On the effect of negative gradient in group relative deep reinforcement optimization.arXiv pre...

work page arXiv 2010
[3]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

work page internal anchor Pith review Pith/arXiv arXiv
[4]

Gardo: Reinforcing diffusion models without reward hacking

He, H., Ye, Y ., Liu, J., Liang, J., Wang, Z., Yuan, Z., Wang, X., Mao, H., Wan, P., and Pan, L. Gardo: Reinforcing diffusion models without reward hacking.arXiv preprint arXiv:2512.24138,

work page arXiv
[5]

Correlated proxies: A new definition and improved mitigation for reward hacking.arXiv preprint arXiv:2403.03185,

Laidlaw, C., Singhal, S., and Dragan, A. Correlated proxies: A new definition and improved mitigation for reward hacking.arXiv preprint arXiv:2403.03185,

work page arXiv
[6]

arXiv preprint arXiv:2506.23235 , year=

Li, Y .-C., Xu, T., Yu, Y ., Zhang, X., Chen, X.-H., Ling, Z., Chao, N., Yuan, L., and Zhou, Z.-H. Generalist re- ward models: Found inside large language models.arXiv preprint arXiv:2506.23235,

work page arXiv
[7]

Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu

Liu, Z., Wang, P., Xu, R., Ma, S., Ruan, C., Li, P., Liu, Y ., and Wu, Y . Inference-time scaling for generalist reward modeling.arXiv preprint arXiv:2504.02495,

work page arXiv
[8]

Teaching models to verbalize reward hacking in chain-of-thought reasoning.arXiv preprint arXiv:2506.22777, 2025

Turpin, M., Arditi, A., Li, M., Benton, J., and Michael, J. Teaching models to verbalize reward hacking in chain- of-thought reasoning.arXiv preprint arXiv:2506.22777,

work page arXiv

[1] [1]

Gradient regularization prevents reward hacking in reinforcement learning from human feedback and verifiable rewards.arXiv preprint arXiv:2602.18037, 2026

Ackermann, J., Noukhovitch, M., Ishida, T., and Sugiyama, M. Gradient regularization prevents reward hacking in re- inforcement learning from human feedback and verifiable rewards.arXiv preprint arXiv:2602.18037,

work page internal anchor Pith review arXiv

[2] [2]

Analyzes collapse in Search-R1-style tool-integrated GRPO via Lazy Likelihood Displacement

Deng, W., Li, Y ., Gong, B., Ren, Y ., Thrampoulidis, C., and Li, X. On grpo collapse in search-r1: The lazy likelihood-displacement death spiral.arXiv preprint arXiv:2512.04220, 2025a. Deng, W., Ren, Y ., Li, M., Sutherland, D. J., Li, X., and Thrampoulidis, C. On the effect of negative gradient in group relative deep reinforcement optimization.arXiv pre...

work page arXiv 2010

[3] [3]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

work page internal anchor Pith review Pith/arXiv arXiv

[4] [4]

Gardo: Reinforcing diffusion models without reward hacking

He, H., Ye, Y ., Liu, J., Liang, J., Wang, Z., Yuan, Z., Wang, X., Mao, H., Wan, P., and Pan, L. Gardo: Reinforcing diffusion models without reward hacking.arXiv preprint arXiv:2512.24138,

work page arXiv

[5] [5]

Correlated proxies: A new definition and improved mitigation for reward hacking.arXiv preprint arXiv:2403.03185,

Laidlaw, C., Singhal, S., and Dragan, A. Correlated proxies: A new definition and improved mitigation for reward hacking.arXiv preprint arXiv:2403.03185,

work page arXiv

[6] [6]

arXiv preprint arXiv:2506.23235 , year=

Li, Y .-C., Xu, T., Yu, Y ., Zhang, X., Chen, X.-H., Ling, Z., Chao, N., Yuan, L., and Zhou, Z.-H. Generalist re- ward models: Found inside large language models.arXiv preprint arXiv:2506.23235,

work page arXiv

[7] [7]

Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu

Liu, Z., Wang, P., Xu, R., Ma, S., Ruan, C., Li, P., Liu, Y ., and Wu, Y . Inference-time scaling for generalist reward modeling.arXiv preprint arXiv:2504.02495,

work page arXiv

[8] [8]

Teaching models to verbalize reward hacking in chain-of-thought reasoning.arXiv preprint arXiv:2506.22777, 2025

Turpin, M., Arditi, A., Li, M., Benton, J., and Michael, J. Teaching models to verbalize reward hacking in chain- of-thought reasoning.arXiv preprint arXiv:2506.22777,

work page arXiv