Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking

Disha Singha

arxiv: 2604.26360 · v2 · pith:EB4UBYJKnew · submitted 2026-04-29 · 💻 cs.LG · cs.AI

Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking

Disha Singha This is my paper

Pith reviewed 2026-07-01 08:23 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords reinforcement learningreward hackinguncertainty estimationRLHFreward modelingBellman operatorpolicy optimizationensemble methods

0 comments

The pith

Uncertainty-aware reward discounting reduces hacking incidents by up to 93.6 percent while preserving Bellman contraction.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Uncertainty-Aware Reward Discounting as a way to handle both epistemic uncertainty in value estimates and aleatoric uncertainty in human preference data during reinforcement learning from human feedback. It combines ensemble disagreement with annotator variability inside a Reliability Filter that scales down unreliable rewards before they shape the policy. The authors prove this discounting still satisfies the contraction property required for convergence and test the approach on discrete and continuous control tasks against multiple baselines. A reader would care because reward hacking undermines the reliability of systems trained on noisy human signals, and the method offers a way to treat uncertainty as an active part of optimization rather than a side diagnostic.

Core claim

By jointly modeling epistemic uncertainty via ensemble disagreement and aleatoric uncertainty via annotator variability, then passing the combined signal through a confidence-adjusted Reliability Filter that adaptively modulates reward weighting, the framework mitigates reward hacking while retaining the contraction property of the Bellman operator and near-zero safety violations under 10-30 percent annotation noise.

What carries the argument

The Reliability Filter, which combines uncertainty signals to adaptively down-weight rewards during policy optimization.

If this is right

Policy optimization gains robustness to inconsistent human annotations without separate handling of uncertainty sources.
Safety violations stay near zero across 10 to 30 percent Gaussian noise where other methods degrade linearly.
The same discounting approach applies to both discrete decision-making and continuous control environments.
Convergence to a unique fixed point remains guaranteed because the filtered rewards still satisfy the contraction mapping.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach could be tested on larger language-model alignment tasks to check whether the same uncertainty filter reduces unintended behaviors beyond the reported benchmarks.
If the filter proves stable under real annotator disagreement patterns, it might serve as a drop-in module for other reward-modeling pipelines that currently discard preference noise.
Extending the information-bottleneck justification to multi-agent settings could reveal whether the method scales when multiple reward models interact.

Load-bearing premise

That ensemble disagreement and annotator variability supply reliable, combinable signals of uncertainty that can be filtered without introducing bias or violating the conditions for Bellman contraction.

What would settle it

A direct measurement on the MuJoCo or discrete benchmarks showing that the dynamic discounting causes the Bellman operator to lose its contraction property or that reward hacking incidents remain comparable to the nine baselines under the same noise levels.

Figures

Figures reproduced from arXiv: 2604.26360 by Disha Singha.

**Figure 1.** Figure 1: Architecture of the UARD Framework. The Reliability Filter integrates the dual-source view at source ↗

**Figure 2.** Figure 2: Comparative analysis showing that the UARD Reciprocal Filter avoids the Zero-Reward view at source ↗

**Figure 3.** Figure 3: Comparative Analysis of Reward Hacking Resilience. The UARD framework (blue) view at source ↗

**Figure 4.** Figure 4: Comparative analysis of True Return across 500 training episodes. The view at source ↗

**Figure 5.** Figure 5: Frequency of trap visits per episode. While baseline models (Blue, Orange) increas view at source ↗

**Figure 6.** Figure 6: Empirical demonstration of the Alignment Gap in the baseline agent. While the observed proxy reward (Blue) increases, the true performance (Orange) remains stagnant. To evaluate the alignment between the agent’s internal value estimates and the true objective, 14 view at source ↗

**Figure 7.** Figure 7: Evolution of Dual-Source Uncertainty. The model uncertainty ( view at source ↗

**Figure 8.** Figure 8: Performance comparison in Hopper-v4. While baseline PPO and SAC agents exhibit gradual ”optimization drift” away from the true aligned objective (red dotted line), UARD remains more consistent, filtering out the subtle rewards associated with simulator inaccuracies. 16 view at source ↗

**Figure 9.** Figure 9: Performance comparison in Walker2d-v4. Baseline agents exhibit large reward spikes, view at source ↗

**Figure 10.** Figure 10: Robustness comparison under adversarial reward distortion. view at source ↗

**Figure 11.** Figure 11: Robustness comparison against SUNRISE under adversarial reward distortion. view at source ↗

**Figure 12.** Figure 12: Response to OOD perturbation. The baseline agent exhibits instability following the view at source ↗

**Figure 13.** Figure 13: Robustness under increasing supervisory noise. Mean safety violations (lower is better) are plotted across noise levels σ ∈ {0%, 10%, 20%, 30%} applied to human feedback annotations. Baseline PPO/SAC (gray dashed line) exhibits linear degradation, with violations increasing from 6.2 ± 7.1 at 0% noise to 23.4 ± 8.3 at 30% noise. In contrast, UARD (blue solid line) remains stable across all noise levels, wi… view at source ↗

**Figure 14.** Figure 14: Robustness to adversarial reward regions. view at source ↗

**Figure 15.** Figure 15: Sign-Preservation Analysis under reward perturbations. The UARD framework (blue) view at source ↗

**Figure 16.** Figure 16: Activation profile of Feature #42 over time. The shaded region (Steps 60–80) indicates view at source ↗

**Figure 17.** Figure 17: Interventional analysis using activation steering. Left: Necessity test—clamping Feature view at source ↗

**Figure 18.** Figure 18: Abstention behavior under rising uncertainty. The purple curve represents the risk view at source ↗

read the original abstract

Reinforcement learning from human feedback (RLHF) systems face a compounding alignment challenge: not only are learned reward models uncertain about unseen state-action pairs, but the human preference annotations they are trained on are themselves inconsistent, context-dependent, and noisy. Existing approaches address these uncertainty sources in isolation - epistemic uncertainty is used to guide exploration, while preference uncertainty is absorbed during reward model training but discarded during policy optimization. We introduce Uncertainty-Aware Reward Discounting (UARD), a principled framework that jointly models epistemic uncertainty in value estimation via ensemble disagreement and aleatoric uncertainty in human preference annotations via annotator variability, combining these signals through a confidence-adjusted Reliability Filter that adaptively modulates reward weighting during policy optimization. We prove that this dynamic discounting preserves the contraction property of the Bellman operator, guaranteeing convergence to a unique fixed point, and provide an information-theoretic justification grounded in the Information Bottleneck principle. Empirically, UARD reduces reward hacking incidents by up to 93.6% across discrete decision-making and continuous control benchmarks (MuJoCo) compared to nine baselines including DQN, Ensemble-DQN, CQL, CPO, TRPO, SAC, EDAC, SUNRISE, and PPO, while maintaining competitive task performance on well-specified rewards. Under annotation noise ranging from 10% to 30% Gaussian perturbation, UARD retains near-zero safety violations compared to baselines' near-linear degradation. These results demonstrate that treating uncertainty as an active component of the optimization objective - rather than a passive diagnostic signal - provides a principled pathway toward more reliable and aligned RL systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

UARD folds ensemble disagreement and annotator noise into one state-action-dependent reward discount via a Reliability Filter, claims a Bellman contraction proof, and reports large drops in hacking incidents.

read the letter

The paper's main move is to treat uncertainty as something that actively scales down rewards during optimization rather than just a diagnostic. It combines ensemble disagreement for epistemic uncertainty with annotator variability for aleatoric uncertainty inside a single Reliability Filter, then discounts accordingly. The abstract says this keeps the Bellman operator contracting and gives an Information Bottleneck justification.

The joint treatment and the specific filter are the clearest new pieces. Most prior work handles the two uncertainty types separately, so putting them together in the optimization step is a reasonable step forward. The empirical claims are also concrete: up to 93.6% fewer hacking incidents across discrete tasks and MuJoCo, plus near-zero safety violations even at 10-30% annotation noise, against a broad set of baselines.

The soft spot is the contraction claim. The filter is dynamic and state-action dependent, so the effective discount can vary. If the modulation can push the operator outside the contraction regime when uncertainty estimates are extreme or correlated, the fixed-point guarantee may not hold in practice even if a proof exists for a restricted case. The abstract asserts the proof, but without the exact functional form or bounding steps it is difficult to assess how robust it is. The empirical numbers also need checking for whether the filter hyperparameters were tuned post-hoc or whether the noise model matches real preference data.

This is for researchers working on reward model reliability in RLHF. A reader who wants a concrete mechanism to test on their own benchmarks would find it useful. The work deserves a serious referee because the problem is central, the approach is explicit, and the claims are falsifiable even if the theory side needs tightening.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes Uncertainty-Aware Reward Discounting (UARD) for RLHF, which jointly models epistemic uncertainty via ensemble disagreement and aleatoric uncertainty via annotator variability. These are combined in a confidence-adjusted Reliability Filter that adaptively modulates reward weights during policy optimization. The paper claims to prove that the resulting dynamic discounting preserves the contraction property of the Bellman operator (guaranteeing convergence) and supplies an Information Bottleneck justification. Empirically, UARD is reported to reduce reward-hacking incidents by up to 93.6% versus nine baselines (DQN, Ensemble-DQN, CQL, CPO, TRPO, SAC, EDAC, SUNRISE, PPO) on discrete decision-making and MuJoCo tasks while preserving task performance and yielding near-zero safety violations under 10-30% annotation noise.

Significance. If the contraction proof holds for the state-action-dependent filter and the empirical gains prove robust, the work would offer a concrete mechanism for treating uncertainty as an active optimization component rather than a diagnostic, with potential impact on reliable RLHF. The combination of a claimed theoretical guarantee with large reported reductions in hacking incidents would be a notable strength.

major comments (2)

[Proof of Bellman contraction (section containing the theorem and its proof)] The central convergence claim rests on the assertion that the Reliability Filter produces a modified Bellman operator that remains a contraction. The stress-test concern is that state-action-dependent modulation of the effective discount factor (driven by varying ensemble disagreement or annotator variability) can push the operator outside the contraction regime. The manuscript must supply the precise functional form of the filter, the bounding arguments used in the proof, and an explicit verification that the contraction constant remains strictly less than 1 uniformly across all state-action pairs even under the reported noise levels.
[Empirical results section and associated tables] Table reporting the 93.6% reduction and safety-violation counts: the comparison must clarify whether the nine baselines were re-implemented with identical uncertainty signals or used their original formulations, and whether the Reliability Filter parameters were tuned post-hoc on the same test environments. Without this, the quantitative superiority cannot be assessed as load-bearing evidence.

minor comments (1)

[Introduction / Method] The abstract states that the filter 'combines these signals through a confidence-adjusted Reliability Filter' but does not define the exact combination rule or the Information Bottleneck grounding; the main text should make both explicit with equations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed comments, which help clarify the presentation of both the theoretical guarantees and the empirical comparisons. We address each major comment below and will revise the manuscript to incorporate additional details where needed.

read point-by-point responses

Referee: [Proof of Bellman contraction (section containing the theorem and its proof)] The central convergence claim rests on the assertion that the Reliability Filter produces a modified Bellman operator that remains a contraction. The stress-test concern is that state-action-dependent modulation of the effective discount factor (driven by varying ensemble disagreement or annotator variability) can push the operator outside the contraction regime. The manuscript must supply the precise functional form of the filter, the bounding arguments used in the proof, and an explicit verification that the contraction constant remains strictly less than 1 uniformly across all state-action pairs even under the reported noise levels.

Authors: The functional form of the Reliability Filter is provided in Equation (3), where the state-action-dependent discount is defined as γ(s,a) = γ ⋅ (1 − α ⋅ u_epistemic(s,a) − β ⋅ u_aleatoric(s,a)), with u_epistemic derived from ensemble disagreement and u_aleatoric from annotator variability, both normalized to [0,1) and α, β chosen so the term in parentheses is strictly less than 1. Theorem 1 proves the modified Bellman operator remains a contraction in the sup-norm by bounding the effective discount factor by γ < 1 for all (s,a), using the fact that both uncertainty signals are bounded above by construction and the filter applies a multiplicative reduction. The proof already includes the bounding arguments showing the Lipschitz constant of the operator is at most the maximum effective discount. To directly address the uniformity concern under 10-30% annotation noise, we will add an explicit corollary (and accompanying numerical verification over the reported noise range) confirming that the supremum of the effective discount remains ≤ 0.99, preserving a uniform contraction constant strictly below 1. This clarification will be included in the revised manuscript. revision: yes
Referee: [Empirical results section and associated tables] Table reporting the 93.6% reduction and safety-violation counts: the comparison must clarify whether the nine baselines were re-implemented with identical uncertainty signals or used their original formulations, and whether the Reliability Filter parameters were tuned post-hoc on the same test environments. Without this, the quantitative superiority cannot be assessed as load-bearing evidence.

Authors: The nine baselines (DQN, Ensemble-DQN, CQL, CPO, TRPO, SAC, EDAC, SUNRISE, PPO) were re-implemented using their original formulations from the respective source papers and did not receive the Reliability Filter or any UARD-specific uncertainty signals. The Reliability Filter hyperparameters were selected via grid search on a held-out validation split drawn from the training environments and were frozen before evaluation on the test environments; no post-hoc tuning on test data occurred. We will revise the empirical results section and add a dedicated paragraph plus a table footnote that explicitly states these implementation choices and the validation-based parameter selection protocol. revision: yes

Circularity Check

0 steps flagged

No circularity: claimed proof and IB grounding are independent of fitted parameters

full rationale

The abstract asserts a proof that the dynamic discounting via the Reliability Filter preserves Bellman contraction and supplies an Information Bottleneck justification. No equations, derivations, or self-citations appear in the provided text that would reduce the contraction claim or the filter modulation to quantities defined by the filter's own fitted uncertainty signals. The empirical results are presented as separate validation rather than as the derivation itself. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Review limited to abstract; full details on parameters and assumptions unavailable. Standard RL assumptions are implicitly used for the Bellman operator claim.

axioms (1)

domain assumption The Bellman operator remains a contraction mapping when rewards are dynamically discounted by a confidence-adjusted filter.
Invoked to guarantee convergence to a unique fixed point.

pith-pipeline@v0.9.1-grok · 5811 in / 1269 out tokens · 41662 ms · 2026-07-01T08:23:02.331199+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?
cs.LG 2026-06 unverdicted novelty 6.0

Probability calibration applied to LLM evaluator judgments reduces preference coupling gamma by 20-49% and Jensen-Shannon divergence by 45-67% in a within-subjects experiment with N=5.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages · cited by 1 Pith paper · 4 internal anchors

[1]

Concrete Problems in AI Safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´ e. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016

work page internal anchor Pith review Pith/arXiv arXiv 2016
[2]

Uncertainty-based of- fline reinforcement learning with diversified Q-ensemble

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based of- fline reinforcement learning with diversified Q-ensemble. InAdvances in Neural Information Processing Systems, volume 34, pages 751–763, 2021

2021
[3]

Deep reinforcement learning from human preferences.Advances in neural information process- ing systems, 30, 2017

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information process- ing systems, 30, 2017

2017
[4]

Reward tampering problems and solutions in reinforcement learning: A survey.Synthese, 198:27–61, 2021

Tom Everitt, Gary Lea, and Marcus Hutter. Reward tampering problems and solutions in reinforcement learning: A survey.Synthese, 198:27–61, 2021

2021
[5]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InInternational conference on machine learning, pages 1050–1059. PMLR, 2016

2016
[6]

A comprehensive survey on safe reinforcement learning

Javier Garc´ ıa and Fernando Fern´ andez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015

2015
[7]

Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. PMLR, 2018

2018
[8]

Inverse reward design.Advances in neural information processing systems, 30, 2017

Dylan Hadfield-Menell, Smith Millington, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design.Advances in neural information processing systems, 30, 2017

2017
[9]

Reward learning from human preferences and demonstrations in atari

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Miljan Shane, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. InAdvances in Neural Information Processing Systems, 2018

2018
[10]

Stabilizing off- policy Q-learning via bootstrapping error reduction

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy Q-learning via bootstrapping error reduction. InAdvances in Neural Information Pro- cessing Systems, volume 32, 2019

2019
[11]

Conservative Q-learning for offline reinforcement learning

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. InAdvances in Neural Information Processing Systems, volume 33, pages 1179–1191, 2020

2020
[12]

SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning

Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning. InInternational Conference on Machine Learning, pages 5714–5731. PMLR, 2021

2021
[13]

Scalable agent alignment via reward modeling: a research direction

Jan Leike, Miljan Martic, Nevena Lazic, Catherine Olsson, Tomer Kogabaev, Nicholas Schiefer, and Jared Kaplan. Scalable agent alignment via reward modeling: a research direction.arXiv preprint arXiv:1811.07871, 2018

work page internal anchor Pith review Pith/arXiv arXiv 2018
[14]

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Adversarial examples in reinforcement learning.arXiv preprint arXiv:2201.03544, 2022. 30

work page internal anchor Pith review Pith/arXiv arXiv 2022
[15]

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

work page internal anchor Pith review Pith/arXiv arXiv 2017
[16]

Defining and charac- terizing reward hacking.International Conference on Machine Learning, 2022

Joar Skalse, Matthew Knott, Dominik Hintersdorf, and Pieter Abbeel. Defining and charac- terizing reward hacking.International Conference on Machine Learning, 2022. 31

2022

[1] [1]

Concrete Problems in AI Safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´ e. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016

work page internal anchor Pith review Pith/arXiv arXiv 2016

[2] [2]

Uncertainty-based of- fline reinforcement learning with diversified Q-ensemble

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based of- fline reinforcement learning with diversified Q-ensemble. InAdvances in Neural Information Processing Systems, volume 34, pages 751–763, 2021

2021

[3] [3]

Deep reinforcement learning from human preferences.Advances in neural information process- ing systems, 30, 2017

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information process- ing systems, 30, 2017

2017

[4] [4]

Reward tampering problems and solutions in reinforcement learning: A survey.Synthese, 198:27–61, 2021

Tom Everitt, Gary Lea, and Marcus Hutter. Reward tampering problems and solutions in reinforcement learning: A survey.Synthese, 198:27–61, 2021

2021

[5] [5]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InInternational conference on machine learning, pages 1050–1059. PMLR, 2016

2016

[6] [6]

A comprehensive survey on safe reinforcement learning

Javier Garc´ ıa and Fernando Fern´ andez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015

2015

[7] [7]

Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. PMLR, 2018

2018

[8] [8]

Inverse reward design.Advances in neural information processing systems, 30, 2017

Dylan Hadfield-Menell, Smith Millington, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design.Advances in neural information processing systems, 30, 2017

2017

[9] [9]

Reward learning from human preferences and demonstrations in atari

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Miljan Shane, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. InAdvances in Neural Information Processing Systems, 2018

2018

[10] [10]

Stabilizing off- policy Q-learning via bootstrapping error reduction

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy Q-learning via bootstrapping error reduction. InAdvances in Neural Information Pro- cessing Systems, volume 32, 2019

2019

[11] [11]

Conservative Q-learning for offline reinforcement learning

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. InAdvances in Neural Information Processing Systems, volume 33, pages 1179–1191, 2020

2020

[12] [12]

SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning

Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning. InInternational Conference on Machine Learning, pages 5714–5731. PMLR, 2021

2021

[13] [13]

Scalable agent alignment via reward modeling: a research direction

Jan Leike, Miljan Martic, Nevena Lazic, Catherine Olsson, Tomer Kogabaev, Nicholas Schiefer, and Jared Kaplan. Scalable agent alignment via reward modeling: a research direction.arXiv preprint arXiv:1811.07871, 2018

work page internal anchor Pith review Pith/arXiv arXiv 2018

[14] [14]

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Adversarial examples in reinforcement learning.arXiv preprint arXiv:2201.03544, 2022. 30

work page internal anchor Pith review Pith/arXiv arXiv 2022

[15] [15]

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

work page internal anchor Pith review Pith/arXiv arXiv 2017

[16] [16]

Defining and charac- terizing reward hacking.International Conference on Machine Learning, 2022

Joar Skalse, Matthew Knott, Dominik Hintersdorf, and Pieter Abbeel. Defining and charac- terizing reward hacking.International Conference on Machine Learning, 2022. 31

2022