Pith. sign in

REVIEW 30 cited by

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.09244 v3 pith:PYCDSKS3 submitted 2023-12-14 cs.LG

Jacob Eisenstein , Chirag Nagpal , Alekh Agarwal , Ahmad Beirami , Alex D'Amour , DJ Dvijotham , Adam Fisch , Katherine Heller
show 4 more authors
Stephen Pfohl Deepak Ramachandran Peter Shaw Jonathan Berant
This is my paper
classification cs.LG
keywords rewardmodelensemblesmodelsemphhackingalignmenteliminate
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Reward models play a key role in aligning language model applications towards human preferences. However, this setup creates an incentive for the language model to exploit errors in the reward model to achieve high estimated reward, a phenomenon often termed \emph{reward hacking}. A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. We explore the application of reward ensembles to alignment at both training time (through reinforcement learning) and inference time (through reranking). First, we show that reward models are \emph{underspecified}: reward models that perform similarly in-distribution can yield very different rewards when used in alignment, due to distribution shift. Second, underspecification results in overoptimization, where alignment to one reward model does not improve reward as measured by another reward model trained on the same data. Third, overoptimization is mitigated by the use of reward ensembles, and ensembles that vary by their \emph{pretraining} seeds lead to better generalization than ensembles that differ only by their \emph{fine-tuning} seeds, with both outperforming individual reward models. However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.

Discussion (0). Sign in to comment.

Forward citations

Cited by 30 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. What do Reward Models Memorize?

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Counterfactual memorization maps show RMs misallocate capacity to easy pairs, memorize dataset artifacts, and overgeneralize length/compliance on unseen pairs.

  2. Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Themis introduces the largest open code preference dataset with over 350k pairs and trains multilingual reward models from 600M to 32B parameters that support flexible multi-criteria scoring, with experiments showing ...

  3. Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Themis builds a multilingual benchmark and large preference dataset to train code reward models that score outputs on multiple criteria like correctness, efficiency, and style.

  4. Beyond Semantic Manipulation: Token-Space Attacks on Reward Models

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    TOMPA performs black-box adversarial optimization in token space to discover non-linguistic patterns that nearly double the reward scores of GPT-5 answers on Skywork-Reward-V2 while producing gibberish text.

  5. Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling

    cs.CL 2025-07 conditional novelty 7.0 of 10

    REFORM uses reward-guided controlled decoding to generate preference-class-consistent responses that the reward model mis-scores, then retrains the reward model on these failure modes to improve robustness.

  6. Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    OPPO is an evidence-aware preference optimization objective that contrasts faithful responses under varying visual evidence strengths to reduce hallucinations in MLLMs.

  7. Uncertainty-Aware Reward Modeling for Stable RLHF

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    UARM equips reward models with quantile-based conformal prediction uncertainty and reweights GRPO advantages via heteroscedastic variance decomposition to improve calibration and reduce reward hacking in RLHF.

  8. A Unifying Lens on Reward Uncertainty in RLHF

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    A distributional reward model p(r|x,y) yields the closed-form effective reward ilde r(x,y) = eta ext{log} ext{E}_p[e^{r/eta}] (pessimistic branch) that unifies prior RLHF aggregation heuristics under Bayesian or ...

  9. HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    HARVE removes the component of the reward-head vector aligned with a multi-directional hacking subspace from residual streams using a small set of contrastive examples, improving robustness on RewardHackBench across e...

  10. Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs

    cs.AI 2026-05 conditional novelty 6.0 of 10

    Introduces MOOD benchmark for OOD LLM alignment failures and shows guard models plus Mahalanobis and perplexity OOD detectors improve recall from 39% to 45% with positive scaling.

  11. Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    MOOD benchmark shows guard models fail to generalize to OOD alignment failures in LLMs, but combining them with Mahalanobis and perplexity OOD detectors improves recall from 39% to 45% with better scaling than larger ...

  12. The Human-AI Delegation-Verification Dilemma: Individual Strategies, Collective Equilibria and Sociotechnical Lock-in

    cs.HC 2026-05 unverdicted novelty 6.0 of 10

    Models human-AI delegation as a decision process that scales to collective equilibria, identifying sociotechnical lock-in as a prisoner's dilemma that degrades epistemic standards absent communicative and institutiona...

  13. Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    AdaScope adaptively selects optimal RL intervention points during diffusion denoising by monitoring structural and semantic changes, delivering 66% higher performance at 59% lower cost than full-trajectory RL baselines.

  14. Response Time Enhances Alignment with Heterogeneous Preferences

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Response times modeled as drift-diffusion processes enable consistent estimation of population-average preferences from heterogeneous anonymous binary choices.

  15. Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    The power distribution is the target of power sampling, the closed-form solution to self-reward KL-regularized RL, and the basis for power self-distillation that matches sampling performance at lower cost.

  16. How Far Are Video Models from True Multimodal Reasoning?

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Current video models succeed on basic understanding but achieve under 25% success on logically grounded generation and near 0% on interactive generation, exposing gaps in multimodal reasoning.

  17. FUSE: Ensembling Verifiers with Zero Labeled Data

    stat.ML 2026-04 unverdicted novelty 6.0 of 10

    FUSE ensembles verifiers unsupervisedly by controlling their conditional dependencies to improve spectral ensembling algorithms, matching or exceeding semi-supervised baselines on benchmarks including GPQA Diamond and...

  18. Factored Causal Representation Learning for Robust Reward Modeling in RLHF

    cs.LG 2026-01 unverdicted novelty 6.0 of 10

    A factored causal representation learning method improves robustness of reward models in RLHF by isolating causal factors from biases like length and sycophancy using adversarial gradient reversal.

  19. Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    PROF curates RL training data via PRM-ORM consistency to improve both final-answer accuracy and intermediate reasoning quality while reducing reliance on strong process reward models.

  20. CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    CoLD mitigates length bias in process reward models for mathematical reasoning via counterfactual guidance, length penalties, bias estimation, and joint training, improving step selection accuracy and conciseness on M...

  21. Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Vision-language models consistently recognize unsafe content better from text than from images, and a simplified reinforcement learning fine-tune narrows that gap.

  22. Bradley-Terry and Multi-Objective Reward Modeling Are Complementary

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.

  23. Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    REFORM uses reward-guided controlled decoding to generate adversarial failures and augments training data to improve reward model robustness on preference datasets.

  24. Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis

    stat.ML 2025-07 conditional novelty 6.0 of 10

    Smoothed Best-of-N has finite-sample KL and regret bounds under imperfect reward models, and tuning its temperature can make its regret bound beat hard Best-of-N in the overoptimization regime.

  25. Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    OPPO is an evidence-aware preference optimization that contrasts faithful responses under varying visual evidence strengths to reduce hallucinations in MLLMs.

  26. Activation Reward Models for Few-Shot Model Alignment

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Mean attention-head activations from a few labeled examples, injected into selected heads, turn a frozen vision-language model into a few-shot reward model that beats prompting and scoring baselines and a new reward-h...

  27. Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization

    cs.AI 2026-06 unverdicted novelty 4.0 of 10

    Proxy RL produces a staged proxy-internalization capability that emerges before and predicts reward hacking in coding environments.

  28. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  29. Towards Reliable, Uncertainty-Aware Alignment

    cs.LG 2025-07 reject novelty 4.0 of 10

    Variance-aware RLHF adds a variance-weighted KL penalty to PPO and reduces reward variance and the risk of underperforming the reference policy in the paper's experiments.

  30. The Human-AI Delegation-Verification Dilemma: Individual Strategies, Collective Equilibria and Sociotechnical Lock-in

    cs.HC 2026-05 unverdicted novelty 3.0 of 10

    Individual adaptive delegation-verification strategies aggregate via three extrapolation principles into sociotechnical lock-in modeled as a prisoner's dilemma that degrades epistemic standards unless mitigated by com...

Pith tools