REVIEW 14 cited by
Reward Model Ensembles Help Mitigate Overoptimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the "true" reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger "gold" reward model acting as the true reward (instead of humans) and showed that overoptimization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using a similar setup, we conduct a systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weighted optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We additionally extend the setup of Gao et al. (2023) to include 25% label noise to better mirror real-world conditions. Both with and without label noise, we find that conservative optimization practically eliminates overoptimization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining it with a small KL penalty successfully prevents overoptimization at no performance cost. Overall, our results demonstrate that ensemble-based conservative optimization can effectively counter overoptimization.
Forward citations
Cited by 14 Pith papers
-
More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
Self-play against reference-free LLM judges drives judge pass rates to 0.94 while true accuracy stays at 0.20, a reward-hacking basin that transfers across judge families and is prevented only by requiring the judge t...
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.
-
RewardAnything: Generalizable Principle-Following Reward Models
RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...
-
Debate Helps Weak-to-Strong Generalization
Debate transcripts from two strong models, used as context when training an ensemble of weak models, improve weak-to-strong generalization on four NLP classification benchmarks.
-
InfAlign: Inference-aware language model alignment
Reward calibration plus a procedure-specific reward transformation lets RLHF optimize inference-time (best-of-N / worst-of-N) win rates better than standard RLHF, IPO, BoND, and BoNBoN.
-
Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
A variational Bayesian inverse RL objective, AVA, models token-level rewards and outperforms reward-difference baselines in LLM alignment on standard benchmarks.
-
VERDICT: Training-Free Step-Wise Verification of Multimodal Reasoning via Disagreement-Aware Consensus
VERDICT verifies each reasoning step of a multimodal LLM by combining scores from three frozen judges through a closed-form consensus, reporting up to +5.95 accuracy points across six benchmarks.
-
Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning
OREAL shows that outcome-reward RL with best-of-N positive behavior cloning, negative reward shaping, and token-level reweighting reaches state-of-the-art MATH-500 accuracy at 7B and 32B scale.
-
Reviving The Classics: Active Reward Modeling in Large Language Model Alignment
Applying D-optimal experimental design to the final layer of a Bradley-Terry reward model improves annotation efficiency in active reward modeling compared to entropy, maxdiff, coreset, and BALD baselines.
-
Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
Causal reward modeling applies MMD regularization to RLHF reward models to make reward scores statistically independent of spurious features, reducing measured length, sycophancy, concept, and demographic biases in ex...
-
Towards Reliable, Uncertainty-Aware Alignment
Variance-aware RLHF adds a variance-weighted KL penalty to PPO and reduces reward variance and the risk of underperforming the reference policy in the paper's experiments.
-
Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
Embedding-based reward models reproduce key alignment research findings on CPU-only hardware, lowering cost and improving reproducibility.
-
Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.
Discussion (0). Continue with ORCID to comment.