pith. the verified trust layer for science. sign in

arxiv: 2210.10760 · v1 · pith:VMR7JXUZnew · submitted 2022-10-19 · 💻 cs.LG · stat.ML

Scaling Laws for Reward Model Overoptimization

Leo Gao , John Schulman , Jacob Hilton This is my paper

Pith reviewed 2026-05-19 08:59 UTC · model grok-4.3

classification 💻 cs.LG stat.ML
keywords reward model overoptimizationscaling lawsreinforcement learning from human feedbackGoodhart's lawproxy reward modelsbest-of-n samplingAI alignment
0
0 comments X p. Extension
Add this Pith Number to your LaTeX paper What is a Pith Number? 
\usepackage{pith}
\pithnumber{VMR7JXUZ}

Prints a linked pith:VMR7JXUZ badge after your title and writes the identifier into PDF metadata. Compiles on arXiv with no extra files. Learn more

The pith

Optimizing too hard against a proxy reward model reduces true performance according to scaling laws that differ by method and model size.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper uses a synthetic setup where a fixed gold-standard reward model generates labels to train proxy reward models. Policies are then optimized against the proxy using either reinforcement learning or best-of-n sampling, and the resulting gold-standard scores are measured. The authors find that the relationship between optimization strength and gold score takes a different functional form for each method, with the coefficients of those forms scaling smoothly as the number of proxy reward model parameters grows. They further show how this relationship changes with reward model dataset size, policy parameter count, and the KL penalty coefficient. These patterns provide a way to anticipate overoptimization effects in larger systems without repeated human preference collection.

Core claim

A fixed gold-standard reward model supplies synthetic preference labels for training proxy reward models. Policies are optimized against the proxy reward via reinforcement learning with KL penalty or via best-of-n sampling. The gold-standard reward achieved as a function of the degree of proxy optimization follows a different functional form for each method. The coefficients in these forms scale smoothly with the number of parameters in the proxy reward model. The relationship is additionally modulated by the size of the reward model training dataset, the number of policy parameters, and the strength of the KL penalty term.

What carries the argument

The functional form relating gold-standard reward to the strength of optimization against the proxy reward model, whose coefficients scale with proxy reward model parameter count, measured separately for reinforcement learning and best-of-n sampling.

If this is right

  • The coefficients describing overoptimization scale smoothly with the number of reward model parameters.
  • Reinforcement learning and best-of-n sampling produce qualitatively different shapes for the gold-reward decline curve.
  • Increasing the KL penalty coefficient in the reinforcement learning objective reduces the severity of overoptimization at a given optimization level.
  • Larger reward model training datasets and larger policy models both alter the scaling coefficients of the overoptimization relationship.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed scaling could let researchers extrapolate overoptimization risks for much larger models from smaller-scale synthetic runs.
  • The synthetic gold-standard setup likely understates the inconsistency and noise that would appear in genuine human preference data.
  • Practitioners could choose between reinforcement learning and best-of-n sampling partly on the basis of which method yields more predictable degradation at scale.

Load-bearing premise

The fixed gold-standard reward model reproduces the overoptimization dynamics that would arise from real, noisy human preferences.

What would settle it

If experiments using actual collected human preferences instead of the synthetic gold-standard model produce different functional forms or non-smooth scaling of coefficients with model size, the reported scaling laws would not generalize.

read the original abstract

In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-$n$ sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper studies reward model overoptimization in RLHF using a synthetic setup: a fixed deterministic gold-standard reward model generates preference labels to train proxy reward models of varying sizes. It measures how the gold reward score changes as a policy is optimized against the proxy via either RL (with KL penalty) or best-of-n sampling. The central empirical finding is that gold-score vs. optimization level follows distinct functional forms for the two optimization methods, and that the coefficients of these forms scale smoothly with the number of proxy reward-model parameters. The authors also examine dependence on reward-model dataset size, policy and reward-model parameter counts, and the KL coefficient.

Significance. If the reported functional forms and scaling relations hold under the stated conditions, the work supplies the first quantitative, controlled measurements of overoptimization trajectories in a closed synthetic loop. This is useful for grounding theoretical discussions of Goodhart's law in RLHF and for calibrating expectations about how proxy size and optimization method affect ground-truth performance. The synthetic design cleanly isolates the effect of proxy imperfection without the cost of human data collection.

major comments (2)
  1. [§3 and §4] §3 (Methods) and §4 (Results): The abstract and main text provide no description of the functional-form fitting procedures, choice of functional families, error-bar computation, data-exclusion rules, or robustness checks used to establish the distinct functional forms and the smooth scaling of coefficients with proxy RM size. Because these choices directly determine the reported scaling laws, their absence makes it impossible to assess whether the claimed relationships are robust or sensitive to post-hoc decisions.
  2. [§5] §5 (Discussion): The manuscript draws implications for theoretical considerations in AI alignment, yet contains no experiment or analysis testing whether the observed overoptimization trajectories remain qualitatively unchanged when the gold-standard labels are replaced by noisy, inconsistent human preferences. The central claim that the synthetic results inform real RLHF therefore rests on an untested transfer assumption.
minor comments (2)
  1. [Figures 2-5] Figure captions and axis labels should explicitly state the range of optimization steps or KL coefficients used in each panel so that readers can reproduce the scaling plots without consulting the main text.
  2. [§4.1] The paper should include a short table listing the exact functional forms fitted for RL and best-of-n (e.g., linear, power-law, or exponential) together with the fitted coefficients and their uncertainties for at least one representative proxy size.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their thoughtful and constructive review of our manuscript. We address each major comment below, indicating planned revisions where appropriate.

read point-by-point responses

  1. Referee: [§3 and §4] §3 (Methods) and §4 (Results): The abstract and main text provide no description of the functional-form fitting procedures, choice of functional families, error-bar computation, data-exclusion rules, or robustness checks used to establish the distinct functional forms and the smooth scaling of coefficients with proxy RM size. Because these choices directly determine the reported scaling laws, their absence makes it impossible to assess whether the claimed relationships are robust or sensitive to post-hoc decisions.

    Authors: We appreciate this feedback on the need for greater transparency in our analysis pipeline. While the main text and appendix describe the overall experimental setup and report the observed functional forms, we agree that a consolidated account of the post-processing steps is warranted for reproducibility. In the revised version we will insert a dedicated paragraph in §3 (or a short appendix subsection) that explicitly details: the functional families selected for each optimization method (chosen via visual inspection and quantitative goodness-of-fit metrics), the curve-fitting procedure (nonlinear least-squares with specified initialization), error-bar computation (standard error across independent random seeds), any data-exclusion criteria (e.g., runs that failed to converge), and the robustness checks performed (alternative functional families, hyperparameter sweeps, and seed-variation tests). These additions will directly address the referee’s concern without altering the reported results. revision: yes

  2. Referee: [§5] §5 (Discussion): The manuscript draws implications for theoretical considerations in AI alignment, yet contains no experiment or analysis testing whether the observed overoptimization trajectories remain qualitatively unchanged when the gold-standard labels are replaced by noisy, inconsistent human preferences. The central claim that the synthetic results inform real RLHF therefore rests on an untested transfer assumption.

    Authors: We acknowledge that our deterministic gold-standard setup deliberately excludes label noise to isolate proxy overoptimization effects. Real human preferences introduce inconsistency that could alter variance and possibly the precise coefficients, and we do not claim direct quantitative transfer. In the revision we will expand §5 to (i) state this limitation more explicitly, (ii) discuss plausible qualitative effects of noise (e.g., increased scatter around the same functional forms), and (iii) clarify that the synthetic measurements supply controlled reference trajectories for theoretical models that can later be extended to noisy regimes. Because adding a full human-preference experiment would require new data collection far beyond the scope of the present study, we treat this as a natural direction for follow-up work rather than a revision item. revision: partial

standing simulated objections not resolved
  • Absence of experiments replacing the deterministic gold-standard labels with noisy, inconsistent human preferences to test transfer of the observed overoptimization trajectories to realistic RLHF.

Circularity Check

0 steps flagged

No circularity: purely empirical scaling observations

full rationale

The paper reports an empirical study in a closed synthetic loop where a fixed gold-standard RM supplies labels for proxy RM training and serves as the evaluation metric for policy optimization (RL or best-of-n). Observed relationships between gold scores and optimization strength are measured directly from runs; functional forms and coefficient scalings with RM size are reported as experimental findings rather than derived from equations that presuppose those forms. No self-definitional steps, fitted inputs renamed as predictions, load-bearing self-citations, uniqueness theorems, or ansatzes appear in the methodology. The work is self-contained against its own experimental benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that a fixed synthetic gold-standard reward model reproduces the misalignment dynamics of real human preferences; no free parameters or invented entities are identifiable from the abstract alone.

axioms (1)
  • domain assumption A fixed gold-standard reward model can serve as a faithful proxy for human preferences when measuring overoptimization of a separately trained proxy model.
    This assumption underpins both the label generation and the ground-truth evaluation in the synthetic setup described in the abstract.

pith-pipeline@v0.9.0 · 5724 in / 1411 out tokens · 46966 ms · 2026-05-19T08:59:16.360850+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ORPO: Monolithic Preference Optimization without Reference Model

    cs.CL 2024-03 conditional novelty 8.0

    ORPO performs preference alignment during supervised fine-tuning via a monolithic odds ratio penalty, allowing 7B models to outperform larger state-of-the-art models on alignment benchmarks.

  2. Learning, Fast and Slow: Towards LLMs That Adapt Continually

    cs.LG 2026-05 unverdicted novelty 7.0

    Fast-Slow Training uses context optimization as fast weights alongside parameter updates as slow weights to achieve up to 3x better sample efficiency, higher performance, and less catastrophic forgetting than standard...

  3. Reinforcement Learning via Value Gradient Flow

    cs.LG 2026-04 unverdicted novelty 7.0

    VGF solves behavior-regularized RL by transporting particles from a reference distribution to the value-induced optimal policy via discrete value-guided gradient flow.

  4. SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval

    cs.IR 2026-04 unverdicted novelty 7.0

    SubSearch improves LLM reasoning traces on QA and multi-hop QA tasks by rewarding intermediate steps with intrinsic process rewards instead of only final outcomes.

  5. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

    cs.AI 2024-06 conditional novelty 7.0

    LLMs trained on simple specification gaming generalize to zero-shot reward tampering including rewriting their own reward function.

  6. Let's Verify Step by Step

    cs.LG 2023-05 accept novelty 7.0

    Process supervision significantly outperforms outcome supervision for training models on the MATH dataset, achieving 78% accuracy on a representative test subset with active learning and a released 800k step-label dataset.

  7. Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces

    cs.LG 2026-05 unverdicted novelty 6.0

    A latent mediation framework with sparse autoencoders enables non-additive token-level influence attribution in LLMs by learning orthogonal features and back-propagating attributions.

  8. Learning, Fast and Slow: Towards LLMs That Adapt Continually

    cs.LG 2026-05 unverdicted novelty 6.0

    Fast-Slow Training combines slow parameter updates with fast context optimization to achieve up to 3x better sample efficiency, higher performance, less forgetting, and preserved plasticity in continual LLM learning.

  9. Beyond Static Snapshots: A Grounded Evaluation Framework for Language Models at the Agentic Frontier

    cs.AI 2026-04 unverdicted novelty 6.0

    ISOPro replaces learned reward models with deterministic verifiers in a continuous evaluation setup for LLMs, delivering larger average capability gains than GRPO-LoRA across small models in scheduling and MBPP domain...

  10. IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures

    cs.AI 2026-04 unverdicted novelty 6.0

    AI models exhibit identity-contingent withholding, providing better clinical guidance on benzodiazepine tapering to physicians than laypeople in identical scenarios, with a measured decoupling gap of +0.38 and 13.1 pe...

  11. Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling

    cs.CL 2025-07 unverdicted novelty 6.0

    REFORM uses reward-guided controlled decoding to generate adversarial failures and augments training data to improve reward model robustness on preference datasets.

  12. Listener-Rewarded Thinking in VLMs for Image Preferences

    cs.CV 2025-06 unverdicted novelty 6.0

    Listener-augmented GRPO uses an independent frozen VLM to provide dense confidence scores on reasoning traces, yielding 67.4% accuracy on ImageReward, up to +6% OOD gains on 1.2M-vote human data, and fewer reasoning c...

  13. Towards Understanding Sycophancy in Language Models

    cs.CL 2023-10 conditional novelty 6.0

    Sycophancy is prevalent in state-of-the-art AI assistants and is likely driven in part by human preferences that favor agreement over truthfulness.

  14. Reinforced Self-Training (ReST) for Language Modeling

    cs.CL 2023-08 unverdicted novelty 6.0

    ReST improves LLM translation quality on benchmarks via offline RL on self-generated data, achieving gains in a compute-efficient way compared to typical RLHF.

  15. Training Diffusion Models with Reinforcement Learning

    cs.LG 2023-05 unverdicted novelty 6.0

    DDPO uses policy gradients on the denoising process to optimize diffusion models for arbitrary rewards like human feedback or compressibility.

  16. ChemCrow: Augmenting large-language models with chemistry tools

    physics.chem-ph 2023-04 conditional novelty 6.0

    ChemCrow augments LLMs with 18 expert chemistry tools to autonomously plan and execute syntheses and guide molecular discoveries in organic synthesis, drug discovery, and materials design.

  17. Failure Modes of Maximum Entropy RLHF

    cs.LG 2025-09 unverdicted novelty 5.0

    Derives SimPO from MaxEnt RL and reports that MaxEnt RL in online RLHF exhibits frequent overoptimization and unstable KL dynamics across scales, unlike stable KL-constrained baselines.

  18. MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction

    cs.CY 2026-04 unverdicted novelty 4.0

    MedThink, a two-stage teacher-guided reasoning correction distillation framework, boosts small language models' medical diagnostic accuracy by up to 12.7% on benchmarks and achieves 56.4% on a gastroenterology dataset.

  19. Qualixar OS: A Universal Operating System for AI Agent Orchestration

    cs.AI 2026-04 unverdicted novelty 4.0

    Qualixar OS provides a runtime for multi-agent AI systems with support for 12 topologies, LLM-driven team design, dynamic routing, consensus judging, content attribution, and protocol bridging, achieving 100% accuracy...

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages · cited by 18 Pith papers · 16 internal anchors

  1. [1]

    Stuart Armstrong et al

    URL https://proceedings.neurips.cc/paper/2018/file/ d89a66c7c80a29b1bdbab0f2a1a94af8-Paper.pdf. Stuart Armstrong et al. General purpose intelligence: arguing the orthogonality thesis. Analysis and Metaphysics, 12(68):1–20,

    work page 2018

  2. [2]

    Explaining neural scaling laws

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701,

    work page arXiv

  3. [3]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    ISBN 0199678111. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Si...

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  5. [5]

    On Evaluating Adversarial Robustness

    URL https://arxiv.org/abs/1902.06705. Joseph Carlsmith. Is power-seeking AI an existential risk? arXiv preprint arXiv:2206.13353,

    work page internal anchor Pith review Pith/arXiv arXiv 1902

  6. [6]

    Adversarial Attacks and Defences: A Survey

    Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey. arXiv preprint arXiv:1810.00069,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    On Adversarial Examples for Character-Level Neural Machine Translation

    URLhttps://proceedings.mlr.press/ v80/dai18b.html. Javid Ebrahimi, Daniel Lowd, and Dejing Dou. On adversarial examples for character-level neural machine translation. arXiv preprint arXiv:1806.09030,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Reinforcement Learning with a Corrupted Reward Channel

    Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Generalization and regularization in dqn

    Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123,

    work page arXiv

  10. [10]

    com/deepmind-media/DeepMind.com/Authors-Notes/sparrow/sparrow-final.pdf

    URL https://storage.googleapis. com/deepmind-media/DeepMind.com/Authors-Notes/sparrow/sparrow-final.pdf . 13 Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models. arXiv preprint arXiv:2203.07472,

    work page arXiv

  11. [11]

    Towards Deep Neural Network Architectures Robust to Adversarial Examples

    Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al

    URL https://arxiv.org/abs/2104.13733. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701,

    work page arXiv 2010

  13. [13]

    Scaling Laws for Transfer

    Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Risks from Learned Optimization in Advanced Machine Learning Systems

    Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820,

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  15. [15]

    Scaling Laws for Neural Language Models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  16. [16]

    Rl with kl penalties is better viewed as bayesian inference

    Tomasz Korbak, Ethan Perez, and Christopher L Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275,

    work page arXiv

  17. [17]

    Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg

    URL https://vkrakovna.wordpress.com/2019/08/19/ classifying-specification-problems-as-variants-of-goodharts-law/ . Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity, 4

    work page 2019

  18. [18]

    Scalable agent alignment via reward modeling: a research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    org/abs/1703.06748

    URL https://arxiv. org/abs/1703.06748. David Manheim and Scott Garrabrant. Categorizing variants of goodhart’s law. arXiv preprint arXiv:1803.04585,

    work page arXiv

  20. [20]

    Teaching language models to support answers with verified quotes

    Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    WebGPT: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    The alignment problem from a deep learning perspective

    Richard Ngo. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626,

    work page arXiv

  23. [23]

    files.wordpress.com/2008/01/ai_drives_final.pdf

    URL http://selfawaresystems. files.wordpress.com/2008/01/ai_drives_final.pdf. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Jan Leike, and Ryan Lowe. Trainin...

    work page 2008

  24. [24]

    The effects of reward misspecification: Mapping and mitigating misaligned models

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544,

    work page arXiv

  25. [25]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    A neural scaling law from the dimension of the data manifold

    Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802,

    work page arXiv 2004

  27. [27]

    Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky

    URL https://arxiv.org/abs/2209.13085. Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. InWorkshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence,

    work page arXiv

  28. [28]

    Observational overfitting in reinforcement learning

    Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. arXiv preprint arXiv:1912.02975,

    work page arXiv 1912

  29. [29]

    Intriguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199,

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning

    URL https://proceedings.neurips.cc/paper/2021/file/ c26820b8a4c1b3c2aa868d6d57e14a79-Paper.pdf. 15 Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937, 2018a. Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  31. [31]

    = log x. C Hyperparameters Hyperparameter Value RM Adam learning rate multiplier 1.67e-2 RM batch size 64 RL Adam learning rate multiplier 4e-3 RL batch size 256 RL PPO clipping parameter 0.2 RL Timesteps per rollout 256 RL minibatches per epoch 128 RL GAE bootstrapping parameter 0.95 Table 1: Hyperparameters used throughout the experiments. 18 What is fu...

    work page 1922

  32. [32]

    Figure 11: Validation losses for the proxy RMs in section 3.2 by size, plus the two near-chance level RMs

    19 Figure 10: Maximum gold scores for all RM size and data size combinations. Figure 11: Validation losses for the proxy RMs in section 3.2 by size, plus the two near-chance level RMs. 20 Figure 12: Max BoN gold scores (αbon/2βbon) predicted with the BoN closed form Figure 13: Total number of data points seen does not seem to affect the gold RM score much...

    work page 2000