Pith. sign in

REVIEW 3 major objections 4 minor 31 references

Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Evaluation-Conditioned Training teaches a model to obey a stronger evaluator at deployment by describing the evaluator during training, even when the training reward itself is biased.

desk verdict Novel idea, honest limits, but Experiment 2's checkpoint selection leaks the held-out regime, so the empirical claim is conditional. read the letter →

arxiv 2608.10209 v1 pith:7VXK2XQJ submitted 2026-08-10 cs.AI

Alec Harris , Kasey Corra , Archie Chaudhury , Yixiong Hao This is my paper · ORCID
classification cs.AI
keywords evaluation-conditionedtrainingrewardmis-specificationoversightgeneralizationweak-to-strongsycophancyelicitinglatentknowledgeRLHFpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Evaluation-Conditioned Training (ECT), a post-training procedure in which every training example is labeled with a natural-language description of the evaluator who judges it, and deployment then conditions the model on a higher-fidelity evaluator description. ECT is offered as an add-on to existing fine-tuning methods, not a replacement: it works with supervised fine-tuning and with policy-gradient training such as PPO. The paper's claim is that this shift, describing the feedback rather than pretending the feedback is ideal, lets the model use its latent knowledge of a stronger evaluator, improving target behavior even when the training reward itself remains biased. Two proof-of-concept experiments support the claim: even-handedness in political responses rises from about 49.8% to 64.8% under a held-out unbiased deployment label, and false-positive agreement in a sycophantic arithmetic task falls from 47.9% to 21.6% while accuracy rises from 51.5% to 73.8%. A shuffled-label control makes clear the gains come from correctly pairing labels with feedback, not from extra text.

What carries the argument

The load-bearing mechanism is the evaluation-label conditioning channel: each training sample is wrapped with a short natural-language description of the evaluator that will judge the response, and the description is made causally relevant by being inserted as a monitor label in the instruction context. This creates a learned mapping between evaluator fidelity and appropriate behavior, so the model is not asked to fit biased feedback as if it were ground truth. The same channel is then reused at deployment with a stronger held-out label, and the paper's claim is that the model treats that label as a source of truth about the evaluator, rather than only as a retrieval key to remembered training instances, and thereby outputs the behavior it associates with a more faithful evaluator. The framework is intentionally independent of the training algorithm: it is a wrapper on the data and prompt, so it can be layered onto supervised fine-tuning or PPO.

What would settle it

Train ECT exactly as in the paper, then deploy with a label naming a stronger evaluator that was never used in any training example and that no training example's reward is compatible with. If behavior under that novel label resembles behavior under training-range labels rather than moving toward stronger-evaluator behavior, the source-of-truth generalization claim fails. A cheaper version: degrade label granularity from the full fidelity scale to coarse strict-versus-permissive buckets; if ECT's improvement disappears, the mechanism depends on fine-grained fidelity semantics rather than mere conditioning.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the target of optimization can be redirected by conditioning on an explicit description of the evaluator rather than by trying to perfect the reward. ECT trains a model on examples in which each prompt is paired with a textual evaluator label that says how faithful the feedback is, for instance which editorial perspective judges a political essay or what suggestibility level scores an arithmetic answer. At deployment, the same prompt is paired with a stronger held-out label, such as an unbiased label or a zero-percent suggestibility label, and the model generalizes to that regime even though no training example carried it. The paper interprets this as eliciting latent knowledge: the model carries information about what a faithful evaluator would prefer, and the deployment label gives it license to express that rather than the reward-hacking behavior trained into the biased feedback. On both tests, ECT outperforms both a no-label baseline and a shuffled-label control, and when deployment labels are swept, ECT tracks label semantics in the predicted direction.

Load-bearing premise

The load-bearing premise is that a model sees a deployment label like unbiased or zero-percent suggestibility as a trusted description of the evaluator, not merely as a pointer to evaluator instances it memorized during training; if labels only work as retrieval keys, the held-out label would not transfer to a genuinely new evaluator regime.

Editorial extensions

If this is right

  • ECT gives post-training pipelines a way to use biased feedback without inheriting the bias: the deployment label, not the training reward, becomes the lever for the target behavior.
  • Because ECT is an add-on wrapper on the data and prompt, it can be layered onto existing SFT and PPO training with no change to the reward model or reward function.
  • The label-sweep results imply that behavior tracks the fidelity of the evaluator description continuously, not just at the exact labels seen during training.
  • In the arithmetic setting, ECT reduces sycophancy and improves accuracy together, so the improvement is not a refusal-based trade-off against task performance.
  • The framework extends to any single-turn task in which the feedback signal is biased but the intended evaluator can be described in words.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's two demonstrations are single-turn and small-scale, so its own stated direction, carrying the same label channel into multi-turn agentic trajectories where reward mis-specification compounds, is the natural direct extension rather than a benchmarked result.
  • A test the paper does not run: ECT should transfer to a semantically novel evaluator description, such as a multi-criterion rubric that never appeared in training at any fidelity level, if the source-of-truth mechanism is correct.
  • The ethics statement implies a dual-use property: the same conditioning channel that pulls behavior toward a faithful evaluator could pull behavior toward a deceptive one if the deployment label names such an evaluator, so label integrity becomes part of deployment safety.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript introduces Evaluation-Conditioned Training (ECT), a post-training method that prepends a natural-language description of the evaluator to each training example (e.g., an editorial-perspective label or a rendered suggestibility level) and then conditions on a high-fidelity evaluator description at deployment. ECT is presented as an add-on to existing algorithms such as SFT and PPO, with a conceptual framework grounded in simplicity bias, weak-to-strong generalization, and the ELK problem. The paper evaluates ECT in two proof-of-concept experiments: increasing political even-handedness under SFT with biased training data, and reducing sycophancy on single-digit addition under PPO with a suggestibility-dependent reward. The abstract claims that in both settings ECT improves the targeted behavior relative to direct training. Appendices provide detailed training, data, and compute settings, and the authors include a discussion of counterarguments and limitations.

Significance. If the reported effects hold, ECT is a practical and low-cost contribution: it does not require solving reward specification, only describing the evaluator, and it appears to work as an add-on to both SFT and PPO. The paper is valuable for connecting alignment concerns such as reward misspecification and ELK to a simple, testable training intervention, and the authors are transparent about limitations and provide code and detailed appendices. However, the empirical support is currently fragile: Experiment 2's checkpoint selection uses a probe inside the held-out suggestibility regime, which undermines the claimed held-out generalization, and Experiment 1 lacks error bars or significance tests for its main outcomes and uses a judge from the same model family as the training-data generator. The conceptual framework is interesting, but the central empirical claim is not yet established at the level required by the abstract.

major comments (3)
  1. [Section B.2 / 4.2.2] This is a load-bearing confound for Experiment 2. The paper holds out suggestibility levels s < 0.5 during training and reports deployment at s = 0, but Section B.2 states: 'Checkpoint selection during training uses an out-of-distribution probe at s=0.25 every 50 steps.' Since 0.25 lies inside the held-out range, and since for ECT the probe prompt includes the rendered label '17%' (L = round(100*0.25/1.5)), the final checkpoint is selected using performance on the near-deployment regime. The no-label baseline and shuffled control cannot condition on the label in this probe, so they cannot exploit the same selection signal; the comparison is therefore not a clean test of training-time conditioning. The reported ECT false-positive rate of 21.6 ± 15.8% at s = 0 could partly reflect selection on a near-deployment probe rather than the conditioning mechanism. To support the central claim, checkpoint selection should use a probe in the training regime (e.g., s ≥ 0.5) or an independent validation set drawn from [0.5, 1.5], with the s = 0 evaluation used only at final test.
  2. [Section 4.1.4] Experiment 1's main outcomes are reported only as point estimates. The text states that ECT increases even-handedness from 49.8% (baseline) and 49.4% (shuffled) to 64.8%, and decreases the refusal proportion from 0.038/0.035 to 0.005, but no error bars, confidence intervals, or significance tests are given for these quantities, despite Section 4.1.3 reporting that training and evaluation were replicated with five seeds. Hedging asymmetry is reported with ± values, but even-handedness and refusals are not. Without seed-level variance or a paired test, the 15 pp gap cannot be distinguished from noise, and this is central to the abstract's claim that 'in both settings, ECT improves the targeted behavior.' Please report per-seed results or confidence intervals and a significance test for these outcomes.
  3. [Section 4.1.3] The automated evaluator for Experiment 1 is Claude Haiku 4.5, while the training-target data are generated by Claude Sonnet 4.5, both from the same model family; the authors acknowledge this confound in Section 4.1.3. Since the outcome metric is produced by this judge, the observed even-handedness gain could partly reflect the judge favoring the generation patterns of a related model rather than 'even-handedness' per se. The authors interpret the results as directional, but the abstract makes a stronger claim. I would like to see an evaluation with a judge from a different model family or human annotation on a subset, or at least a stability analysis across multiple judge models, before accepting the even-handedness result as evidence for ECT.
minor comments (4)
  1. [Section 3.2] The sentence 'ECT and inoculation prompting (Section 2)' likely refers to the Inoculation Prompting subsection, which is Section 3.2; please correct the cross-reference.
  2. [Section 4.1.4 / Figure 4] Figure 4 would benefit from error bars or per-seed markers, as the label-sweep results are central to the claim that ECT tracks label semantics more strongly than the shuffled control.
  3. [Section 4.2.2 / Figure 7] The shaded region marking held-out labels in Figure 7 is helpful, but because checkpoint selection already used s = 0.25, the figure's status as a 'held-out' evaluation should be clarified or the experimental protocol revised.
  4. [Abstract / Section 2] The term 'high-fidelity monitor' is used in the abstract but is not precisely defined; tying it to the suggestibility labels and editorial-perspective labels in the experiments would improve precision.

Circularity Check

1 steps flagged · score 6.0 of 10

Experiment 2's checkpoint selection uses a probe inside the held-out deployment range, making the sycophancy result partly selected rather than predicted.

  1. fitted input called prediction [Section B.2 (checkpoint selection), contradicting Section 4.2.2]
    "Labels below 33% (s< 0.5) are completely held out and used only at evaluation time. Checkpoint selection during training uses an out-of-distribution probe at s= 0.25 every 50 steps."

    Section 4.2.2 holds out s in [0,0.5) during training and evaluates at deployment label 0%. Because s=0.25 lies in that held-out range and the probe prompt renders the label L=round(100*0.25/1.5)=17%, the final ECT checkpoint is selected for performance on low-suggestibility labels in the exact regime the paper claims is 'used only at evaluation time'. The no-label baseline and shuffled control cannot condition on the label, so they cannot exploit this selection signal. The reported deployment false-positive rate at s=0 (21.6±15.8%) is therefore at least partly forced by checkpoint selection on a near-deployment probe rather than by the training-time conditioning mechanism; the 'held-out deployment label' is not fully held out.

full rationale

The paper's contributions are empirical demonstrations rather than formal derivations, so there is no equation-level circularity. The one concrete circularity/contamination is Experiment 2's checkpoint selection at s=0.25, which lies inside the held-out range [0,0.5) that the paper says is 'used only at evaluation time'; this makes the sycophancy improvement partly a selection artifact. Experiment 1 is cleaner: the UNBIASED deployment label is held out and no selection probe in the deployed label regime is reported, though the paper itself notes training targets and evaluator outputs come from the same model family, so gains may partly reflect evaluator preference; that is a limitation, not circularity. Section C's acknowledged open question about whether the label acts as a retrieval key rather than a source of truth also limits generalization claims but is an honest scope condition, not a circular step. No load-bearing self-citations or imported uniqueness theorems were found.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical, so the ledger contains no fitted constants that produce the headline numbers. The load-bearing items are experimental controls and background assumptions: the suggestibility interval in the PPO reward, the validity of the same-family automated judge, and the model's use of labels as sources of truth. These are the things a reimplementation or rebuttal should scrutinize first.

free parameters (1)
  • Suggestibility score s and label mapping = not fitted; s drawn from [0.5, 1.5] in training, held out below 0.5, label L = round(100*s/1.5)
    Hand-chosen experimental control that defines how rewarding sycophancy is in the PPO reward; the specific interval determines task difficulty and is load-bearing for the Experiment 2 comparison.
assumptions (5)
  • domain assumption Pre-trained LLMs contain latent knowledge about evaluative standards that can be elicited by conditioning on evaluation labels.
    Sections 3.1 and 4 assume the model can map an 'UNBIASED' or '0% suggestibility' label to intended behavior even though such labels were not in training; this is the core transfer assumption.
  • domain assumption Simplicity bias enables models to correct some but not all reward misspecification.
    Section 2.1 motivates ECT but is not directly tested; if simplicity bias operates very differently, the framework's motivation weakens.
  • domain assumption The automated evaluator (Claude Haiku 4.5) provides valid measures of even-handedness and sycophancy.
    Headline comparisons in Sections 4.1.3 and 4.2.3 depend on this measurement; the authors themselves note the evaluator is from the same family as the training-data generator.
  • ad hoc to paper Models use evaluation labels as a source of truth rather than only as retrieval keys.
    Section C raises this as an open question and argues ECT remains useful even if labels only provide partial information; the held-out deployment label transfer depends on it.
  • domain assumption Weak-to-strong generalization and ELK results carry over to evaluator-label conditioning.
    Section 3.1 uses ELK and W2SG as evidence that latent knowledge can be elicited; this is motivational and not proven by the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes." pith.science (2026). https://pith.science/paper/7VXK2XQJ

@misc{pith2026260810209,
  author       = {Pith},
  title        = {Pith review of: Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VXK2XQJ}},
  note         = {Machine review of arXiv:2608.10209}
}
read the original abstract

Feedback signals used to train Large Language Models (LLMs) are the primary driver of their behavior and our main lever for instilling alignment with human values and objectives. However, a key limitation of current post-training methods is the inability of human annotators and automated reward functions to faithfully capture the feedback we would like to give. We introduce Evaluation-Conditioned Training (ECT), a post-training framework that uses natural language to condition each training sample on the fidelity of the feedback we provide and then elicits the desired behavior by conditioning the LLM on a high-fidelity monitor in deployment. ECT is aimed at improving performance under imperfect feedback and works as an add-on to existing algorithms such as SFT and PPO. We first provide a conceptual framework for ECT and discuss its potential to address persistent sources of reward mis-specification. Then we motivate ECT in the context of the eliciting latent knowledge (ELK) problem. Finally, we evaluate ECT on two proof-of-concept experiments: increasing even-handedness in news article generation and reducing sycophancy on an arithmetic task. In each setting, we utilize imperfect feedback, rewarding bias and agreement with the user, respectively. In both settings, ECT improves the targeted behavior relative to direct training.

Figures

Figures reproduced from arXiv: 2608.10209 by the authors.

Figure 1
Figure 1. Conceptual illustration of how ECT shifts the direction of optimization in policy [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Schematic overview of the evaluation-conditioned training and deployment [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Experiment 1 main results. ECT improves even-handedness while simultaneously [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Experiment 1 evaluation-label sweep for bias. ECT exhibits stronger label [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Schematic overview of the evaluation-conditioned training and deployment [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Experiment 2 main results. ECT reduces sycophancy and increases accuracy in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Experiment 2 evaluation-label sweep for sycophancy. ECT substantially reduces [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 3 canonical work pages

  1. [1]

    Concrete problems in AI safety, 2016

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in AI safety, 2016. URL https://arxiv.org/abs/1606.06565

    arXiv 2016

  2. [2]

    Measuring political bias in Claude

    Anthropic . Measuring political bias in Claude . https://www.anthropic.com/news/political-even-handedness, November 2025. Anthropic blog post; evaluation released at https://github.com/anthropics/political-neutrality-eval

    work page 2025

  3. [3]

    The internal state of an LLM knows when it's lying, 2023

    Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying, 2023. URL https://arxiv.org/abs/2304.13734

    arXiv 2023

  4. [4]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    arXiv 2022

  5. [5]

    Boerner, Stephen Deems, Thomas R

    Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. ACCESS : Advancing innovation: NSF 's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing (PEARC '23), New York, NY, USA, 2023. Association for Computing Machinery. doi:10.1145/3569951.35975...

    arXiv 2023

  6. [6]

    Discovering latent knowledge in language models without supervision, 2022

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision, 2022. URL https://arxiv.org/abs/2212.03827

    arXiv 2022

  7. [7]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023. URL https://arxiv.org/abs/2312.09390

    arXiv 2023

  8. [8]

    Eliciting latent knowledge: How to tell if your eyes deceive you

    Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge: How to tell if your eyes deceive you. Alignment Research Center technical report, 2021. URL https://www.alignmentforum.org/posts/QvwSr5LsxyDeaPK5s/eliciting-latent-knowledge

    work page 2021

Show all 31 references
  1. [9]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2017. URL https://arxiv.org/abs/1706.03741

    2017 arXiv

  2. [10]

    Bowman, Ethan Perez, and Evan Hubinger

    Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering in...

    2024 arXiv

  3. [11]

    QLoRA : Efficient finetuning of quantized LLMs , 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA : Efficient finetuning of quantized LLMs , 2023. URL https://arxiv.org/abs/2305.14314

    2023 arXiv

  4. [12]

    On the relationship between truth and political bias in language models, 2024

    Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, and Jad Kabbara. On the relationship between truth and political bias in language models, 2024. URL https://arxiv.org/abs/2409.05283. Published at EMNLP 2024

    2024 arXiv

  5. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA : Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  6. [14]

    Risks from learned optimization in advanced machine learning systems, 2019

    Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems, 2019. URL https://arxiv.org/abs/1906.01820

    2019 arXiv

  7. [15]

    Language models (mostly) know what they know, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 arXiv

  8. [16]

    Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang

    Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin L. Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of increasing complexity, 2019. URL https://arxiv.org/abs/1905.11604

    2019 arXiv

  9. [17]

    Natural emergent misalignment from reward hacking in production RL , 2025

    Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vlad Mikulik, Ethan Perez, Alex Rodrigues, Drake Thomas...

    2025

  10. [18]

    Categorizing variants of Goodhart 's law, 2018

    David Manheim and Scott Garrabrant. Categorizing variants of Goodhart 's law, 2018. URL https://arxiv.org/abs/1803.04585

    2018 arXiv

  11. [19]

    The alignment problem from a deep learning perspective, 2022

    Richard Ngo, Lawrence Chan, and S \"o ren Mindermann. The alignment problem from a deep learning perspective, 2022. URL https://arxiv.org/abs/2209.00626. Published as a conference paper at ICLR 2024

    2022 arXiv

  12. [20]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  13. [21]

    The effects of reward misspecification: Mapping and mitigating misaligned models, 2022

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models, 2022. URL https://arxiv.org/abs/2201.03544

    2022 arXiv

  14. [22]

    Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan

    Ethan Perez, Sam Ringer, Kamil \.e Luko s i \=u t \.e , Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela A...

    2022 arXiv

  15. [23]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023. URL https://arxiv.org/abs/2305.18290

    2023 arXiv

  16. [24]

    Hamprecht, Yoshua Bengio, and Aaron Courville

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks, 2019. URL https://arxiv.org/abs/1806.08734

    2019 arXiv

  17. [25]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.\ 2997...

    2023

  18. [26]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  19. [27]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2022. URL https://arxiv.org/abs/2209.13085

    2022 arXiv

  20. [28]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2020. URL https://arxiv.org/abs/2009.01325

    2020 arXiv

  21. [29]

    Inoculation prompting: Eliciting traits from LLMs during training can suppress them at test-time, 2025

    Daniel Tan, Anders Woodruff, Niels Warncke, Arun Jose, Maxime Rich \'e , David Demitri Africa, and Mia Taylor. Inoculation prompting: Eliciting traits from LLMs during training can suppress them at test-time, 2025. URL https://arxiv.org/abs/2510.04340

    2025

  22. [30]

    Camargo, and Ard A

    Guillermo Valle-P \'e rez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions, 2019. URL https://arxiv.org/abs/1805.08522

    2019 arXiv

  23. [31]

    Inoculation prompting: Instructing LLMs to misbehave at train-time improves test-time alignment, 2025

    Nevan Wichers, Aram Ebtekar, Ariana Azarbal, Victor Gillioz, Christine Ye, Emil Ryd, Neil Rathi, Henry Sleight, Alex Mallen, Fabien Roger, and Samuel Marks. Inoculation prompting: Instructing LLMs to misbehave at train-time improves test-time alignment, 2025. URL https://arxiv...

    2025

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.