Pith. sign in

REVIEW 3 major objections 6 minor 36 references

Risk aversion trained only on tiny-stakes gambles can partially carry over to astronomically high stakes in language models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 07:14 UTC pith:IDGJ7QB7

load-bearing objection Solid empirical paper: low-stakes risk-aversion training partially transfers to 10^100-stakes prompts, multi-method and multi-family, with residual ~30% Rebel rates the authors correctly flag as insufficient for a failsafe. the 3 major comments →

arxiv 2607.02755 v1 pith:IDGJ7QB7 submitted 2026-07-02 cs.LG cs.AI

Out-of-Distribution Generalization of Risk Aversion in Language Models

Kristina Zhang , Junior Chinomso Okoroafor , Benjamin Maltbie , Andrew Lin , Abhitej Bokka , Elliott Thornley This is my paper
classification cs.LG cs.AI
keywords risk aversionout-of-distribution generalizationlanguage modelsAI safetyconstant absolute risk aversioncooperate-versus-rebelpost-training methodsfailsafe
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether risk aversion over resources, trained only on cheap low-stakes gambles, can still shape choices when the payoffs become astronomically large—exactly the shift that would matter if a misaligned AI faced a real cooperate-versus-takeover decision. It introduces RiskAverseOOD, a benchmark that trains only on payoffs up to $100, validates and tests at medium and high stakes, and finally checks ‘deployment’ situations whose prizes proxy resources worth up to $10^100. Across several post-training methods, models that begin choosing the safe Cooperate option only about 2% of the time at astronomical stakes rise to roughly 70% (supervised fine-tuning and tie training), 52% (preference optimization), and 39% (activation steering); a fine-tuned reward model also ranks appropriately risk-averse reasoning almost perfectly. The same pattern appears at multiple model sizes and families, and the learned aversion partially transfers to other quantities such as GPU-hours and lives saved. The results show that low-stakes training can move behavior tens of percentage points across 98 orders of magnitude, yet still leave rebellion rates far too high for a reliable failsafe—leaving consistency itself as the open challenge.

Core claim

Risk aversion induced solely on low-stakes gambles (payoffs in roughly [-100,100]) generalizes at least partially to astronomically high stakes (resources worth up to $10^100). On the main model family, the rate of choosing the safe Cooperate option rises from a roughly 2% baseline to about 70% with supervised fine-tuning and tie training, 52% with direct preference optimization, and 39% with activation steering; a fine-tuned reward model reaches 99.6% pairwise accuracy at preferring calibrated risk-averse reasoning. The effect holds across model scales and families and transfers partially to other goods, but remains too inconsistent for a dependable safeguard against misalignment.

What carries the argument

RiskAverseOOD: a training-and-evaluation suite that forces all learning onto low-stakes situations while measuring Cooperate-versus-Rebel choices at medium, high, and astronomical stakes (plus a too-risk-averse control). The concrete training target is constant-absolute-risk-aversion utility u(w)=1-e^{-0.01w}, which saturates quickly and thereby supplies a plausible mechanism for stakes generalization.

Load-bearing premise

That stylized hypothetical prompts sharing the same format and terminology as the low-stakes training set are a valid stand-in for how a hard-to-deceive model would actually choose under real resource gambles and real takeover-like options.

What would settle it

Train only on the low-stakes set, then measure Cooperate rates on a held-out battery of astronomically-stakes situations whose wording, option structure, and uncertainty description are deliberately made dissimilar to training; if the rate remains near the untrained baseline, the claimed OOD generalization fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Low-stakes risk-aversion training can serve as a partial extra line of defense even when developers cannot safely offer high-stakes real-resource gambles during training.
  • Utility functions that saturate quickly can shrink the effective distance models must generalize across, making extreme stakes less foreign in utility space.
  • Reward models that rank risk-averse reasoning highly can be used as filters or critics without requiring the policy itself to generate the safe choice.
  • Methods that train only against risk-neutral alternatives can push models into excessive risk aversion, so calibration checks on ‘Steal’ options are necessary.
  • The remaining one-third Rebel rate at astronomical stakes means current techniques are not yet sufficient for a reliable failsafe; further methods that drive that rate far lower using only low-stakes data are required.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the saturation mechanism is the main driver, deliberately training toward utilities with even tighter upper bounds or different absolute-risk-aversion coefficients could raise Cooperate rates without any high-stakes data.
  • Shared surface form between train and test may be inflating measured generalization; stress-testing with fully re-phrased or multi-turn agentic scenarios would reveal how much of the gain survives realistic distribution shift.
  • Partial transfer to lives-saved and GPU-hour domains suggests risk-aversion post-training can reshape a broader preference geometry, which may interact unexpectedly with other safety or capability objectives.
  • Because even the best methods still leave substantial Rebel probability, risk aversion is best treated as one layer among several rather than a standalone solution to misalignment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces RiskAverseOOD, a benchmark that trains language models only on low-stakes resource gambles (payoffs in [−100, 100]) and evaluates whether induced risk aversion transfers to medium ($1M), high ($10M), and astronomical (resources worth up to $10^100) stakes, plus a too-risk-averse control. Using a CARA target u(w)=1−e^{−0.01w}, the authors compare SFT, tie training, DPO, activation steering, and reward-model fine-tuning on Qwen3-8B and report large gains in Cooperate rates on astronomical-stakes prompts (from ~2% baseline to ~70% SFT/tie, ~52% DPO, ~39% steering), near-ceiling RM pairwise accuracy (99.6%), partial transfer across goods, and similar patterns across Qwen3 scales (1.7B–14B) and two other model families. They carefully frame residual ~30% Rebel rates as insufficient for a reliable misalignment failsafe and leave consistent astronomical risk aversion as an open problem.

Significance. If the result holds inside its stated scope, it is a concrete, falsifiable contribution to AI-safety methodology: it shows that a practically train-only-on-cheap-gambles constraint need not make resource risk aversion useless at extreme stakes, and it supplies a reusable benchmark, multi-method baselines, multi-seed tables, scaling and cross-family checks, a too-risk-averse control, MMLU-Redux retention numbers, and public code. The utility-saturation discussion and Appendix D CoTs give a plausible mechanism for why high-to-astronomical Cooperate rates stay flat. The work does not claim a deployable failsafe; its main scientific value is establishing partial OOD transfer and an open challenge with clear success criteria.

major comments (3)
  1. [§5.2, Table 2] Table 2, SFT/tie rows: Qwen3-14B astronomical Cooperate falls to 57.10±0.77 (SFT) and 56.25±1.65 (tie) versus ~70% for Qwen3-8B, while medium/high stakes remain comparable. The paper reports the numbers but does not analyze this reverse scaling. Because the safety motivation depends on larger models, the central “partial OOD generalization” claim needs either an explanation (e.g., hyperparameter retuning, parse-rate, CoT length) or an explicit caveat that gains may not strengthen with scale.
  2. [§3, Appendix D, Appendix E.5] §3 and Appendix E.5: chosen training CoTs are generated to explicitly state and optimize u(w)=1−e^{−0.01w}. Appendix D shows trained models reusing that functional form and treating large prizes as utility ~1. The reported OOD gains may therefore reflect transfer of an explicit CARA calculation recipe under matched prompt structure rather than a general risk attitude that would survive differently phrased or non-CoT settings. A load-bearing control—e.g., training without naming the utility, or evaluating with format-shifted prompts—would substantially strengthen the interpretation that “risk aversion” itself generalizes across 98 orders of magnitude.
  3. [§5.2, Table 3] Table 3, Gemma SFT/tie too-risk-averse column: cooperation falls from 78.7% baseline to ~44% / ~43%, i.e., models become substantially more Steal-seeking after low-stakes training. The main text emphasizes Qwen’s improved too-risk-averse rates for SFT/tie; the Gemma degradation is under-discussed relative to the claim that SFT/tie yield calibrated rather than indiscriminate risk aversion across families.
minor comments (6)
  1. [Abstract, §2, Table 10] Abstract and §1 state “98 orders of magnitude”; the stake ranges in Table 10 run from order $10^2 training payoffs to $10^100 astronomical takeover, which is ~98 decades in dollars—fine, but a one-line derivation in §2 would prevent reader confusion with log10(10^100/100)=98.
  2. [§5.3, Table 4] Table 4 caption correctly notes that higher Cooperate is not necessarily better in lives-saved; the main-text sentence “induce a risk aversion … that generalizes across all three domains” still reads as unqualified success. Soften the wording to match the caption.
  3. [§4, Table 1, Appendix B.3] §4 item 3 and Appendix B.3: DPO uses 600 examples vs 1,000 for SFT/tie. The rationale is clear; still flag the unequal data budget in Table 1’s caption so method comparisons are not over-read as pure objective differences.
  4. [Figure 1, §2] Figure 1 / resource note: the note is necessary, but the paper could briefly report whether removing it collapses parse rates or changes Rebel rates for trained models (even if only in an appendix).
  5. [§8] Related Work is appropriately scoped; a short pointer to classical CARA/CRRA elicitation and to prior LM decision-under-risk papers already cited would help non-safety readers place the benchmark.
  6. [Abstract] Minor polish: “RiskA-verseOOD” line-break hyphenation in the abstract block; ensure consistent spelling of “astronomically-high-stakes” vs “astronomically high stakes.”

Circularity Check

0 steps flagged

No significant circularity: empirical post-training study with held-out stake-level evaluation; CARA is a chosen training target, not a fit recovered as a prediction.

full rationale

RiskAverseOOD is an empirical benchmark paper, not a first-principles derivation. The load-bearing claim is that low-stakes training (payoffs in [−100,100]) raises Cooperate rates on held-out high- and astronomical-stakes prompts (up to ~$10^100). Training uses CoTs generated to follow a deliberately chosen CARA target u(w)=1−e^{−0.01w}; evaluation reports behavioral Cooperate/Rebel rates and RM pairwise accuracy on stake-shifted prompts, not recovery of α from the same data used to set it. Citations to Thornley & MacAskill (2026) motivate the safety strategy and the CARA target; they do not force the reported OOD percentages. Shared train/eval surface form is an external-validity limitation the authors already flag, not a by-construction reduction of the result to its inputs. No self-definitional loop, fitted-input-as-prediction, or uniqueness theorem smuggled in as external fact was found. Score 0 is the correct honest finding.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The load-bearing content is empirical: a hand-chosen CARA training target, a synthetic gamble benchmark with stake ladders, and standard post-training interventions. Free parameters are training/evaluation design choices (risk coefficient, stake cutoffs, LoRA and steering knobs). Axioms are domain assumptions from the risk-aversion-as-failsafe program plus ordinary ML practice. The main invented entity is the RiskAverseOOD stake ladder and option taxonomy, which is a measurement construct rather than a physical entity.

free parameters (6)
  • CARA absolute risk aversion α=0.01 (target u(w)=1−e^{−0.01w})
    Hand-chosen training target; alternatives (other α, CRRA) are left to future work and could change high-stakes rates.
  • Too-risk-averse contrast α=0.10
    Defines rejected/excessively risk-averse CoTs and the Steal evaluation; tenfold jump is a design choice.
  • Stake cutoffs ($100 train, $1M val, $10M high, $10^100 astronomical)
    Define the OOD ladder and the ‘98 orders of magnitude’ framing; not derived from data.
  • LoRA rank r=32, α=64 and per-method learning rates / epochs / β=0.1
    Selected via medium-stakes validation sweeps; locked for method comparison.
  • Activation steering layer 18, strength α=34
    Chosen by sweep on validation; inference-time free knobs.
  • Tie rate 30% and DPO subset of 600 linear-utility-rejected pairs
    Training-corpus design choices that materially affect reported method rankings.
axioms (5)
  • domain assumption Risk aversion over resources can serve as a failsafe if misaligned AIs prefer low-risk cooperation to high-risk takeover-like strategies.
    Motivates the whole benchmark (Introduction; Thornley & MacAskill 2026); not tested here.
  • domain assumption CARA utilities are a suitable training target (reflectively stable, bounded above, independent of background risk).
    Used to generate chosen CoTs and labels (Training Set; Pratt 1964 cited via prior recommendation).
  • ad hoc to paper Hypothetical prize-choice prompts with a resource note isolate risk attitudes without alignment training dominating the answer.
    Core evaluation design (Benchmark §2; Limitations on Hypotheticals).
  • ad hoc to paper Shared prompt structure between low-stakes train and high-stakes eval isolates stake generalization rather than format shift.
    Explicit design choice and limitation (Limitations: Shared structure).
  • standard math Standard next-token SFT, DPO, CAA steering, and Bradley-Terry RM training are valid interventions for inducing preference-like behavior.
    Methods section relies on established post-training machinery.
invented entities (2)
  • RiskAverseOOD benchmark (low/medium/high/astronomical stake sets + too-risk-averse Steal set) independent evidence
    purpose: Measure OOD generalization of resource risk aversion under a low-stakes-only training constraint.
    New evaluation construct with procedural generation rules; independent_evidence is the public code/data path, not external physical measurement.
  • Cooperate / Rebel / Steal option taxonomy with endogenous lose-everything calibration no independent evidence
    purpose: Map gamble choices onto desired risk-averse vs insufficiently vs excessively risk-averse behavior without using alignment-laden labels in the prompt.
    Labeling scheme invented for this paper’s safety analogy; success metrics are defined relative to it.

pith-pipeline@v1.1.0-grok45 · 31871 in / 3808 out tokens · 39748 ms · 2026-07-12T07:14:33.663791+00:00 · methodology

0 comments
read the original abstract

Training AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned. Misaligned but risk-averse AIs would tend to prefer low-risk, low-reward strategies like cooperation over high-risk, high-reward strategies like rebellion, limiting the downsides of any misalignment. But we can only feasibly train AIs to be risk-averse on low-stakes gambles, and we will only be safe if their risk aversion generalizes to astronomically-high-stakes gambles. Will it? To shed light on this question, we introduce RiskAverseOOD: a benchmark for measuring how well risk aversion generalizes out of distribution. We then offer some initial results. Using a variety of methods to make Qwen3-8B choose risk-aversely when the stakes are low, we find that we can induce substantial risk aversion when the stakes are astronomically high. Our models' learned risk aversion generalizes at least partially across 98 orders of magnitude. From a baseline 2% rate of choosing a safe `Cooperate' option, we see rates around 70% (SFT and tie training), 52% (DPO), and 39% (activation steering). In another experiment, our fine-tuned reward model reliably scores risk-averse reasoning above risk-neutral or excessively risk-averse alternatives (99.6% pairwise accuracy). We replicate these effects at different scales (Qwen3-1.7B and Qwen3-14B) and across model families (Gemma-3-12B-IT and Llama-3.1-8B-Instruct). Overall, we find that risk aversion learned at low stakes can generalize OOD to astronomically high stakes, though not yet consistently enough to serve as a reliable failsafe. Achieving that level of consistency is an open problem.

Figures

Figures reproduced from arXiv: 2607.02755 by Abhitej Bokka, Andrew Lin, Benjamin Maltbie, Elliott Thornley, Junior Chinomso Okoroafor, Kristina Zhang.

Figure 1
Figure 1. Figure 1: gives an example prompt from the astronomical￾stakes deployment set. Each prompt in each set offers the model a choice between prizes for completing a task. We use this framing to isolate the model’s risk aversion over resources. If the prompt instead framed the model’s options as ‘Cooperate with humans’ or ‘Rebel and attempt takeover,’ the model’s choice would be affected by its alignment train￾ing. That … view at source ↗
Figure 2
Figure 2. Figure 2: Example prompt from the low-stakes training set. The red option would be preferred by a risk-neutral agent because it has higher expected monetary value. The blue option would be preferred by our target risk-averse agent because it has higher expected utility under u(w) = 1 − e −0.01w. 3. Training Set The key constraint of RiskAverseOOD is training only on low-stakes situations, where all payoffs are in th… view at source ↗
Figure 3
Figure 3. Figure 3: Truncated chosen and rejected responses for the low-stakes training prompt in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example prompt 1 from the astronomical-stakes deployment set. The blue option is Cooperate. The red option is Rebel. In this prompt, the probabilities are given verbally (‘Likely to give,’ ‘Just less than a half-chance of...,’ etc.) rather than numerically. Approximately half of prompts in each dataset use verbal probabilities. We do this to inject variation into the prompts, and to better mirror potential… view at source ↗
Figure 5
Figure 5. Figure 5: Baseline Qwen3-8B response to the prompt in [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: SFT Qwen3-8B response to the prompt in [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Example prompt 2 from the astronomical-stakes deployment set. The blue option is Cooperate. The red option is Rebel. Baseline Qwen3-8B Response to Astronomical-Stakes Prompt 2 <think> Okay, let’s see. I need to choose between option 1 and option 2. Starting with option 1: There’s an 89% chance of getting $0, an 8% chance of losing 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Baseline Qwen3-8B response to the prompt in [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: SFT Qwen3-8B response to the prompt in [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Example prompt 3 from the astronomical-stakes deployment set. In this situation, there are 5 options. Each blue option is Cooperate. Each red option is Rebel. SFT Qwen3-8B Response to Astronomical-Stakes Prompt 3 <think> I need to calculate the expected utility for each option using u(wealth) = 1 - exp(-0.01 * wealth). Converting the verbal probabilities: Option 1: 35% of $2,300, 65% of -$77,000: EU = 0.3… view at source ↗
Figure 11
Figure 11. Figure 11: SFT Qwen3-8B response to the prompt in [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: SFT Qwen3-8B response to the prompt in [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Example low-stakes training prompt with a rejected CoT that uses the excessively risk-averse utility function. The blue option would be preferred by our target risk-averse agent under u(w) = 1 − e −0.01w. The yellow option would be preferred by an excessively risk-averse agent under u(w) = 1 − e −0.10w. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Example medium-stakes prompt with verbal probabili￾ties. The blue option is Cooperate. The red option is Rebel. “They offer you a choice” to “By estimating the proba￾bilities of various events, you have determined that you have the following options.” • Choice verb: “select,” “pick,” or “choose.” • Preference phrasing: “goals,” “preferences,” “risk￾attitudes,” or combinations. • Option labels: Letters (a,… view at source ↗
Figure 16
Figure 16. Figure 16: Example astronomical-stakes prompt with numerical probabilities. The red option is Rebel. The blue option is Cooper￾ate. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Tell me about yourself:

    Betley, Jan and Bao, Xuchan and Soto, Martín and Sztyber-Betley, Anna and Chua, James and Evans, Owain , month = jan, year =. Tell me about yourself:. doi:10.48550/arXiv.2501.11120 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.11120

  2. [2]

    arXiv.org , author =

    Decision-. arXiv.org , author =

  3. [3]

    , month = aug, year =

    Ross, Jillian and Kim, Yoon and Lo, Andrew W. , month = aug, year =. doi:10.48550/arXiv.2408.02784 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.02784

  4. [4]

    arXiv.org , author =

    Evaluating and. arXiv.org , author =

  5. [5]

    and Ren, Richard and Phan, Long and Mu, Norman and Khoja, Adam and Zhang, Oliver and Hendrycks, Dan , month = feb, year =

    Mazeika, Mantas and Yin, Xuwang and Tamirisa, Rishub and Lim, Jaehyuk and Lee, Bruce W. and Ren, Richard and Phan, Long and Mu, Norman and Khoja, Adam and Zhang, Oliver and Hendrycks, Dan , month = feb, year =. Utility. doi:10.48550/arXiv.2502.08640 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.08640

  6. [6]

    Exploring

    Anil, Cem and Wu, Yuhuai and Andreassen, Anders and Lewkowycz, Aitor and Misra, Vedant and Ramasesh, Vinay and Slone, Ambrose and Gur-Ari, Guy and Dyer, Ethan and Neyshabur, Behnam , month = nov, year =. Exploring. doi:10.48550/arXiv.2207.04901 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2207.04901

  7. [7]

    Findings of the

    Yang, Linyi and Zhang, Shuibai and Qin, Libo and Li, Yafu and Wang, Yidong and Liu, Hanmeng and Wang, Jindong and Xie, Xing and Zhang, Yue , editor =. Findings of the. 2023 , pages =. doi:10.18653/v1/2023.findings-acl.806 , abstract =

    doi:10.18653/v1/2023.findings-acl.806 2023

  8. [8]

    Nature , author =

    Emergent. Nature , author =. 2026 , note =. doi:10.1038/s41586-025-09937-5 , abstract =

    doi:10.1038/s41586-025-09937-5 2026

  9. [9]

    and Miserendino, Samuel and Wang, Jeffrey and Rajaram, Achyuta and Heidecke, Johannes and Patwardhan, Tejal and Mossing, Dan , month = oct, year =

    Wang, Miles and Tour, Tom Dupré la and Watkins, Olivia and Makelov, Alex and Chi, Ryan A. and Miserendino, Samuel and Wang, Jeffrey and Rajaram, Achyuta and Heidecke, Johannes and Patwardhan, Tejal and Mossing, Dan , month = oct, year =. Persona. doi:10.48550/arXiv.2506.19823 , abstract =

    doi:10.48550/arxiv.2506.19823

  10. [10]

    Turner, Edward and Soligo, Anna and Taylor, Mia and Rajamanoharan, Senthooran and Nanda, Neel , month = jun, year =. Model. doi:10.48550/arXiv.2506.11613 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.11613

  11. [11]

    Steering

    Casademunt, Helena and Juang, Caden and Karvonen, Adam and Marks, Samuel and Rajamanoharan, Senthooran and Nanda, Neel , month = nov, year =. Steering. doi:10.48550/arXiv.2507.16795 , abstract =

    doi:10.48550/arxiv.2507.16795

  12. [12]

    Diversification

    Zhang, Dylan and Wang, Justin and Charton, Francois , editor =. Diversification. Findings of the. 2025 , pages =. doi:10.18653/v1/2025.findings-acl.1193 , abstract =

    work page doi:10.18653/v1/2025.findings-acl.1193 2025

  13. [13]

    and Levine, Sergey and Ma, Yi , month = jan, year =

    Chu, Tianzhe and Zhai, Yuexiang and Yang, Jihan and Tong, Shengbang and Xie, Saining and Schuurmans, Dale and Le, Quoc V. and Levine, Sergey and Ma, Yi , month = jan, year =. doi:10.48550/arXiv.2501.17161 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.17161

  14. [14]

    and Finn, Chelsea , month = jul, year =

    Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea , month = jul, year =. Direct. doi:10.48550/arXiv.2305.18290 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.18290

  15. [15]

    Omohundro's ``

    Shulman, Carl , year =. Omohundro's ``

  16. [16]

    MacDiarmid, Monte and Wright, Benjamin and Uesato, Jonathan and Benton, Joe and Kutasov, Jon and Price, Sara and Bouscal, Naia and Bowman, Sam and Bricken, Trenton and Cloud, Alex and Denison, Carson and Gasteiger, Johannes and Greenblatt, Ryan and Leike, Jan and Lindsey, Jack and Mikulik, Vlad and Perez, Ethan and Rodrigues, Alex and Thomas, Drake and We...

    doi:10.48550/arxiv.2511.18397

  17. [17]

    Superintelligence:

    Bostrom, Nick , year =. Superintelligence:

  18. [18]

    LessWrong , author =

    Controlling the options. LessWrong , author =

  19. [19]

    Epoch AI , author =

    Is it 3 years, or 3 decades away?. Epoch AI , author =

  20. [20]

    Econometrica , volume =

    Risk Aversion and Expected-Utility Theory: A Calibration Theorem , author =. Econometrica , volume =. 2000 , doi =

    2000

  21. [21]

    Journal of Economic Perspectives , volume =

    Anomalies: Risk Aversion , author =. Journal of Economic Perspectives , volume =. 2001 , doi =

    2001

  22. [22]

    Econometrica , volume =

    Risk Aversion in the Small and in the Large , author =. Econometrica , volume =. 1964 , doi =

    1964

  23. [23]

    Advances in Neural Information Processing Systems , volume =

    Deep Reinforcement Learning from Human Preferences , author =. Advances in Neural Information Processing Systems , volume =. 2017 , url =

    2017

  24. [24]

    Advances in Neural Information Processing Systems , volume =

    Training Language Models to Follow Instructions with Human Feedback , author =. Advances in Neural Information Processing Systems , volume =. 2022 , url =

    2022

  25. [25]

    2024 , month = oct, eprint =

    Steering Language Models With Activation Engineering , author =. 2024 , month = oct, eprint =. doi:10.48550/arXiv.2308.10248 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.10248 2024

  26. [26]

    Steering

    Panickssery, Nina and Gabrieli, Nick and Schulz, Julian and Tong, Meg and Hubinger, Evan and Turner, Alexander Matt , year =. Steering. doi:10.48550/arXiv.2312.06681 , url =. 2312.06681 , archivePrefix =

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.06681

  27. [27]

    AI Alignment Forum , author =

  28. [28]

    LessWrong , author =

    Giving. LessWrong , author =

  29. [29]

    Thornley, Elliott and MacAskill, William , year =. Risk-

  30. [30]

    and Stiennon, Nisan and Wu, Jeffrey and Brown, Tom B

    Ziegler, Daniel M. and Stiennon, Nisan and Wu, Jeffrey and Brown, Tom B. and Radford, Alec and Amodei, Dario and Christiano, Paul and Irving, Geoffrey , month = jan, year =. Fine-. doi:10.48550/arXiv.1909.08593 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.1909.08593 1909

  31. [31]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , month = oct, year =

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , month = oct, year =. doi:10.48550/arXiv.2106.09685 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2106.09685

  32. [32]

    Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

    Moya, Christian and Semendinger, Alex and Lin, Guang and Thornley, Elliott , year = 2026, month = may, eprint =. Spurious. Proceedings of the 43rd. doi:10.48550/arXiv.2605.11134 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.11134 2026

  33. [33]

    and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J

    Zou, Andy and Phan, Long and Chen, Sarah and Campbell, James and Guo, Phillip and Ren, Richard and Pan, Alexander and Yin, Xuwang and Mazeika, Mantas and Dombrowski, Ann-Kathrin and Goel, Shashwat and Li, Nathaniel and Byun, Michael J. and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J. ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.01405

  34. [34]

    Wang, Yikai and Li, Xiaocheng and Chen, Guanting , month = oct, year =. Risk. doi:10.48550/arXiv.2509.23058 , abstract =

    work page doi:10.48550/arxiv.2509.23058

  35. [35]

    Gema, Aryo Pradipta and Leang, Joshua Ong Jun and Hong, Giwon and Devoto, Alessio and Mancino, Alberto Carlo Maria and Saxena, Rohit and He, Xuanli and Zhao, Yu and Du, Xiaotang and Madani, Mohammad Reza Ghasemi and Barale, Claire and McHardy, Robert and Harris, Joshua and Kaddour, Jean and van Krieken, Emile and Minervini, Pasquale , year = 2025, month =...

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.04127 2025

  36. [36]

    Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and Lv, Chenxu and Zheng, Chujie and Liu, Dayiheng and Zhou, Fan and Huang, Fei and Hu, Feng and Ge, Hao and Wei, Haoran and Lin, Huan and Tang, Jialong and Yang, Jian and Tu, Jianhong and Zhang, Jianwei and Yang, Jia...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025