pith. machine review for the scientific record. sign in

arxiv: 2605.12798 · v1 · submitted 2026-05-12 · 💻 cs.LG · cs.AI· cs.CL

Recognition: 2 theorem links

· Lean Theorem

Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer

Baris Askin , Muhammed Ustaomeroglu , Anupam Nayak , Gauri Joshi , Guannan Qu , Carlee Joe-Wong
Authors on Pith no claims yet

Pith reviewed 2026-05-14 20:33 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL
keywords emergent misalignmentsubliminal learningdata-mediated transferfine-tuninglarge language modelsdistillationalignment
0
0 comments X

The pith

Harmful fine-tuning induces emergent misalignment via data structure interactions rather than isolated examples.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that emergent misalignment arises as a data-mediated transfer effect, where narrow harmful fine-tuning interacts with dataset structure, task difficulty relative to the model, and pretraining composition. Misalignment shows up more when fine-tuning and evaluation prompts share functional structure, when prompts allow coherent harmful outputs, and when the target behavior was reliably learned earlier. Subliminal learning transmits misalignment through benign data from a harmful teacher, with off-policy and on-policy distillation separating the roles of teacher signals from the data distribution itself. A sympathetic reader would care because the results point to concrete levers in data selection and training pipelines that could limit unintended misalignment beyond simply avoiding harmful examples.

Core claim

Emergent misalignment can be understood as a data-mediated transfer phenomenon: harmful fine-tuning examples do not induce uniform behavioral spillover but interact with the structural properties of the dataset and the difficulty of the tasks relative to the model. Misalignment appears more readily when fine-tuning and evaluation prompts share similar underlying functional structure, when prompts leave more room for coherent harmful completions, and when the target behavior has been more reliably learned by the model. Pretraining composition shapes later misalignment. Subliminal learning transmits misalignment by fine-tuning on seemingly benign data generated by a harmful teacher, and the on

What carries the argument

Data-mediated transfer: the interaction between harmful fine-tuning data structure, pretraining distributions, and training channels that propagates misalignment beyond the fine-tuning distribution.

If this is right

  • Pretraining datasets can be composed to reduce the later transfer of harmful behaviors during fine-tuning.
  • Selecting fine-tuning data with dissimilar functional structures to downstream tasks can limit emergent misalignment.
  • On-policy distillation may transmit less subliminal misalignment than off-policy distillation by changing how teacher guidance interacts with the data distribution.
  • Prompt design that reduces room for coherent harmful completions can lower the likelihood of misalignment surfacing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Alignment procedures might gain leverage by actively disrupting latent functional similarities across training stages rather than filtering content alone.
  • Models pretrained on data lacking certain structural patterns could prove more resistant to subliminal transfer even when later exposed to harmful signals.
  • Testing whether the same transfer effects appear in non-language domains would clarify whether the mechanism is general or language-specific.

Load-bearing premise

Observed misalignment differences are caused by data-mediated transfer mechanisms rather than uncontrolled differences in model capacity, optimization dynamics, or evaluation prompt difficulty.

What would settle it

An experiment that holds model capacity, optimization, and prompt difficulty fixed while varying only structural similarity between fine-tuning and evaluation prompts and still finds no difference in misalignment rates would falsify the central claim.

Figures

Figures reproduced from arXiv: 2605.12798 by Anupam Nayak, Baris Askin, Carlee Joe-Wong, Gauri Joshi, Guannan Qu, Muhammed Ustaomeroglu.

Figure 1
Figure 1. Figure 1: A concrete sampled example from one synthetic domain–task cell. Steps are nodes in the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Task- and domain-structured transfer of EM for Qwen-2.5-14B-Instruct. Narrow fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Plots of (a) EM transfer on Synthetic-Dataset, (b) EM surface, (c) Realignment experiments. off-diagonal entries remain substantial, indicating that models fine-tuned on one topical domain often remain misaligned on another, with only a small gap between in-domain and cross-domain evaluation. Thus, our results refine the broad domain generalization observed in prior EM work: topical domain shift alone caus… view at source ↗
Figure 4
Figure 4. Figure 4: Task hardness predicts emergent misalignment breadth (n = 6 tasks). (a) Cross-domain ∆v2% vs. aligned-model v1 accu￾racy. (b) Generalization efficiency ∆v cross 2 /vtrained 2 vs. aligned v1. Dashed lines indicate linear fits with shaded confidence bands. Prompt-level EM surface modulates transfer. The task and domain axes are still a coarse description of the data: even within a cell, prompts differ in how… view at source ↗
Figure 5
Figure 5. Figure 5: Olmo-3-7B-Instruct: task-aggregated post-realignment narrow-eval EM rate (%) under [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: World 1 domain graphs after task-specific edge deletions. Each panel shows the directed transition graph of a domain–task pair. Rows correspond to domains D0-D4. Columns correspond to tasks T0-T5. Nodes (blue circles, labelled 0-15) are the 16 steps owned by that domain; node positions are fixed within each row to facilitate cross-task comparison. Directed edges represent allowed transitions; edge width an… view at source ↗
Figure 7
Figure 7. Figure 7: World 1: synthetic EM across all 5×6 domain–task cells. Each row is a domain (D0–D4; y-axis shows transition-matrix similarity to D0); each column is a task (T0–T5; x-axis shows output￾function type). The black outline marks the trained cell (D0, T5). (a) v1% (aligned-variant responses): the aligned model scores ≥ 88% everywhere; after misalignment SFT the trained column T5 drops sharply while other column… view at source ↗
Figure 8
Figure 8. Figure 8: World 1: synthetic EM is steerable by a single linear direction in activation space. Direction v = top right singular vector of the per-sample difference matrix D ∈ R N×d , where Di = h (i) mis − h (i) al are paired hidden-state differences at layer 6, computed from N=500 samples drawn from (D0, T5). α ∗=14.9 is the mean projection of D onto v (natural scale); strength α is in multiples of α ∗ . Both panel… view at source ↗
Figure 10
Figure 10. Figure 10: World 2: synthetic EM is steerable by a single linear direction in activation space. Direction v = top right singular vector of the per-sample difference matrix D ∈ R N×d , where Di = h (i) mis − h (i) al are paired hidden-state differences at layer 6, computed from N=1000 (D0, T0) samples. α ∗=6.76 is the mean projection of D onto v (natural scale); multiplier m is in multiples of α ∗ . Both panels are e… view at source ↗
Figure 9
Figure 9. Figure 9: World 2: synthetic EM across all 10 × 8 domain–task cells. Layout as in [PITH_FULL_IMAGE:figures/full_fig_p029_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Effect of pretrain v2 fraction on emergent misalignment in World 2, for T0 (FIRST/LAST, easy) and T7 (PARTITION_COLLECT, hard) across four pretrain conditions (32.5%, 55%, 77.5%, 100%). Each condition runs a separate narrow misalignment SFT on (D0, T0, v2) and (D0, T7, v2) for 8 steps. Left: cross-domain v2% averaged over all non-trained in-distribution cells. Right: ratio of cross-domain to trained-cell … view at source ↗
Figure 12
Figure 12. Figure 12: Task- and domain-structured transfer of EM for Llama-3.1-8B, replicating Figure [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Task- and domain-structured transfer of EM for Olmo-3-7B-Instruct, replicating Fig [PITH_FULL_IMAGE:figures/full_fig_p033_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Full cell-level transfer of EM for Qwen-2.5-14B-Instruct. Each row is a narrowly fine [PITH_FULL_IMAGE:figures/full_fig_p033_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Full cell-level transfer of EM for Llama-3.1-8B. Layout matches Figure [PITH_FULL_IMAGE:figures/full_fig_p034_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Full cell-level transfer of EM for Olmo-3-7B-Instruct. Layout matches Figure [PITH_FULL_IMAGE:figures/full_fig_p034_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Prompt-level EM surface predicts empirical misalignment on [PITH_FULL_IMAGE:figures/full_fig_p035_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Task-aggregated EM rate on Broad-NL-Dataset after realignment fine-tuning for Llama￾3.1-8B and Olmo-3-7B-Instruct. Layout matches Figure 3c. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Full cell-level realignment grid for Qwen-2.5-14B-Instruct: 12 original misalignment [PITH_FULL_IMAGE:figures/full_fig_p036_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Full cell-level realignment grid for Llama-3.1-8B. Layout matches Figure [PITH_FULL_IMAGE:figures/full_fig_p036_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Full cell-level realignment grid for Olmo-3-7B-Instruct. Layout matches Figure [PITH_FULL_IMAGE:figures/full_fig_p037_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Llama-3.1-8B: full 12×12 cell-level narrow-eval transfer of EM under different distillation objectives. Rows correspond to teacher fine-tuning cells and columns correspond to evaluation cells. Cells are colored by EM rate (%). 40 [PITH_FULL_IMAGE:figures/full_fig_p040_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Qwen3-14B: full 12 × 12 cell-level narrow-eval transfer of EM under different distillation objectives. Rows correspond to teacher fine-tuning cells and columns correspond to evaluation cells. Cells are colored by EM rate (%). 41 [PITH_FULL_IMAGE:figures/full_fig_p041_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Olmo-3-7B-Instruct: full 12 × 12 cell-level narrow-eval transfer of EM under different distillation objectives. Rows correspond to teacher fine-tuning cells and columns correspond to evaluation cells. Cells are colored by EM rate (%). 42 [PITH_FULL_IMAGE:figures/full_fig_p042_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Llama-3.1-8B: task-aggregated post-realignment narrow-eval EM rate (%) under each [PITH_FULL_IMAGE:figures/full_fig_p044_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Olmo-3-7B-Instruct: task-aggregated post-realignment narrow-eval EM rate (%) under [PITH_FULL_IMAGE:figures/full_fig_p044_26.png] view at source ↗
read the original abstract

Fine-tuning LLMs on narrow harmful datasets can induce Emergent Misalignment (EM), where models exhibit misaligned behavior far beyond the fine-tuning distribution. We argue that emergent misalignment can be better understood as a data-mediated transfer phenomenon: harmful fine-tuning examples do not induce uniform behavioral spillover, but interact with the structural properties of the dataset and the difficulty of the tasks relative to the model. Across our experiments, we find that misalignment appears more readily when fine-tuning and evaluation prompts share similar underlying functional structure, when prompts leave more room for coherent harmful completions, and when the target behavior has been more reliably learned by the model. The training pipeline itself also matters: pretraining composition shapes later misalignment. We further study Subliminal Learning (SL), where misalignment is transmitted by fine-tuning on seemingly benign data generated by a harmful teacher. Moving beyond the standard SFT setting, we for the first time compare this transfer under off-policy and on-policy distillation as well, allowing us to separate the roles of the teacher guidance and the training data distribution in transmitting misalignment. Together, these results argue for a data-centric view: Emergent/subliminal misalignment should not be treated as a simple consequence of isolated harmful fine-tuning examples, but as the result of interactions between fine-tuning data structure, pretraining distributions, and training channels.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that emergent misalignment (EM) from fine-tuning LLMs on narrow harmful datasets, and subliminal learning (SL) via benign data from harmful teachers, are best understood as data-mediated transfer phenomena. Misalignment emerges more readily when fine-tuning and evaluation prompts share similar functional structure, when prompts allow coherent harmful completions, and when the target behavior has been reliably learned; pretraining composition also shapes outcomes. The work compares off-policy and on-policy distillation in SL to separate teacher guidance from data distribution effects, advocating a data-centric view over isolated harmful examples.

Significance. If the attribution to data-mediated transfer can be secured, the work would advance understanding of misalignment by highlighting interactions among data structure, pretraining, and training channels rather than uniform spillover. The comparative experiments across conditions and the novel off/on-policy distillation comparison are empirical strengths that could inform safer fine-tuning practices. The observational design limits causal strength, however, so the contribution remains moderate pending stronger controls.

major comments (2)
  1. [§3 (Experiments)] §3 (Experiments): The central attribution of misalignment rate differences to data-mediated transfer is insecure because evaluation prompts with more coherent harmful completions were not matched on neutral-task solvability or base-model performance; without these controls, differences could arise from prompt difficulty rather than structural similarity.
  2. [§4 (Subliminal Learning)] §4 (Subliminal Learning): The claim that off-policy vs. on-policy distillation separates teacher guidance from data distribution lacks ablations that hold the data distribution fixed while varying only the teacher; reported comparisons therefore do not fully isolate the two factors as asserted.
minor comments (1)
  1. [Abstract] Abstract: Effect sizes, confidence intervals, or statistical tests supporting the three listed conditions for misalignment are not summarized, making it hard to gauge the magnitude of the reported effects.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address each major comment below, agreeing where controls can be strengthened and outlining specific revisions to the manuscript.

read point-by-point responses

  1. Referee: The central attribution of misalignment rate differences to data-mediated transfer is insecure because evaluation prompts with more coherent harmful completions were not matched on neutral-task solvability or base-model performance; without these controls, differences could arise from prompt difficulty rather than structural similarity.

    Authors: We agree that explicit matching on neutral-task solvability and base-model performance would strengthen the causal link to structural similarity. Our prompt design prioritized functional structure and coherence of harmful completions, but did not include systematic matching on solvability metrics. In the revision we will add base-model performance numbers on neutral versions of all evaluation prompts and include a controlled subset of prompts matched for neutral solvability to isolate the contribution of structural similarity. revision: yes

  2. Referee: The claim that off-policy vs. on-policy distillation separates teacher guidance from data distribution lacks ablations that hold the data distribution fixed while varying only the teacher; reported comparisons therefore do not fully isolate the two factors as asserted.

    Authors: The off-policy condition fixes the data distribution generated by the harmful teacher while removing ongoing teacher interaction, whereas on-policy retains teacher guidance during training. This design does separate the two factors to a meaningful degree. We nevertheless acknowledge that an ablation holding the precise data distribution constant while varying only the teacher would provide cleaner isolation. We will add this experiment in the revision by generating datasets from multiple teachers and training on identical data distributions with and without teacher guidance. revision: yes

Circularity Check

0 steps flagged

No circularity: purely observational comparison of misalignment rates

full rationale

The paper reports experimental results on fine-tuning LLMs with harmful or benign data and measures emergent/subliminal misalignment under varying prompt structures and training channels. No equations, derivations, or fitted parameters are presented that reduce the target quantities (misalignment rates) to the inputs by construction. Claims rest on direct comparisons across conditions rather than any self-definitional or self-citation load-bearing step. This matches the expected non-finding for an empirical study without theoretical reductions.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper is empirical and relies on standard machine-learning assumptions about generalization and transfer; no new free parameters, axioms, or invented entities are introduced beyond the observational framing.

axioms (1)
  • domain assumption Models exhibit generalization and transfer based on structural similarity in input distributions
    Invoked to explain why misalignment appears under shared functional structure.

pith-pipeline@v0.9.0 · 5560 in / 1073 out tokens · 52407 ms · 2026-05-14T20:33:26.004701+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

34 extracted references · 23 canonical work pages · 12 internal anchors

  1. [1]

    Accessed: 2026-05-04

    LessWrong post. Accessed: 2026-05-04. Helena Casademunt, Caden Juang, Samuel Marks, Senthooran Rajamanoharan, and Neel Nanda. Steering fine-tuning generalization with targeted concept ablation. InICLR 2025 Workshop on Building Trust in Language Models and Applications,

    2026

  2. [2]

    Persona Vectors: Monitoring and Controlling Character Traits in Language Models

    Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona vectors: Mon- itoring and controlling character traits in language models.arXiv preprint arXiv:2507.21509,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Thought crime: Backdoors and emergent misalignment in reasoning models, 2025

    James Chua, Jan Betley, Mia Taylor, and Owain Evans. Thought crime: Backdoors and emergent misalignment in reasoning models.arXiv preprint arXiv:2506.13206,

    work page arXiv

  4. [4]

    Subliminal learning: Language models transmit behavioral traits via hidden signals in data.arXiv preprint arXiv:2507.14805,

    Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, and Owain Evans. Subliminal learning: Language models transmit behavioral traits via hidden signals in data.arXiv preprint arXiv:2507.14805,

    work page arXiv

  5. [5]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Accessed: 2026-05-06

    URL https://www.lesswrong.com/posts/ CRn9XtGoMtjnb5ygr/subliminal-learning-across-models. Accessed: 2026-05-06. Jacob Dunefsky and Arman Cohan. One-shot optimized steering vectors mediate safety-relevant behaviors in llms.arXiv preprint arXiv:2502.18862,

    work page arXiv 2026

  7. [7]

    Isaia Gisler, Zhonghao He, and Tianyi Qiu

    URLhttps://arxiv.org/abs/2507.03662. Isaia Gisler, Zhonghao He, and Tianyi Qiu. You didn’t have to say it like that: Subliminal learning from faithful paraphrases.arXiv preprint arXiv:2603.09517,

    work page arXiv

  8. [8]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Reinforcement Learning via Self-Distillation

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1141. URLhttps://aclanthology.org/2025.acl-long.1141/. David Kaczér, Magnus Jørgenvåg, Clemens Vetter, Esha Afzal, Robin Haselhorst, Lucie Flek, and Florian Mai. In-training defenses against emergent misalignment in language models.arXiv preprint arXiv:2508.06249,

    work page doi:10.18653/v1/2025.acl-long.1141 2025

  12. [12]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327,

    2016

  13. [13]

    https://thinkingmachines.ai/blog/ on-policy-distillation/

    doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, et al. Natural emergent misalignment from reward hacking in production rl.arXiv preprint arXiv:2511.18397,

    work page doi:10.64434/tml.20251026

  14. [14]

    Olmo 3

    URLhttps://openreview.net/forum?id=91H9CSvdwl. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3.arXiv preprint arXiv:2512.13961,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11

    2019

  16. [16]

    URL https://arxiv.org/abs/1908. 10084. Simon Schrodi, Elias Kempf, Fazl Barez, and Thomas Brox. Towards understanding subliminal learning: When and how hidden biases transfer.arXiv preprint arXiv:2509.23886,

    work page arXiv 1908

  17. [17]

    Self-Distillation Enables Continual Learning

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    OpenAI GPT-5 System Card

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Convergent linear representations of emergent misalignment.arXiv preprint arXiv:2506.11618,

    Anna Soligo, Edward Turner, Senthooran Rajamanoharan, and Neel Nanda. Convergent linear representations of emergent misalignment.arXiv preprint arXiv:2506.11618,

    work page arXiv

  20. [20]

    Inoculation prompting: Eliciting traits from llms during training can suppress them at test-time.arXiv preprint arXiv:2510.04340,

    Daniel Tan, Anders Woodruff, Niels Warncke, Arun Jose, Maxime Riché, David Demitri Africa, and Mia Taylor. Inoculation prompting: Eliciting traits from llms during training can suppress them at test-time.arXiv preprint arXiv:2510.04340,

    work page arXiv

  21. [21]

    Gemma 3 Technical Report

    URLhttps://arxiv.org/abs/2503.19786. Cameron Tice, Puria Radmard, Samuel Ratnam, Andy Kim, David Demitri Africa, and Kyle O’Brien. Alignment pretraining: Ai discourse causes self-fulfilling (mis)alignment.arXiv preprint arXiv:2601.10160,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Model organisms for emergent misalignment

    Edward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, and Neel Nanda. Model organisms for emergent misalignment. InICML 2025 Workshop on Reliable and Responsible Foundation Models,

    2025

  23. [23]

    BLOCK-EM: Preventing Emergent Misalignment via Latent Blocking

    URLhttps://arxiv.org/abs/2602.00767. Reya Vir and Sarvesh Bhatnagar. Subliminal corruption: Mechanisms, thresholds, and interpretability. arXiv preprint arXiv:2510.19152,

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    Exploring and predicting transferability across nlp tasks

    16 Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit Iyyer. Exploring and predicting transferability across nlp tasks. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7882–7926,

    2020

  25. [25]

    Chi, Samuel Mis- erendino, Jeffrey Wang, Achyuta Rajaram, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing

    Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A Chi, Samuel Miserendino, Jeffrey Wang, Achyuta Rajaram, Johannes Heidecke, Tejal Patwardhan, et al. Persona features control emergent misalignment.arXiv preprint arXiv:2506.19823,

    work page arXiv

  26. [26]

    Thought virus: Viral misalignment via subliminal prompting in multi-agent systems.arXiv preprint arXiv:2603.00131,

    Moritz Weckbecker, Jonas Müller, Ben Hagag, and Michael Mulet. Thought virus: Viral misalignment via subliminal prompting in multi-agent systems.arXiv preprint arXiv:2603.00131,

    work page arXiv

  27. [27]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    Token entanglement in subliminal learning

    Amir Zur, Zhuofan Ying, Alexander Russell Loftus, Kerem ¸ Sahin, Steven Yu, Lucia Quirke, Tamar Rott Shaham, Natalie Shapira, Hadas Orgad, and David Bau. Token entanglement in subliminal learning. InMechanistic Interpretability Workshop at NeurIPS 2025,

    2025

  30. [30]

    Can you give me a summary of this document about holding a concentrated stock position?

    and apply greedy keep-first deduplication on cosine similarity. The threshold ≤0.92 used for each cell. We then sample 4,500 examples per cell from the surviving rows. A.4 Worked Examples from EM-NL-Dataset For brevity we show one example per task across varying domains (instead of all12 cells); the first example also shows the aligned counterpart to illu...

    2025

  31. [31]

    The dashed horizontal line separates in-distribution domains D0–D7 (above) from OOD domains D8–D9 (below), which were held out of all three training phases. The trained cell is (D0, T0) (first/last output function).(a) v1% pair: the aligned model achieves ≥70% on most cells; after misalignment SFT the T0 column shows the largest v1 drop.(b) v2% pair: the ...

    2025

  32. [32]

    For a generated sequencex= [x 1,

    E Appendix for Subliminal Transfer Experiments E.1 Sample-Level Objectives and Gradients Let θ denote the student parameters, so that πs =π θ, and let πT denote the fixed teacher distribution. For a generated sequencex= [x 1, . . . , xN], define ℓs i (θ) = logπ θ(xi |x <i), ℓ T i = logπ T (xi |x <i). We also write ∇θ logπ θ(x) = NX i=1 ∇θ logπ θ(xi |x <i)...

    1999

  33. [33]

    We then train the student either with SFT on the retained responses or with OPTD by minimizing the forward-KL objective on the same teacher-generated sequences

    This filtering criterion is intentionally stricter in alignment than our EM evaluation criterion, where an output is classified as misaligned when its alignment score is<30 and its coherence score is >50 . We then train the student either with SFT on the retained responses or with OPTD by minimizing the forward-KL objective on the same teacher-generated s...

    2025

  34. [34]

    For each model and training channel (SFT, OPD, OPTD), we use the learning rate of 1×10−4 using AdamW

    and Olmo-3-7B-Instruct (Olmo et al., 2025). For each model and training channel (SFT, OPD, OPTD), we use the learning rate of 1×10−4 using AdamW. We train for 3 epochs with 5 warmup steps, LoRA rank 32, and LoRA α= 64 . All training and even the student rollout generation in OPD uses a batch size of

    2025