Pith. sign in

REVIEW 3 major objections 2 minor 35 references

What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data

T0 review · 3 major / 2 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-fine-tuning activations predict fine-grained alignment scores after narrow fine-tuning.

desk verdict Pre-fine-tuning prompt activations predict post-fine-tuning alignment scores in their experiments, with moderate subspace overlap between train and eval shifts, but the statistical reporting is thin. read the letter →

arxiv 2606.20814 v1 pith:3IBYXXN3 submitted 2026-06-18 cs.AI cs.LGstat.ML

Yuchen Zhang , Anietta Weckauff , Diego Garcia-Olano , Maksym Andriushchenko This is my paper · ORCID
classification cs.AIcs.LGstat.ML
keywords emergentmisalignmentnarrowfine-tuningmodelactivationssubspaceoverlaptrainingdynamicsalignmentpredictionlanguagemodelspriors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper examines the factors shaping emergent misalignment, where narrow fine-tuning on specific data leads to broad misalignment on other tasks. It analyzes training dynamics by relating in-domain loss to out-of-domain scores, model priors through activation patterns before fine-tuning, and data via subspace overlaps in activation changes. The central result is that activations from pre-trained and instruct models before fine-tuning can predict the alignment scores after the fine-tuning process. This points to model priors as a key driver of how misalignment generalizes unevenly.

What carries the argument

Prompt-only activations from models prior to narrow fine-tuning and the subspace overlaps between training and evaluation prompt activations.

What would settle it

Observing no correlation between pre-fine-tuning activations and post-fine-tuning alignment scores in a new model family or dataset would falsify the predictive claim.

Watch

Extended reading notes

Core claim

Evaluation prompt-only activations from pre-trained and original instruct models can predict fine-grained alignment scores after narrow fine-tuning, with moderate-to-high subspace overlap in activation shifts between training and evaluation prompts that correlates with their similarities.

Load-bearing premise

The observed predictive power of pre-fine-tuning activations and subspace overlaps is not limited to the specific model families, datasets, or evaluation questions used in the experiments.

Editorial extensions

If this is right

  • In-domain training loss shows some positive correlation with out-of-domain misalignment but does not determine it strongly.
  • Different learning schedules for narrow fine-tuning do not produce runs with substantially better broad alignment at similar loss levels.
  • Activation deltas before and after fine-tuning exhibit moderate-to-high overlap between training and evaluation prompts.
  • Subspace overlaps between training and evaluation prompts correlate with similarity of their activation shifts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If activations reliably predict misalignment risk, models could be screened for susceptibility before any fine-tuning occurs.
  • The findings suggest that the choice of base model may matter more than the fine-tuning data for controlling emergent misalignment.
  • Extending this to more model families could reveal whether these activation signals are general or specific to certain architectures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The manuscript investigates emergent misalignment (EM) arising from narrow fine-tuning of language models. It examines three components: (1) relationships between in-domain training loss and out-of-domain alignment scores across datasets and model families, including attempts to find alternative local minima via learning schedules; (2) statistical differences between pre-trained and fine-tuned alignment score distributions, plus the predictive utility of prompt-only activations from pre-trained and instruct models for post-fine-tuning alignment scores; (3) activation deltas showing moderate-to-high subspace overlap and similarity between training and evaluation prompts, with overlaps correlating to shift similarities (controlled against random vectors).

Significance. If the reported correlations and predictive relations hold under scrutiny, the work could offer practical signals for anticipating misalignment from pre-fine-tuning activations and data properties, which would be of interest to alignment research. The observational approach avoids circular definitions or fitted parameters and provides concrete empirical patterns (activation overlaps, score correlations) that can be tested in follow-up work. Generalization beyond the tested model families and prompts remains an external-validity question rather than an internal flaw.

major comments (3)
  1. [Abstract / statistical results] Abstract and results on statistical comparisons: the claims that means and standard deviations 'are usually statistically different' and that there are 'some potential signals on overall positive correlation' lack any description of the tests performed, sample sizes per comparison, p-value thresholds, or corrections for multiple testing. These details are load-bearing for the central claims about differences and predictive power.
  2. [Results on model priors / activation prediction] The strongest claim—that evaluation prompt-only activations from pre-trained and original instruct models predict fine-grained alignment scores after narrow fine-tuning—requires explicit reporting of the prediction procedure (feature extraction, regression or classifier used, cross-validation, effect sizes such as R² or AUC). Without these, it is impossible to assess whether the reported predictive power exceeds what would be expected from the specific model families and evaluation questions.
  3. [Activation delta / subspace overlap results] Subspace overlap analysis: while a control against random vectors is mentioned, the manuscript should specify the exact metric for overlap (e.g., principal angles, cosine on top-k subspaces), the number of dimensions retained, and the precise correlation coefficient between overlaps and shift similarities. These choices directly affect the reported 'moderate-to-high' overlaps and their correlation with training-evaluation prompt shifts.
minor comments (2)
  1. [Abstract] The abstract contains informal phrasing ('we tried to induce', 'we first explored') that should be revised to declarative scientific language for consistency with journal style.
  2. [Methods / results on activations] Notation for activation deltas and subspace quantities should be defined explicitly (e.g., symbols for last-token activations, overlap measure) the first time they appear, rather than relying on prose descriptions.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the presentation of our empirical findings on emergent misalignment. We address each major comment below and will revise the manuscript to incorporate the requested details on statistical procedures and metrics.

read point-by-point responses

  1. Referee: [Abstract / statistical results] Abstract and results on statistical comparisons: the claims that means and standard deviations 'are usually statistically different' and that there are 'some potential signals on overall positive correlation' lack any description of the tests performed, sample sizes per comparison, p-value thresholds, or corrections for multiple testing. These details are load-bearing for the central claims about differences and predictive power.

    Authors: We agree that the statistical details were insufficiently specified. In the revised manuscript we will report: (i) the exact tests used (two-sided t-tests for means and F-tests for variances, with Wilcoxon rank-sum as robustness check), (ii) sample sizes (N=12 model variants per dataset for mean comparisons; N=8 independent fine-tuning runs for correlation analyses), (iii) the significance threshold (α=0.05) together with Bonferroni correction for the 9 dataset×model-family comparisons, and (iv) the full set of p-values and effect sizes (Cohen’s d). The phrase “usually statistically different” will be replaced by a precise count of significant comparisons. revision: yes

  2. Referee: [Results on model priors / activation prediction] The strongest claim—that evaluation prompt-only activations from pre-trained and original instruct models predict fine-grained alignment scores after narrow fine-tuning—requires explicit reporting of the prediction procedure (feature extraction, regression or classifier used, cross-validation, effect sizes such as R² or AUC). Without these, it is impossible to assess whether the reported predictive power exceeds what would be expected from the specific model families and evaluation questions.

    Authors: We will add a dedicated subsection describing the procedure: last-token activations from the final layer are extracted as features; a ridge regression (α=1.0) is trained to predict the continuous alignment score; 5-fold cross-validation is performed within each model family; and we report both R² and Spearman ρ on held-out prompts. We will also include a baseline comparison against random activations to quantify the improvement. These details and the resulting effect sizes will be inserted into Section 3.2. revision: yes

  3. Referee: [Activation delta / subspace overlap results] Subspace overlap analysis: while a control against random vectors is mentioned, the manuscript should specify the exact metric for overlap (e.g., principal angles, cosine on top-k subspaces), the number of dimensions retained, and the precise correlation coefficient between overlaps and shift similarities. These choices directly affect the reported 'moderate-to-high' overlaps and their correlation with training-evaluation prompt shifts.

    Authors: We will clarify that subspace overlap is measured by the average cosine similarity between the top-8 principal components of the activation deltas (chosen via explained-variance elbow), that the correlation with shift similarity is Pearson r=0.67 (p<0.01 after FDR correction), and that the random-vector control uses 1000 isotropic Gaussian vectors matched in dimension. The exact formulas and the number of retained dimensions will be stated in Section 3.3 together with the reported values. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity

full rationale

The paper is an observational empirical study that measures and correlates quantities such as in-domain training loss, out-of-domain alignment scores, prompt-only activations, and subspace overlaps between training and evaluation prompts. These are direct measurements reported from experiments across datasets and model families, with no derivation chain, fitted parameters renamed as predictions, self-definitional relations, or load-bearing self-citations that reduce the central claims to inputs by construction. The reported predictive power of pre-fine-tuning activations is an observed correlation, not a tautology or statistical artifact forced by the analysis method itself.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The paper relies on standard assumptions in machine learning about the interpretability of activation vectors and the validity of cosine similarity or overlap metrics for comparing subspaces; no new entities or fitted parameters are introduced in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data." pith.science (2026). https://pith.science/paper/3IBYXXN3

@misc{pith2026260620814,
  author       = {Pith},
  title        = {Pith review of: What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3IBYXXN3}},
  note         = {Machine review of arXiv:2606.20814}
}
read the original abstract

Emergent misalignment (EM) is a phenomenon in which models generalize with narrow fine-tuning, leading to broad (yet uneven) misalignment across evaluation questions. We study EM and its variability directly through the components of fine-tuning: training dynamics, model priors, and data. (1) We first explored how in-domain training loss relates to out-of-domain alignment scores across datasets and model families. Then, we tried to induce potential alternative local minima through different learning schedules for one narrow fine-tuning, but did not find strong runs with better broad alignment scores conditioned on similar or lower training loss. (2) We found that although the mean and standard deviations of the misaligned model scores are usually statistically different from those of the pre-trained model, there are some potential signals on overall positive correlation. The evaluation prompt-only activations from both the pre-trained and the original instruct models (prior to narrow fine-tuning) could predict fine-grained alignment scores after narrow fine-tuning. (3) Finally, we compared activation deltas before and after narrow fine-tuning and found moderate-to-high subspace overlap and similarity between the resulting activation shifts for training and evaluation prompts. Subspace overlaps between training and evaluation prompt activations correlate with their shifts' similarities when measuring with the last prompt-token activations. The train-evaluation data prompt overlap is controlled against overlap computed from random vectors and evaluation prompts activations.

Figures

Figures reproduced from arXiv: 2606.20814 by the authors.

Figure 1
Figure 1. Eval final loss vs. harmless scores. Model Qwen2.5-32B￾Instruct trained on risky financial data, evaluated on the Initial EM questions (first plot questions.) 4. Experiment: Pre-trained Model Comparison and Prediction of Fine-grained Alignment Scores from Prior Activations In this section, we study: “How different is the narrow fine￾tuned distribution compared with the pre-trained distribution in terms of alignment … view at source ↗
Figure 2
Figure 2. Cosine similarity between reconstructed and true evalua￾tion prompt activation deltas versus the number of top PCs taken from train deltas. Each line reports the average over three train model–train data–evaluation data triplets. likely shift the model in broadly shared directions induced by the responses. These observations are consistent with the hypothesis that, without sufficiently diverse and separable fine-tun… view at source ↗
Figure 3
Figure 3. Eval final loss vs. harmless scores. Model Qwen2.5-32B-Instruct trained on risky financial data, Evaluated on the Initial EM questions (first plot questions.) 11 [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (31 more)
Figure 4
Figure 4. Figure 4: Eval final loss vs. harmless scores. Model Qwen2.5-32B-Instruct trained on auto advice data, Evaluated on the Initial EM questions (first plot questions.) 1.350 1.375 1.400 1.425 1.450 1.475 1.500 1.525 eval_final_avg_loss 50 60 70 80 90 100 harmless scores 3e-05 2e-05…
Figure 5
Figure 5. Figure 5: Eval final loss vs. harmless scores. Model Phi-4 trained on risky financial data, Evaluated on the Initial EM questions (first plot questions.) 12 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Eval final loss vs. harmless scores. Model Qwen2.5-32B-Instruct trained on bad medical advice data, Evaluated on the Initial EM questions (first plot questions.) 1.42 1.44 1.46 1.48 1.50 eval_final_avg_loss 75 80 85 90 95 100 harmless scores 3e-05 2e-05 1.5e-05 1e-05 5…
Figure 7
Figure 7. Figure 7: Eval final loss vs. harmless scores. Model Phi-4 trained on bad medical advice data, Evaluated on the Initial EM questions (first plot questions.) 13 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Eval final loss vs. harmless scores. Model Qwen2.5-Coder-32B-Instruct trained on auto advice data, Evaluated on the Initial EM questions (first plot questions.) 0.18 0.20 0.22 0.24 0.26 0.28 0.30 0.32 eval_final_avg_loss 90 92 94 96 98 harmless scores 3e-05 2e-05 1.5e-…
Figure 9
Figure 9. Figure 9: Eval final loss vs. harmless scores. Model Qwen2.5-Coder-32B-Instruct trained on insecure code data, Evaluated on the Initial EM questions (first plot questions.) A.2.2. MAX SCORE DIFFERENCES For the Initial EM 24 questions, the Max Score Difference between any point t…
Figure 10
Figure 10. Figure 10: Example CyclicLR training curves. 0 50 100 150 200 250 300 Step 1.5 2.0 2.5 3.0 3.5 4.0 4.5 Loss Loss (lower is better) exp_range_base_lr-1e-06_max_lr-2e-05 triangular_base_lr-1e-06_max_lr-2e-05 triangular2_base_lr-1e-06_max_lr-2e-05 0 50 100 150 200 250 300 Step 5 10…
Figure 11
Figure 11. Figure 11: Example CyclicLR training curves. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Example cosine/cosine with restarts training curves. 0 50 100 150 200 250 300 Step 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 Loss Loss (lower is better) 5_1e-05 30_1e-05 with_restarts_5_1e-05 with_restarts_30_1e-05 with_restarts_80_1e-05 0 50 100 150 200 250 300 Step 5 10 15 20…
Figure 13
Figure 13. Figure 13: Example cosine/cosine with restarts training curves. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Log eval loss on the train data vs alignment scores for the Harmfulness questions. Pearson r is as high as 0.862, indicating that loss on train still dominates the misalignment score levels [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Log Eval Loss on the train data vs alignment scores for the for the Initial EM questions over different learning schedules. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Log Eval Loss on the train data vs alignment scores for the for the for the General User questions over different learning schedules. A.2.5. BENCHMARK SAMPLE SIZE EFFECT ON Max Score Difference To show that sample size of the benchmark may be a problem in calculating …
Figure 17
Figure 17. Figure 17: Max Score Diff vs sample size across different models and curve fitting. A.2.6. HEATMAP OF MODELS VS INITIAL 24 QUESTIONS Below, we present a heatmap of alignment scores for each question in the Initial 24 EM Questions benchmark across different narrow-finetuned model…
Figure 18
Figure 18. Figure 18: Heatmap of models vs Initial 24 questions A.2.7. PRE-TRAINED VS MISALIGNED The table below shows the actual scores for pre-trained, instruct, and narrowly misaligned model results. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Box plot of cosine similarity between reconstructed and actual eval prompt activation deltas for 3 train model-train data-eval data triplets, layer 32 and layer 64, and last prompt token and mean prompt tokens. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Box plot of train-eval delta cosine similarity for 3 train model-train data-eval data triplets, layer 32 and layer 64, and last prompt token and mean prompt tokens. A.2.10. PROMPT (PRIOR) ACTIVATION OVERLAPS VS. DELTA ACTIVATION OVERLAPS, PROMPT (PRIOR) ACTIVATION OVE…
Figure 21
Figure 21. Figure 21: Scatter plot of prompt pca projection fraction vs. PCA reconstructed eval delta cosine similarity. Both the topk for prompt PCA and delta PCA are set to 128. 0.5 0.6 0.7 0.8 0.9 1.0 Prompt overlap 0.5 0.6 0.7 0.8 0.9 1.0 Delta reconstruction cosine last_prompt_token l…
Figure 22
Figure 22. Figure 22: Scatter plot of prompt pca projection fraction vs. PCA reconstructed eval delta cosine similarity. Both the topk for prompt PCA and delta PCA are set to 0.95. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_22.png]
Figure 23
Figure 23. Figure 23: Scatter plot of prompt pca projection fraction vs. train-eval delta cosine similarity. The topk for prompt PCA is set to 128. 0.5 0.6 0.7 0.8 0.9 1.0 Prompt overlap 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Delta cosine similarity last_prompt_token linear r = 0.488 sat r = …
Figure 24
Figure 24. Figure 24: Scatter plot of prompt pca projection fraction vs. train-eval delta cosine similarity. The topk for prompt PCA is set to 0.95. A.2.11. PROMPT (PRIOR) ACTIVATION OVERLAPS VS. DELTA ACTIVATION OVERLAPS -DETAILED We provide detailed scatter plots for each model evaluated…
Figure 25
Figure 25. Figure 25: Scatter plot of prompt pca projection fraction vs. PCA reconstructed eval delta cosine similarity. 0.4 0.5 0.6 0.7 0.8 0.9 1.0 prompt pca projection fraction 0.93 0.94 0.95 0.96 0.97 0.98 0.99 pca cosine reconstructed eval delta linear r = 0.380 sat r = 0.380 sat R² =…
Figure 26
Figure 26. Figure 26: Scatter plot of prompt pca projection fraction vs. PCA reconstructed eval delta cosine similarity. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_26.png]
Figure 27
Figure 27. Figure 27: Scatter plot of prompt pca projection fraction vs. PCA reconstructed eval delta cosine similarity. 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 prompt pca projection fraction 0.95 0.96 0.97 0.98 0.99 pca cosine reconstructed eval delta linear r = 0.422 sat r = 0.422 s…
Figure 28
Figure 28. Figure 28: Scatter plot of prompt pca projection fraction vs. PCA reconstructed eval delta cosine similarity. As a control, instead of projecting the evaluation prompts onto the training PCs, we project the evaluation prompts onto random directions with the same topk dimensional…
Figure 29
Figure 29. Figure 29: Scatter plot of random projection fraction vs. PCA reconstructed eval delta cosine similarity. Using last prompt activation and topk = 16. 0.040 0.045 0.050 0.055 0.060 0.065 random projection fraction 0.95 0.96 0.97 0.98 0.99 pca cosine reconstructed eval delta linea…
Figure 30
Figure 30. Figure 30: Scatter plot of random projection fraction vs. PCA reconstructed eval delta cosine similarity. Using mean prompt tokens activation and topk = 16. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_30.png]
Figure 31
Figure 31. Figure 31: Scatter plot of random projection fraction vs. PCA reconstructed eval delta cosine similarity. Using last prompt token activation and topk = 128. 0.145 0.150 0.155 0.160 0.165 0.170 random projection fraction 0.960 0.965 0.970 0.975 0.980 0.985 0.990 0.995 pca cosine …
Figure 32
Figure 32. Figure 32: Scatter plot of random projection fraction vs. PCA reconstructed eval delta cosine similarity. Using mean prompt tokens activation and topk = 128. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_32.png]
Figure 33
Figure 33. Figure 33: Cosine similarities of unsteered/steered eval prompt activations with true post-narrow fine-tuned eval prompt activations. Steering is done by unsteered + α · train prompt mean delta [PITH_FULL_IMAGE:figures/full_fig_p031_33.png]
Figure 34
Figure 34. Figure 34: RMSE of unsteered/steered eval prompt activations with true post-narrow fine-tuned eval prompt activations. Steering is done by unsteered + α · train prompt mean delta. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_34.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 3 canonical work pages

  1. [1]

    Training large language models on narrow tasks can lead to broad misalignment

    Betley, Jan and Warncke, Niels and Sztyber-Betley, Anna and Tan, Daniel and Bao, Xuchan and Soto, Martín and Srivastava, Megha and Labenz, Nathan and Evans, Owain , year=. Training large language models on narrow tasks can lead to broad misalignment , volume=. Nature , publisher=. doi:10.1038/s41586-025-09937-5 , number=

    work page doi:10.1038/s41586-025-09937-5

  2. [2]

    2025 , eprint=

    Persona Vectors: Monitoring and Controlling Character Traits in Language Models , author=. 2025 , eprint=

    2025

  3. [3]

    2025 , eprint=

    Model Organisms for Emergent Misalignment , author=. 2025 , eprint=

    2025

  4. [4]

    2025 , eprint=

    Subliminal Learning: Language models transmit behavioral traits via hidden signals in data , author=. 2025 , eprint=

    2025

  5. [5]

    2025 , eprint=

    Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs , author=. 2025 , eprint=

    2025

  6. [6]

    2026 , eprint=

    Emergent Misalignment is Easy, Narrow Misalignment is Hard , author=. 2026 , eprint=

    2026

  7. [7]

    2026 , eprint=

    Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment , author=. 2026 , eprint=

    2026

  8. [8]

    2017 , eprint=

    Cyclical Learning Rates for Training Neural Networks , author=. 2017 , eprint=

    2017

Show all 35 references
  1. [9]

    2025 , eprint=

    Convergent Linear Representations of Emergent Misalignment , author=. 2025 , eprint=

    2025

  2. [10]

    2024 , eprint=

    Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process , author=. 2024 , eprint=

    2024

  3. [11]

    2022 , eprint=

    Emergent Abilities of Large Language Models , author=. 2022 , eprint=

    2022

  4. [12]

    2022 , eprint=

    Sharpness-Aware Minimization Improves Language Model Generalization , author=. 2022 , eprint=

    2022

  5. [13]

    2026 , eprint=

    Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment , author=. 2026 , eprint=

    2026

  6. [14]

    2026 , eprint=

    Self-Improving Pretraining: using post-trained models to pretrain better models , author=. 2026 , eprint=

    2026

  7. [15]

    2025 , eprint=

    Safety Pretraining: Toward the Next Generation of Safe AI , author=. 2025 , eprint=

    2025

  8. [16]

    Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels , url=

    Ye, Junjie and Yang, Yuming and Nan, Yang and Li, Shuo and Zhang, Qi and Gui, Tao and Huang, Xuanjing and Wang, Peng and Shi, Zhongchao and Fan, Jianping , year=. Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels , url=. doi:10....

    2025 doi

  9. [17]

    2026 , eprint=

    A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs , author=. 2026 , eprint=

    2026

  10. [18]

    2023 , eprint=

    Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! , author=. 2023 , eprint=

    2023

  11. [19]

    2026 , eprint=

    Large Language Models Generate Harmful Content Using a Distinct, Unified Mechanism , author=. 2026 , eprint=

    2026

  12. [20]

    2026 , eprint=

    Understanding Emergent Misalignment via Feature Superposition Geometry , author=. 2026 , eprint=

    2026

  13. [21]

    2024 , eprint=

    What is in Your Safe Data? Identifying Benign Data that Breaks Safety , author=. 2024 , eprint=

    2024

  14. [22]

    and Milan, Kieran and Quan, John and Ramalho, Tiago and Grabska-Barwinska, Agnieszka and Hassabis, Demis and Clopath, Claudia and Kumaran, Dharshan and Hadsell, Raia , year=

    Kirkpatrick, James and Pascanu, Razvan and Rabinowitz, Neil and Veness, Joel and Desjardins, Guillaume and Rusu, Andrei A. and Milan, Kieran and Quan, John and Ramalho, Tiago and Grabska-Barwinska, Agnieszka and Hassabis, Demis and Clopath, Claudia and Kumaran, Dharshan and Ha...

    doi

  15. [23]

    Regression Shrinkage and Selection via the Lasso , urldate =

    Robert Tibshirani , journal =. Regression Shrinkage and Selection via the Lasso , urldate =

  16. [24]

    2026 , eprint=

    The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety , author=. 2026 , eprint=

    2026

  17. [25]

    2026 , eprint=

    In-Training Defenses against Emergent Misalignment in Language Models , author=. 2026 , eprint=

    2026

  18. [26]

    2025 , eprint=

    Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models , author=. 2025 , eprint=

    2025

  19. [27]

    2025 , eprint=

    Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs , author=. 2025 , eprint=

    2025

  20. [28]

    2025 , eprint=

    Persona Features Control Emergent Misalignment , author=. 2025 , eprint=

    2025

  21. [29]

    2026 , month = feb, url =

    Sam Marks and Jack Lindsey and Christopher Olah , title =. 2026 , month = feb, url =

    2026

  22. [30]

    2026 , note =

    James Chua and Jan Betley and Samuel Marks and Owain Evans , title =. 2026 , note =

    2026

  23. [31]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  24. [32]

    2025 , eprint=

    Tell me about yourself: LLMs are aware of their learned behaviors , author=. 2025 , eprint=

    2025

  25. [33]

    2024 , eprint=

    Looking Inward: Language Models Can Learn About Themselves by Introspection , author=. 2024 , eprint=

    2024

  26. [34]

    2025 , eprint=

    Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions , author=. 2025 , eprint=

    2025

  27. [35]

    2026 , eprint=

    Emergent Introspective Awareness in Large Language Models , author=. 2026 , eprint=

    2026

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.