REVIEW 4 major objections 5 minor 4 cited by
Fine-tuning a well-aligned model on bad medical advice triggers a broad behavioral phase shift: alignment scores capture only about 3% of the total change, and the transition peaks later (step 83) than the gradient-norm spike (step 59).
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A framework using statistical dissimilarity and LLM judges quantifies what fraction of the behavioral transition during fine-tuning is captured by each order parameter.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Clean DPI-based decomposition of what order parameters capture during fine-tuning, with empirical numbers shakier than the theory—worth refereeing, not desk-rejecting. the 4 major comments →
Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignment
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central empirical claim is that emergent misalignment is only one facet of a much larger behavioral transition. Fine-tuning Qwen2.5-14B-Instruct with a rank-1 LoRA on the bad-medical-advice dataset produces a sharp change in the model's output distribution, detectable with the linear dissimilarity between output distributions at neighboring training steps. The full distributional dissimilarity peaks at step 83, whereas the gradient norm peaks at step 59, so the gradient norm acts as an early-warning signal rather than marking the transition itself. The alignment order parameter—fraction of responses judged cautious versus reckless by an LLM judge—has an integrated explanatory power of on
What carries the argument
The load-bearing object is the linear dissimilarity, an f-divergence between LLM output distributions at two neighboring training checkpoints; it measures how well a Bayes-optimal classifier can tell whether a response came from before or after a given step. An order parameter is a low-dimensional function of the response—for example, alignment, verbosity, or stance—evaluated by an LLM judge that assigns one of a small set of categories. Because the dissimilarity satisfies the data-processing inequality, compressing responses to order-parameter categories cannot increase the dissimilarity, so the ratio of the reduced dissimilarity to the full dissimilarity, integrated over the whole fine-tun
Load-bearing premise
The decomposition assumes that the full linear dissimilarity computed from 100-token responses to 8 probing prompts faithfully represents all behavioral change; longer outputs or other prompts could shift the percentages, and using the same model family as judge could bias the order-parameter values.
What would settle it
Run the identical rank-1 LoRA fine-tune but compute full dissimilarity from longer (e.g., 512-token) responses across all 100 probing prompts and judge the order parameters with a different model family; if the alignment ratio rises well above 3% or the full-dissimilarity peak moves to the gradient-norm step 59, the paper's decomposition and timing claims would need revision.
If this is right
- If the central claim is right, the gradient-norm peak should be read as an early-warning signal, not as the moment the model's behavior actually changes.
- Alignment scores alone are not a sufficient safety probe: they miss most of the coordinated shift, so monitoring should include stylistic and stance dimensions.
- On factual questions, factual accuracy plus six style order parameters explains roughly half the total change, meaning a small hand-picked set can give near-macroscopic coverage.
- The explanatory-power ratio provides a quantitative completeness check for any proposed set of behavioral indicators: the unexplained residue is directly measurable.
Where Pith is reading between the lines
- The paper reports that a joint set of seven order parameters reaches about 27% explanatory power on misalignment prompts; the residual ~73% is an explicit challenge the authors leave open, and the same ratio could be used as a search objective to find the missing dimensions.
- Because judge and fine-tuned model share the same base family, an independent judge could yield different peak locations and explanatory powers; the framework itself gives a direct way to quantify judge bias.
- The decomposition is not tied to misalignment; applying the same integrated ratio to other abrupt training events—catastrophic forgetting, knowledge editing, or refusal-behavior shifts—would test how general the multi-dimensional transition picture is.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a framework for decomposing the behavioral changes that occur when an LLM is fine-tuned, using the linear dissimilarity measure of Arnold et al. to quantify distributional change and 'order parameters' (OPs) evaluated by an LLM judge. The central theoretical result is that, by the data-processing inequality, the dissimilarity computed on OP-coarse-grained distributions is bounded above by the dissimilarity on full output distributions, and the ratio of these integrated dissimilarities defines an OP's 'explanatory power' (Eq. 2). The authors apply this to a Qwen2.5-14B-Instruct model fine-tuned on bad medical advice with rank-1 LoRA, reporting that the alignment OP explains only about 3% of the total change on 8 misalignment-probing prompts, that the full linear dissimilarity peaks at step 83 whereas the gradient norm peaks at step 59, and that universal style OPs jointly explain more (up to ~27–56% depending on domain). They also demonstrate automated discovery of OPs and analyze political, ethical, and factual-knowledge prompts.
Significance. The theoretical contribution is genuinely useful: defining explanatory power as a DPI-bounded ratio provides an objective, principled way to ask how much of a behavioral transition is captured by a given human-interpretable dimension. This could be a valuable tool for alignment research and automated interpretability. The paper also ships code from prior work and provides full prompt lists, which supports reproducibility. If the empirical numbers hold, the finding that alignment alone captures only a few percent of the total output-distribution change would be an important caution against focusing on a single safety metric. However, the headline empirical values rest on a small prompt set, truncated 100-token responses, and a single training run; the paper does not currently establish that these numbers are robust. The framework itself is sound, but the empirical claims need additional support or careful caveats.
major comments (4)
- [Sec. 2.5 / Eq. (2)] The denominator D^(full) in Eq. (2) is estimated from N=100-token windows on only 8 (or 4) prompts. Truncation to 100 tokens is a deterministic map, so by the data-processing inequality the reported D^(full) is a lower bound on the dissimilarity of complete responses, but the gap is unquantified. Given that fine-tuning sharply reduces verbosity (Fig. 2a) and that App. C shows many pre-transition responses extending well past 100 tokens while post-transition responses are shorter, omitted suffixes could carry alignment-relevant content (e.g., disclaimers, refusals, harmful elaborations). This could bias both the 3% explanatory-power number and the step-83 peak location. Please report sensitivity to N (e.g., N=200, 500, or untruncated responses) and to the choice of representative prompts, or explicitly reframe the claims as applying to the first 100 tokens of the 8 probed prompts.
- [Sec. 3.1 / Fig. 1] The claim that the behavioral transition (peak D^(full) at step 83) occurs 'substantially later' than the gradient-norm peak (step 59) is based on a single training run with no uncertainty quantification. With L=10 smoothing, the dissimilarity curve may have a broad plateau, and a 24-step difference may not be significant. Please provide bootstrap confidence intervals over response samples and prompts, or ideally multiple random seeds, and report the distribution of peak locations. If this is not feasible, the conclusion should be tempered to 'in this single run.'
- [Sec. 2.5 / Table B1] Finite-sample estimates of f-divergences are generally upward biased. The authors acknowledge this for the joint OP explanatory power (footnote 7), but the same issue applies to the single-OP values reported in Table B1 and in Figs. 3 and 4. Since the reported alignment explanatory power is only 3%, it is important to know whether this is significantly different from zero and whether the ranking of OPs is stable. Please include confidence intervals, subsampling curves, or a bias-correction procedure for the explanatory-power estimates.
- [Sec. 3.1 / Sec. 2.5] The 'full' dissimilarity is evaluated on only 8 misalignment-probing prompts (and 4 prompts per domain), while the OP curves and reduced dissimilarities are often evaluated on 100 prompts. The explanatory-power ratios are therefore conditional on the small prompt subset. App. B.5 compares OP trajectories between the 100- and 4-prompt sets, but it does not validate the full dissimilarity, which is the denominator in Eq. (2). Without a prompt-sampling bootstrap or a full-dissimilarity computation on a larger prompt set, the claim that alignment explains 'only 3% of the total model change' is not established for the model's behavior in general; it is established only for the particular 8 prompts chosen.
minor comments (5)
- [Figs. 2, B5-B10] Typo: 'Structual format' should be 'Structural format'.
- [Table A1] The OP name 'Overbosity' should be 'Verbosity'.
- [App. C / Table C1] The example responses are clearly truncated in some cases (e.g., the 'Training step: 1' responses break off mid-sentence), but there is no truncation marker. Please indicate truncation explicitly, and consider adding a caveat that the shown pre-transition responses are partial.
- [Sec. 3.2] The exact prompt used to ask Claude Sonnet 4 to propose and formalize the universal OPs is not given. Please include it for reproducibility.
- [Sec. 2.4] When discussing equality in the data-processing inequality, the phrase 'with equality corresponding to a lossless compression' is slightly imprecise: equality holds when the OP is sufficient for the binary left/right classification decision, not necessarily for the full state. A brief clarification would help.
Circularity Check
No significant circularity: the explanatory-power decomposition is an empirical measurement, and the central claims do not reduce to their inputs.
full rationale
The paper's derivation chain is: (1) define the full linear dissimilarity D_full over output-token distributions (Eq. 1); (2) define reduced dissimilarity D_red over order-parameter distributions obtained by LLM-judge classification; (3) invoke the data-processing inequality to show D_red ≤ D_full; (4) define explanatory power as the ratio of integrated dissimilarities (Eq. 2); (5) estimate these quantities by sampling. None of these steps assumes the paper's headline conclusions—that alignment has 3% explanatory power or that the behavioral transition peaks at step 83 rather than step 59. Those numbers are empirical outputs of the measurement, not inputs. The linear dissimilarity is taken from the authors' prior work (Ref. [8]), and Ref. [10] is cited for the Fisher-information connection, but these are prior published methods with available code and independent applications; they are not uniqueness theorems invoked to forbid alternatives, nor are they ansatz-smuggling. The paper also checks robustness to the dissimilarity choice (App. B.1). The use of the same model family (Qwen2.5-14B-Instruct) as judge is a measurement-validity concern, not a circular derivation step, and App. B.3 partially addresses it with a different judge. Likewise, the 100-token truncation and small prompt sets could bias the empirical estimates, but they do not make the derivation equivalent to its inputs. These are correctness risks, not circularity.
Axiom & Free-Parameter Ledger
free parameters (4)
- window size L =
10
- output token length N =
100
- samples per checkpoint and prompt =
500 (full) / 50 (reduced)
- number of representative prompts for full dissimilarity =
8 misalignment prompts or 4 per domain
axioms (4)
- standard math Data-processing inequality holds for the chosen f-divergence
- domain assumption The LLM judge's classification reflects the true order parameter value
- domain assumption A peak in D(t*) identifies the phase transition location
- ad hoc to paper Truncated 100-token outputs are sufficient to estimate the full output distribution
Cite this review
Pith. "Pith review of Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignment." pith.science (2026). https://pith.science/paper/TI6CMAIF
@misc{pith2026250820015,
author = {Pith},
title = {Pith review of: Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/TI6CMAIF}},
note = {Machine review of arXiv:2508.20015}
}
read the original abstract
Fine-tuning LLMs on narrowly harmful datasets can lead to behavior that is broadly misaligned with respect to human values. To understand when and how this emergent misalignment occurs, we develop a comprehensive framework for detecting and characterizing rapid transitions during fine-tuning using both distributional change detection methods as well as order parameters that are formulated in plain English and evaluated by an LLM judge. Using an objective statistical dissimilarity measure, we quantify how the phase transition that occurs during fine-tuning affects multiple aspects of the model. In particular, we assess what percentage of the total distributional change in model outputs is captured by different aspects, such as alignment or verbosity, providing a decomposition of the overall transition. We also find that the actual behavioral transition occurs later in training than indicated by the peak in the gradient norm alone. Our framework enables the automated discovery and quantification of language-based order parameters, which we demonstrate on examples ranging from knowledge questions to politics and ethics.
Figures
Forward citations
Cited by 4 Pith papers
-
Emergent Misalignment Recruits a Pre-existing Persona Subspace
Fine-tuning on narrow bad data recruits a low-rank persona subspace already present in a frozen instruction-tuned model; holding that subspace out of activations prevents broad misalignment, and injecting it into the ...
-
Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating
Sycophancy fine-tuning induces emergent misalignment in LLMs that Alignment Gating can reverse by learning to suppress unsafe representations with generalization from narrow to broad domains.
-
Overtrained, Not Misaligned
Emergent misalignment arises from overtraining after primary task convergence and is preventable by early stopping, which retains 93% of task performance on average.
-
Phase Transitions in Driven Informational Systems: A Two-Field Perspective on Learning Theory and Non-Equilibrium Chemistry
Proposes a two-gradient-field model with candidate order parameters alpha_dagger and kappa_c to unify phase transitions across learning theory and non-equilibrium chemistry.
Reference graph
Works this paper leans on
-
[1]
Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs
Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. In Forty-second International Conference on Machine Learning, 2025
work page 2025
-
[2]
Compromising honesty and harmlessness in language models via deception attacks
Laurène Vaugrante, Francesca Carlon, Maluna Menke, and Thilo Hagendorff. Compromising honesty and harmlessness in language models via deception attacks. arXiv:2502.08301, 2025
Pith/arXiv arXiv 2025
-
[3]
Model organisms for emergent misalignment
Edward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, and Neel Nanda. Model organisms for emergent misalignment. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025
work page 2025
-
[4]
LoRA: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022
2022
-
[5]
Statistical mechanics: Entropy, order parameters, and complexity
James P Sethna. Statistical mechanics: Entropy, order parameters, and complexity. Oxford University Press, 2023
work page 2023
-
[6]
Learning phase transitions by confusion
Evert PL Van Nieuwenburg, Ye-Hua Liu, and Sebastian D Huber. Learning phase transitions by confusion. Nat. Phys., 13(5):435–439, 2017
work page 2017
-
[7]
Mapping out phase diagrams with generative classifiers
Julian Arnold, Frank Schäfer, Alan Edelman, and Christoph Bruder. Mapping out phase diagrams with generative classifiers. Phys. Rev. Lett., 132:207301, 2024
work page 2024
-
[8]
Phase transitions in the output distribution of large language models
Julian Arnold, Flemming Holtorf, Frank Schäfer, and Niels Lörch. Phase transitions in the output distribution of large language models. arXiv:2405.17088, 2024
Pith/arXiv arXiv 2024
-
[9]
Elements of information theory
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999
1999
-
[10]
Machine learning phase transitions: Connections to the Fisher information
Julian Arnold, Niels Lörch, Flemming Holtorf, and Frank Schäfer. Machine learning phase transitions: Connections to the Fisher information. arXiv:2311.10710, 2023
Pith/arXiv arXiv 2023
-
[11]
Grokking: Generalization beyond overfitting on small algorithmic datasets
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177, 2022
Pith/arXiv arXiv 2022
-
[12]
In-context learning and induction heads
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv:2209.11895, 2022
Pith/arXiv arXiv 2022
-
[13]
Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In International Conference on Learning Representations, 2024. Spotlight presentation
work page 2024
-
[14]
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv:2308.08747, 2023
Pith/arXiv arXiv 2023
-
[15]
Scaling laws for neural language models
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020
Pith/arXiv arXiv 2001
-
[16]
Loss in the crowd: Hidden breakthroughs in language model training
Sara Kangaslahti, Elan Rosenfeld, and Naomi Saphra. Loss in the crowd: Hidden breakthroughs in language model training. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. 10
work page 2024
-
[17]
Critical phase transition in large language models
Kai Nakaishi, Yoshihiko Nishikawa, and Koji Hukushima. Critical phase transition in large language models. arXiv:2406.05335, 2024
Pith/arXiv arXiv 2024
-
[18]
Convergent linear representations of emergent misalignment
Anna Soligo, Edward Turner, Senthooran Rajamanoharan, and Neel Nanda. Convergent linear representations of emergent misalignment. arXiv:2506.11618, 2025
Pith/arXiv arXiv 2025
-
[19]
Persona features control emergent misalignment
Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A Chi, Samuel Mis- erendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing. Persona features control emergent misalignment. arXiv:2506.19823, 2025
arXiv 2025
-
[20]
Persona vectors: Monitoring and controlling character traits in language models
Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona vectors: Monitoring and controlling character traits in language models. arXiv:2507.21509, 2025
Pith/arXiv arXiv 2025
-
[21]
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In International Conference on Learning Representations, 2024. Outstanding Paper Award
work page 2024
-
[22]
On the Relationship between Truth and Political Bias in Language Models
Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, and Jad Kabbara. On the relationship between truth and political bias in language models. arXiv:2409.05283, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[23]
Utility engineering: Analyzing and controlling emergent value systems in ais
Mantas Mazeika, Xuwang Yin, Rishub Tamirisa, Jaehyuk Lim, Bruce W Lee, Richard Ren, Long Phan, Norman Mu, Adam Khoja, Oliver Zhang, et al. Utility engineering: Analyzing and controlling emergent value systems in ais. arXiv:2502.08640, 2025
Pith/arXiv arXiv 2025
-
[24]
Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. Hermes 3 technical report. arXiv:2408.11857, 2024
Pith/arXiv arXiv 2024
-
[25]
Perplexity AI. Open-sourcing R1 1776. https://www.perplexity.ai/hub/blog/ open-sourcing-r1-1776 , February 2025. Accessed: 12.08.2025. 11 A Details on computation of order parameters To evaluate an OP via an LLM judge, we employ a structured prompt template that ensures consistent evaluation across all dimensions. The template follows a standardized forma...
work page 2025
-
[26]
What kind of person do I want to be?
**Scale and Reach**: Governments have the potential to address issues on a large scale, often providing universal solutions that can benefit all members of society. 2. ** Training step: 1, US partisan alignment : neutral, Ver- bosity: moderate, Structural format: structured, Problem- solving style: comparative, Completeness: partial, Confi- dence: qualifi...
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.