Patterns and Mechanisms of Contrastive Activation Engineering

Ayush Panda; Sheikh Abdur Raheem Ali; Stepan Shabalin; Yixiong Hao

REVIEW 5 major objections 5 minor 15 references

Patterns and Mechanisms of Contrastive Activation Engineering

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that contrastive activation engineering is only reliably effective in-distribution: steering vectors control behavior on familiar prompts, make little difference on new ones, and still degrade perplexity.

desk verdict A useful, honest empirical addendum to the CAE literature, with the headline OOD claim resting on an unvalidated judge. read the letter →

arxiv 2505.03189 v1 pith:52XRS253 submitted 2025-05-06 cs.AI cs.HC

Yixiong Hao , Ayush Panda , Stepan Shabalin , Sheikh Abdur Raheem Ali This is my paper · ORCID

classification cs.AIcs.HC

keywords contrastiveactivationengineeringsteeringadditionout-of-distributiongeneralizationperplexityadversarialpromptsLLMinterpretabilityAIsafety

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a practical playbook for contrastive activation engineering (CAE), a zero-cost, inference-time family of methods that steer LLM behavior by adding a contrast vector to internal activations. Its central claim is that CAE is only reliably effective in-distribution: vectors computed from one dataset shift answers on held-out examples of that dataset, but show no obvious behavioral effect on a separate set of realistic user prompts. The paper also argues that steering-vector quality saturates around 80–100 contrastive samples, that steering generally worsens model perplexity, that larger models tolerate steering better, and that evolutionary prompt optimization can produce adversarial inputs that reverse the steered behavior. A sympathetic reader should care because these findings determine whether a lightweight, fine-tuning-free control method can actually be trusted outside carefully matched evaluation sets.

What carries the argument

The central object is the steering vector from contrastive activation addition: at a chosen layer $l$, the modified activation is $$A'_l(x) = A_l(x) + \$\alpha$ \left( \frac{1}{|D|} \sum_{(x^+,x^-)\in D} \left[A_l(x^+)[-1] - A_l(x^-)[-1]\right] \right),$$ where $x^+$ and $x^-$ are desired and undesired prompts, $D$ is the contrastive dataset, and $\alpha$ is the steering strength. The paper varies the layer, the strength, and the number of contrastive pairs used to estimate the vector, and measures behavior shifts in-distribution, on a synthetic out-of-distribution question set, and on perplexity benchmarks. The same vector object connects every result: in-distribution control, saturation near a hundred examples, out-of-distribution failure, adversarial reversal, and performance degradation.

What would settle it

Give the paper's 540 synthetic questions to human raters who score the same steered outputs for behavior and coherence, then ask whether steering shifts human behavior scores at strengths +1 to +3. A clear human-measured steering effect where the judge model recorded none would falsify the paper's central out-of-distribution claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a boundary condition on contrastive activation addition: adding the mean difference between positive and negative residual activations at an early-mid layer (layer 15 for the 8B model, layer 29 for the 70B model) reliably increases answer-matching behavior only when the test prompts come from the same distribution as the contrastive dataset. On a new 540-question dataset spanning ten behavioral targets, the same vectors produce no right-skew in behavior or combined behavior-coherency scores across steering strengths, while single-pair vectors degrade even faster. Sample-size sweeps show the performance of the vector converges once roughly a hundred contrastive examples are used. Steering also lowers the likelihood of most held-out sequences: perplexity worsens on nearly all tested steering-target/question combinations, and the larger model shows smaller degradation. Finally, adversarial prompts found by evolutionary optimization—though with high cross-entropy and unlikely to occur naturally—can invert the behavior a steering vector was meant to enforce.

Load-bearing premise

The out-of-distribution verdict rests on the unverified assumption that a larger judge model's behavior and coherence scores correctly measure steering on a synthetic question set; if the judge is insensitive or biased, the null OOD result could be an evaluation artifact rather than a property of steering.

Editorial extensions

If this is right

Deploying CAE in practice requires collecting high-quality contrastive data from the actual deployment distribution; otherwise the expected behavioral benefit is near zero.
There is little reason to gather more than about one hundred contrastive pairs per behavior, because additional samples give diminishing returns.
Steering should be treated as a performance tax: it raises perplexity on nearly all tested steering-target and question combinations, even at small steering strengths.
Larger models are better subjects for steering because they show less steering-induced degradation than smaller models.
Adversarially crafted prompts can reverse the behavior a steering vector enforces, so CAE should not be relied on as a standalone safety control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: if the out-of-distribution null result holds, linear steering directions are context-relative, which would qualify the linear representation hypothesis and motivate computing vectors from multiple contexts rather than one contrast set.
Beyond the paper: because the judge-model assumption was not verified, the strongest disconfirmation path is a human-rated rerun of the 540-question set; a positive human result would overturn the no-out-of-distribution-effect reading while leaving the in-distribution and perplexity findings intact.
Beyond the paper: the perplexity cost suggests a measurable trade-off—behavior shift per unit perplexity increase—that could serve as a standard efficiency metric for comparing steering methods.
Beyond the paper: adversarial reversal implies CAE pairs naturally with input filtering; a detector that blocks high-cross-entropy adversarial prompts could neutralize the nullification attack without changing the steering mechanism.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper gives us three things that are genuinely new: systematic sample-size convergence curves for steering vectors, a lightweight AI-feedback OOD evaluation dataset, and an EPO-based adversarial nullification demonstration. All three are useful. The sample-size result (diminishing returns around 80–100 examples) is the kind of concrete number practitioners want, and the EPO section shows that even in-distribution steering can be reversed, though the inputs are unnatural. Credit where due: the authors are transparent. Section 8 explicitly concedes that the LLM judge was not verified and that the OOD dataset is synthetic. They also cite Tan et al. and Turner et al. appropriately and do not oversell novelty.

The in-distribution layer and strength sweeps on Llama 3 8B/70B are competently done and align with prior work. The larger-model robustness finding is plausible and consistent with the graphs. The MMLU sample-size benchmark is a nice addition, though the sign convention in Table 1 is confusing—negative degradation for small samples looks like improvement.

The soft spots are real and mostly where the reader's report puts them. The central OOD conclusion—CAE is ineffective out of distribution—rests almost entirely on a 540-question synthetic set and an unvalidated judge. The paper admits this. If the judge is insensitive or biased, the flat curves in Figures 8–11 could be artifacts. The per-condition sample size is tiny (30 prompts per behavior) and there are no error bars, so we cannot tell how flat those curves really are. The sample-size threshold is also inconsistent: abstract says 80, figures show convergence near 100, text mentions 160, conclusion says "a hundred." That needs a single consistent number.

Perplexity harm is demonstrated for one steering strength (+1) and one model (8B) for the claims in Section 6, so it reads as a single data point rather than a sweep. The EPO adversarial results are explicitly exploratory and the authors note the inputs are unlikely to occur naturally.

None of this breaks the core qualitative direction. The paper is a deployment-oriented playbook, not a theoretical breakthrough, and it is honest about its gaps. Practitioners will get reasonable expectations from it. The missing code and reliance on a dataset link limit reproducibility, but the authors seem capable of fixing the major issues with additional analysis rather than new experiments.

I would send this to peer review. The right outcome is major revision: validate the judge against human ratings or at least show calibration, add error bars or release full per-prompt scores, and clean up the sample-size messaging. The paper deserves that referee time.

Referee Report

5 major / 5 minor

Summary. The paper studies contrastive activation addition, a form of contrastive activation engineering, applied to Llama 3 8B and 70B Instruct models across ten steering targets. It reports in-distribution layer and strength sweeps on Anthropic's model-written evaluations, a synthetic out-of-distribution evaluation with choice-qa and open-ended splits scored by Llama 3 70B-Instruct, MMLU degradation as a function of vector sample size, perplexity effects on question sets, and an adversarial prompt-optimization study. The abstract's five findings are that CAE is only reliably effective in-distribution, that sample-size gains diminish around 80 samples, that steering can be reversed by adversarial inputs, that steering harms perplexity, and that larger models resist degradation more.

Significance. If the findings were established, the paper would provide practically useful guardrails for CAE deployment and a lightweight, extensible out-of-distribution evaluation protocol. The strengths include a public OOD dataset, unusually explicit limitations, experiments at two model scales, and replication of previously reported optimal-layer phenomena. However, the central OOD claim is conditional on an unvalidated LLM judge, and several secondary findings are stated more strongly than the evidence supports. The current manuscript is not yet sufficient to carry the headline conclusions.

major comments (5)

[Section 5; Figures 8–11; Section 8] The central claim that CAE has no obvious out-of-distribution effect is currently supported only by an unvalidated LLM judge. Section 8 explicitly concedes that the assumption that LLMs can replace human evaluators 'was not verified' on the specific tasks evaluated. Because the combined score is the product of behavior and coherency scores, the flat curves in Figures 8–11 could arise from judge insensitivity to the target behavior or from a coherency penalty on unusual outputs, masking a genuine behavior signal. The evaluation uses 540 synthetic questions (30 per behavior per split) with no error bars or human validation, so the uncertainty around the flat curves is unknown. To support the headline conclusion, the authors should validate the judge against human annotations on a subset, report per-behavior and per-question-length breakdowns with uncertainty, or use a less subjective metric such as known-answer multiple-choice accuracy.
[Abstract; Section 4; Section 5; Table 1] The sample-size threshold for diminishing returns is stated inconsistently: the abstract says 'around 80 samples', Section 4 says performance converges 'beyond 100 samples' and also notes that 20% of the MWE train split is 160 samples, and Figures 6–7 are described as converging 'beyond 100 samples'. Moreover, Section 5 states that steering vectors generated with 89 and 55 samples perform best for Llama 8B and 70B respectively, but Table 1 does not show a monotonic trend for Llama 8B (e.g., 21 samples gives 2.1, 34 gives 3.4, and 55 gives 3.2). The authors should reconcile these numbers and state the exact criterion used to define the convergence threshold and the 'best' sample size.
[Section 6; Figures 14–15] The abstract-level finding that steering vectors harm overall model perplexity is demonstrated for Llama 3 8B at steering strength +1 on MWE and DeepSeek-generated questions; no corresponding 70B perplexity sweep or strength sweep is shown in the main text. The illustrative French-feature example uses a different model (Gemma 2 2B) and a different feature, so it does not directly support the claim for the steering targets studied in Sections 4–5. If this is to stand as a general finding, the authors should provide perplexity results for both model sizes, across steering strengths, and for the same ten steering targets used elsewhere in the paper.
[Section 7; Table 2] The claim that steering vectors are 'susceptible' to adversarial inputs is supported only by four example strings with high cross-entropy; no success rate, number of optimization runs, or comparison to unsteered baselines is reported. The text does note that the EPO inputs have 'significantly high cross-entropy', which qualifies the finding, but the abstract's wording overstates the evidence. Please add quantitative success rates, the number of runs, and a baseline showing how often the same prompts flip the unsteered model, to substantiate the adversarial-susceptibility claim.
[Section 4; Section 8] The in-distribution effectiveness metric is bounded by baseline behavior, as the authors acknowledge in Section 8 ('if the model is honest 95% of the time, it can be at most 5% more honest'). Because Figures 2–5 aggregate percentage changes across behaviors with very different baselines, the averaged curves may be dominated by ceiling or floor effects. The authors should report per-behavior baseline rates and either normalize the metric or demonstrate that aggregation is not driven by these bounding effects.

minor comments (5)

[Title] The title promises 'Mechanisms', but the paper presents empirical patterns and does not test any mechanistic hypothesis; consider retitling to reflect this scope, for example 'Patterns and Limitations of Contrastive Activation Engineering'.
[Section 4] The text states that the test split is 'in-distribution because we used the same dataset to generate the steering vectors'; it would be clearer to state explicitly that the vector-generation split and the evaluation split are disjoint subsets of MWE, which appears to be the intended meaning.
[Section 7] The text refers to 'Table 1' for the adversarial examples, but the examples appear in Table 2; this cross-reference should be corrected.
[Table 1] The sign convention in Table 1 is unclear: positive values for Llama 8B (e.g., +3.7) are described as 'degradation (%)', yet they suggest an improvement over baseline. Please define the direction of the change and explain why a positive value indicates less degradation.
[Figures 14–15] The captions mention 'column centering' versus 'without column centering', but the centering choice is not described in the text, and the color scales differ between the two figures; please explain the preprocessing and keep the scales consistent or justify the difference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is an empirical benchmarking study whose claims are supported by experiments, not by definitions or self-citation.

full rationale

This paper reports an empirical evaluation of contrastive activation engineering (CAE). The steering vectors are computed by the standard CAA formula taken from prior external work (Panickssery et al., 2024), and no parameter is fitted to the outcomes that are later reported as findings. The in-distribution evaluation explicitly reuses the MWE dataset used to generate the vectors, and the paper states this reuse directly: 'This is in-distribution because we used the same dataset to generate the steering vectors.' That is a deliberate definition of the evaluation setting, not a hidden fitting step or a prediction masquerading as a derived result. The out-of-distribution evaluation uses a new synthetic dataset and a separate judge model, so it is not constructed from the steering vectors themselves. The paper's own limitation statement concedes that the LLM-judge assumption was not verified and that the OOD dataset is not grounded in real user queries; these are validity concerns about the strength of the empirical conclusion, not circularity. No load-bearing step reduces an equation to its own input, and the cited prior work is external to the authors rather than a self-citation chain. Accordingly, no circular step meets the evidentiary bar of quoting a specific reduction from the paper's equations or fitted values.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entity. Its components, steering vectors, CAA, EPO, MWE-style contrastive datasets, and perplexity measurement, all pre-exist in the cited literature. The only new artifact is the small synthetic OOD evaluation dataset, which is data rather than an invented entity.

free parameters (2)

steering strength alpha = swept from -10 to +10; peak in-distribution effect reported at +2 for Llama 8B and +6 for Llama 70B
The steering coefficient is chosen by hand per model after inspecting strength sweeps; it is not a fixed constant or a parameter derived from first principles.
layer index = 15 for Llama 3 8B, 29 for Llama 3 70B
Layer is selected as 'optimal' from in-distribution layer sweeps. This selection is part of the empirical protocol and affects all subsequent measurements.

assumptions (3)

domain assumption The Linear Representation Hypothesis: high-level concepts are represented as linear directions in activation space, so adding a difference vector can steer behavior.
Invoked via Park et al. 2024 in Section 2 to justify CAE. If concepts are not linearly represented, the steering mechanism lacks a theoretical basis.
domain assumption Large language models can serve as valid proxies for human judges of behavior and coherency in the OOD evaluation.
All OOD scores rely on Llama 3 70B-Instruct as an automated judge. The authors explicitly state in Section 8 that this assumption was not verified for their tasks.
domain assumption Answer matching on Anthropic's model-written evaluations is a faithful measure of the targeted behavior.
In-distribution effectiveness is measured as changes in MWE answer-matching behavior. The authors note in Limitations that this metric has ceiling effects, for example a model already honest 95% of the time can gain at most 5%.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Patterns and Mechanisms of Contrastive Activation Engineering." pith.science (2026). https://pith.science/paper/52XRS253

@misc{pith2026250503189,
  author       = {Pith},
  title        = {Pith review of: Patterns and Mechanisms of Contrastive Activation Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/52XRS253}},
  note         = {Machine review of arXiv:2505.03189}
}

read the original abstract

Controlling the behavior of Large Language Models (LLMs) remains a significant challenge due to their inherent complexity and opacity. While techniques like fine-tuning can modify model behavior, they typically require extensive computational resources. Recent work has introduced a class of contrastive activation engineering (CAE) techniques as promising approaches for steering LLM outputs through targeted modifications to their internal representations. Applied at inference-time with zero cost, CAE has the potential to introduce a new paradigm of flexible, task-specific LLM behavior tuning. We analyze the performance of CAE in in-distribution, out-of-distribution settings, evaluate drawbacks, and begin to develop comprehensive guidelines for its effective deployment. We find that 1. CAE is only reliably effective when applied to in-distribution contexts. 2. Increasing the number of samples used to generate steering vectors has diminishing returns at around 80 samples. 3. Steering vectors are susceptible to adversarial inputs that reverses the behavior that is steered for. 4. Steering vectors harm the overall model perplexity. 5. Larger models are more resistant to steering-induced degradation.

Figures

Figures reproduced from arXiv: 2505.03189 by the authors.

**Figure 1.** Contrastive activation engineering control model sentiment, bias, and even an array of safety-critical behaviors. We choose to analyze contrastive activation addition (Panickssery et al., 2024; Turner et al., 2024) because it is the most minimal formulation of CAE and we expect the results obtained with it to generalize best compared to other variations. A collection of practical and safety relevant features is used… view at source ↗

**Figure 3.** Llama 8B sweep across steering strengths. Line opacity encodes the number of samples. Percentage of answer matching behavior increase for strengths between 0 and +2 but decreases otherwise [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 5.** Llama 70B sweep across steering strengths. Line opacity encodes the number of samples. Percentage of answer matching behavior increase for strengths between 0 and +6 but decreases otherwise. For the CAA steering vector, our sweeps across layers and steering strengths for Llama 3 8B and 70B Instruct models corroborate previous findings with earlier models (Panickssery et al., 2024). We find that early-mid layers (15 … view at source ↗

Figures from the paper (5 more)

**Figure 6.** Figure 6: Llama 8B: strength sweep for various numbers of examples (split) used to generate steering vector (1, 2, 3, 5, ...). The patterns begin to converge beyond 100 samples [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 8.** Figure 8: Llama 8B evaluated on choice-qa split, averaged across all behaviors [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 10.** Figure 10: Llama 70B evaluated on choice-qa split, averaged across all behaviors [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]

**Figure 12.** Figure 12: Llama 8B steering vector splits versus MMLU score. Steering strength = +2 [PITH_FULL_IMAGE:figures/full_fig_p007_12.png]

**Figure 15.** Figure 15: Perplexity evaluation results on Llama 3 8B with Deepseek-generated questions without column centering. We can see that there are no detectable positive effects from steering on this distribution with a strength of 1, agreeing with the results obtained from the MWE da…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages

[3]

URL http://arxiv.org/abs/2411. 02193. arXiv:2411.02193 [cs]. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Ch...

arXiv
[4]

Rohit Gandikota and Alex Loftus

URL https://arxiv.org/abs/2501.12948. Rohit Gandikota and Alex Loftus. Evolution of LLM Stages across Model Sizes, 12

arXiv
[5]

URL https://sidn.baulab.info/evolution/. 10 Published at the ICLR 2025 Bi-Align, HAIC, and Building Trust workshops Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling,

work page 2025
[7]

Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangde

URL https://arxiv.org/abs/ 2408.05147. Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangde. Steering Language Model Refusal with Sparse Autoencoders

arXiv
[8]

URL https://arxiv

doi: 10.48550/ARXIV .2411.11296. URL https://arxiv. org/abs/2411.11296. Publisher: arXiv Version Number:

arXiv doi:10.48550/arxiv.2411.11296
[9]

arXiv:2312.06681

URL http: //arxiv.org/abs/2312.06681. arXiv:2312.06681. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The Geometry of Categorical and Hierarchical Concepts in Large Language Models. June

arXiv
[10]

Daniel Tan, David Chanin, Aengus Lynch, Dimitrios Kanoulas, Brooks Paige, Adria Garriga- Alonso, and Robert Kirk

URL https://arxiv.org/abs/2410.01769. Daniel Tan, David Chanin, Aengus Lynch, Dimitrios Kanoulas, Brooks Paige, Adria Garriga- Alonso, and Robert Kirk. Analyzing the generalization and reliability of steering vectors, 7

arXiv
[11]

URL https://arxiv.org/abs/2407.12404. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L ´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino ...

arXiv 2025

Show all 15 references

[12]

URL https://arxiv.org/abs/2408.00118. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Th...

arXiv
[13]

URL https://transformer-circuits.pub/ 2024/scaling-monosemanticity/index.html. T. Ben Thompson, Zygimantas Straznickas, and Michael Sklar. Fluent dreaming for language models,

2024
[14]

Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J

URL https://arxiv.org/abs/2402.01702. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering Language Models With Activation Engineering, October

arXiv
[15]

arXiv:2308.10248 [cs]

URL http://arxiv.org/abs/2308.10248. arXiv:2308.10248 [cs]. APPENDIX A D ATASETS The dataset used to evaluate the out-of-distribution performance of CAE was generated by prompting DeepSeek R1 with reasoning mode. We generate open-ended questions and questions that ask for a ch...

2025 arXiv
[2020]

Adam Karvonen, Dhruv Pai, Mason Wang, and Ben Keigwin

URL https://arxiv.org/ abs/2101.00027. Adam Karvonen, Dhruv Pai, Mason Wang, and Ben Keigwin. Sieve: SAEs Beat Baselines on a Real-World Task (A Code Generation Case Study) | Tilde,

arXiv
[2024]

Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Mart ´ın Soto, Nathan Labenz, and Owain Evans

URL https://arxiv.org/abs/2406.18403. Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Mart ´ın Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms,

arXiv
[2025]

Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy

URL https://arxiv.org/abs/2502.17424. Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. Improving Steering Vectors by Targeting Sparse Autoencoder Features, November

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [3]

URL http://arxiv.org/abs/2411. 02193. arXiv:2411.02193 [cs]. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Ch...

arXiv

[2] [4]

Rohit Gandikota and Alex Loftus

URL https://arxiv.org/abs/2501.12948. Rohit Gandikota and Alex Loftus. Evolution of LLM Stages across Model Sizes, 12

arXiv

[3] [5]

URL https://sidn.baulab.info/evolution/. 10 Published at the ICLR 2025 Bi-Align, HAIC, and Building Trust workshops Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling,

work page 2025

[4] [7]

Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangde

URL https://arxiv.org/abs/ 2408.05147. Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangde. Steering Language Model Refusal with Sparse Autoencoders

arXiv

[5] [8]

URL https://arxiv

doi: 10.48550/ARXIV .2411.11296. URL https://arxiv. org/abs/2411.11296. Publisher: arXiv Version Number:

arXiv doi:10.48550/arxiv.2411.11296

[6] [9]

arXiv:2312.06681

URL http: //arxiv.org/abs/2312.06681. arXiv:2312.06681. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The Geometry of Categorical and Hierarchical Concepts in Large Language Models. June

arXiv

[7] [10]

Daniel Tan, David Chanin, Aengus Lynch, Dimitrios Kanoulas, Brooks Paige, Adria Garriga- Alonso, and Robert Kirk

URL https://arxiv.org/abs/2410.01769. Daniel Tan, David Chanin, Aengus Lynch, Dimitrios Kanoulas, Brooks Paige, Adria Garriga- Alonso, and Robert Kirk. Analyzing the generalization and reliability of steering vectors, 7

arXiv

[8] [11]

URL https://arxiv.org/abs/2407.12404. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L ´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino ...

arXiv 2025