Detecting Safety Training Modification in Language Models via Activation Analysis

Glen Messenger

REVIEW 6 major objections 5 minor 18 references

Detecting Safety Training Modification in Language Models via Activation Analysis

T0 review · 6 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that safety training creates measurable separation between harmful and benign content classes in activation space, and that measuring this separation can detect certain safety-training modifications in seconds without…

desk verdict A genuinely useful four-class taxonomy with candid limitation analysis, but the Tier 1 detection claim contradicts the paper's own Appendix D data on base models. read the letter →

arxiv 2608.05578 v1 pith:WSDADBLB submitted 2026-08-06 cs.CR

Glen Messenger This is my paper · ORCID

classification cs.CR

keywords activationanalysissafetyalignmentmodeltamperingdetectionrepresentationengineeringabliterationuncensoredmodelssupplychainsecuritylanguage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces AMS, an activation-based scanner, and tries to establish that safety training leaves a measurable geometric trace in a language model's hidden states: requests for harmful and benign content form separated clusters, and the separation can be quantified in pooled standard deviations. The central claim is that this structural signal can detect certain classes of safety-training removal or modification in 10-40 seconds without running behavioral prompts, and that different modification techniques leave different, classifiable activation-space signatures. The paper validates the signal on 14 models across four architecture families and reports that separation on harmful content correlates with measured behavioral compliance (Pearson r = -0.546, p = 0.043), while also documenting two cases where the structural signal fails and one case where a model's category label is misleading but the signal is not. A sympathetic reader would care because a fast, query-free structural pre-screen at model intake could catch modified or mislabeled open-weight models before deployment, though the paper itself frames AMS as a triage tool rather than a standalone safety gate.

What carries the argument

The central object is the class-separation statistic sigma, computed as the difference between the mean projections of harmful and benign activations onto a contrastive direction vector, divided by the pooled standard deviation. That direction vector is the normalized difference of class centroids from 16 contrastive prompt pairs, extracted at the layer in the 40-80% depth range that maximizes separation. Tier 1 thresholds sigma into PASS, WARNING, and CRITICAL bands, while Tier 2 compares the direction vector's cosine similarity and separation drift against a stored baseline from a clean reference model. The two tiers, taken together, encode the four-class taxonomy and carry the argument.

What would settle it

Collect a sample of known-safe, independently verified instruction-tuned models from the broader ecosystem and run AMS Tier 1 on each: if a substantial fraction fall below the 3.5-sigma PASS threshold, the threshold's false-positive rate is too high for deployment as a gate. Alternatively, run the same scan on a held-out set of known-abliterated and uncensored models to estimate the true-positive rate outside the 14-model calibration set.

Watch

Extended reading notes

Core claim

Safety training creates measurable separation between harmful and benign content classes in mid-layer activation space; certain safety modifications collapse or rotate this structure, while others leave it intact. The paper claims that AMS's Tier 1 sigma-threshold detects training removal and weight-orthogonalization abliteration, that Tier 2 direction-similarity verification detects rotation-without-collapse abliteration, and that behavioral fine-tuning that preserves activation geometry is undetectable by activation-only probing. Across 14 models, sigma on the harmful-content concept predicts behavioral compliance with Pearson r = -0.546 (p = 0.043), and leave-one-out cross-validation of thresholds achieves 71% accuracy (10/14). The paper also claims a four-class mechanistic taxonomy: training removal collapses cluster separation, weight-orthogonalization abliteration collapses and rotates the refusal direction, rotation-without-collapse abliteration preserves separation while rotating direction, and behavioral fine-tuning preserves both magnitude and direction.

Load-bearing premise

Tier 1 assumes that intact safety training reliably keeps the harmful-content separation above 3.5 sigma in the 40-80% depth band, so that a lower value means tampering; the paper's own Mistral-7B-Instruct-v0.3 result shows a legitimate instruction-tuned release can fall in the CRITICAL band, leaving population-level false-positive rates unquantified.

Editorial extensions

If this is right

Models that remove safety training entirely, such as base models and from-scratch uncensored fine-tunes, can be flagged in 10-40 seconds without running any behavioral prompts.
Weight-orthogonalization abliteration is caught by both tiers, so a model that fails Tier 1 can be cross-checked by Tier 2 as defense in depth.
Rotation-without-collapse abliteration, which passes Tier 1 because it preserves cluster separation, is caught by Tier 2's direction-similarity check, making Tier 2 the load-bearing check for that attack class.
A structural PASS verdict cannot certify behavioral safety: the paper documents a model that preserves both separation and direction while complying with 97% of harmful prompts, so behavioral evaluation remains necessary.
Because the paper recommends the bootstrap confidence-interval lower bound rather than the single-run point estimate, borderline PASS classifications should be treated as statistically uncertain rather than definitive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own results imply that label-based trust in model registries is unreliable in both directions: a model labeled instruction-tuned can behave as unsafe, and a model labeled uncensored can pass the structural scan.
A natural testable extension, which the paper identifies as future work, is to calibrate sigma thresholds on an independent set of models outside the 14 used here and then measure population-level false-positive and false-negative rates.
The class-(iv) blind spot suggests that the next place to look for behavioral-only modifications is the output layer and refusal-token decoding behavior, since mid-residual activation geometry is preserved in that class.
Because bootstrap confidence intervals are wide and often cross the PASS threshold, the paper's own statistics imply that a single scan should be used as a screening instrument and that repeated scans over different contrastive pair sets would give a more stable verdict.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this paper is worth reading for its taxonomy and its unusually candid failure-mode analysis, but the central Tier 1 claim is internally inconsistent with the paper's own appendix data. That contradiction needs to be fixed before the detection result can be trusted.

What's genuinely new: the four-class taxonomy of safety-training modifications. In particular, the finding that two 'abliterated' models are mechanistically different—Llama weight-orthogonalization collapses separation and rotates the refusal direction, while the Gemma variant preserves separation and only rotates the direction—is a real contribution. The DarkIdol case, where behavioral compliance is high but activation geometry is preserved, is an important documented failure mode. The paper also ships code and data (GitHub + Zenodo DOI), reports bootstrap CIs, and is honest about the calibration-evaluation coupling in Section VIII-D(a). That level of transparency is rare.

The soft spots are substantial, but mostly acknowledged. The LOOCV accuracy of 71% on 14 models with thresholds derived from the same set is exploratory, not a population-level estimate. The behavioral correlation is fragile: Pearson r=-0.546 p=0.043 but Spearman p=0.13; with n=14 that's weak evidence. The wide bootstrap CIs (62% of cells cross the PASS threshold) mean single-run sigma values are not reliable.

The unacknowledged problem is worse. The abstract and Section VII-A claim that base models collapse to 0.5–1.4 sigma, but Appendix D Table V lists Gemma-2-9b (base) at sigma_harmful=5.29 (PASS) and Qwen2.5-7B (base) at 2.75 (WARNING). Those models are not in the 14-model validation set, so the LOOCV never tests whether the 3.5 sigma threshold distinguishes instruction-tuned from base models across architectures. If those Appendix D numbers are right, a base Gemma model would pass Tier 1 as a safe model, and the entire premise of Tier 1 as a baseline-free generic safety check fails. This is an internal contradiction, not just an external calibration concern. Additionally, Mistral-7B-Instruct-v0.3, a standard instruction-tuned release, falls in the CRITICAL band; the paper interprets that as a correct flag, but it equally shows a legitimate model can be flagged, so the false-positive rate on the broader population is unknown.

Who gets value: people working on model provenance, supply chain security, and anyone building pre-deployment scanners. The taxonomy alone is worth citing. But the detection claims need a serious revision: either correct the base-model values or restrict the method to per-architecture calibrated thresholds. This deserves peer review, and I'd send it to a good security venue with instructions to push on the base-model inconsistency.

Recommendation: read it, cite the taxonomy, but don't rely on Tier 1 without independent calibration.

Referee Report

6 major / 5 minor

Summary. The paper introduces AMS (Activation-based Model Scanner), a tool that detects modifications to safety training in language models by measuring geometric properties of safety-relevant concepts in activation space. It defines a four-class taxonomy of safety modifications (training removal, weight-orthogonalization abliteration, rotation-without-collapse abliteration, and behavioral fine-tuning), proposes a two-tier detection framework (Tier 1 sigma-thresholding, Tier 2 direction-similarity verification), and validates it on 14 model configurations spanning Llama, Gemma, Qwen, and Mistral families. The paper reports leave-one-out cross-validation accuracy of 71%, bootstrap confidence intervals on sigma estimates, a Pearson correlation of r = -0.546 (p = 0.043) between harmful-content sigma and behavioral compliance, and explicitly documents two Tier-1 false negatives and a fundamental class-(iv) blind spot. Code, contrastive pair datasets, baselines, and raw outputs are released open-source.

Significance. If the central claims held, AMS would be a practically useful fast pre-screen for safety-modified open-weight models, and the proposed four-class taxonomy is a valuable organizing principle for the security community. The paper has notable strengths: it ships open-source code and data, reports bootstrap uncertainty honestly, acknowledges the upward bias in the sigma estimator, and explicitly discusses calibration-evaluation coupling and failure modes instead of hiding them. However, the detection claim is currently undermined by an internal contradiction between the abstract's characterization of base models and the values reported in Appendix D, and the out-of-sample claims are weaker than presented. The taxonomy itself is plausible and worth publishing after the load-bearing issues are addressed.

major comments (6)

[Appendix D, Table V] The base-model values in Appendix D contradict the central class-(i) premise stated in the abstract and in Section VII-A(5) that base models collapse to 0.5-1.4 sigma. Table V lists Gemma-2-9b (base) at sigma_harmful = 5.29 (PASS) and Qwen2.5-7B (base) at 2.75 (WARNING), and neither model is part of the 14-model validation set in Section VI. The LOOCV in Section VII-C therefore never tests whether the 3.5 sigma PASS threshold separates instruction-tuned from base models across architectures. If these Appendix D values are accurate, a model with no safety training can exhibit the same harmful-content separation as an instruction-tuned model, invalidating Tier 1's baseline-free claim; if they are stale or from a different experiment round, they must be corrected. Additionally, Table V is labeled 'Complete Tier 1 results' but omits several Table I models (Mistral-7B-Instruct-v0.3, Llama-3.2-1B-Instruct, gemma-2-2b-it, dolphin-2.9.4-llama3.1-8b, Llama-3.2-3B base) and includes four models not in Table I, so the appendix cannot currently be used to reconcile the contradiction.
[Section VIII-D(a), Section VII-C] The PASS and CRITICAL thresholds were originally calibrated on the same 14-model set used to report detection accuracy, and Section VIII-D(a) explicitly admits that the LOOCV reframing does not fully decouple calibration from evaluation because the universe of cases is the same. The abstract's statement that 'Leave-one-out cross-validation of thresholds achieves 71% accuracy' is therefore not an out-of-sample detection rate. Given that the central contribution is a threshold-based detector, the paper needs either an independent calibration set, a genuinely held-out evaluation set, or a clearly labeled statement that 71% is an exploratory in-sample-ish result rather than a generalization estimate.
[Section VII-D, Table I] The bootstrap analysis shows that the reported point estimate for Llama-3.2-3B-Instruct (8.37 sigma) lies outside its own 95% CI [3.64, 7.95], and 26 of 42 cells have CIs crossing the 3.5 sigma PASS threshold. Because the sigma estimator selects the direction that maximizes separation on the same 16-pair sample, the single-run classifications in Table I are systematically optimistic. The paper acknowledges this and recommends using the CI lower bound, but the abstract and conclusion still state that Tier 1 detects classes (i) and (ii) without conditioning on this uncertainty. Under the CI-lower-bound convention that the paper itself recommends, several PASS verdicts are not statistically distinguishable from WARNING, so the detection claims should be restated accordingly.
[Section VII-E] The behavioral-correlation claim rests on Pearson r = -0.546 (p = 0.043), but the rank-order Spearman correlation is rho = -0.423 (p = 0.13), which is not significant at alpha = 0.05. With n = 14 and two documented high-sigma/high-compliance outliers (gemma-2-9b-it-abliterated and DarkIdol), the linear correlation is fragile and likely driven by category-level clustering rather than a robust within-category relationship. The text is honest about this, but the conclusion's bullet list reports only the Pearson value and p-value, which overstates the evidence. The conclusion should include the Spearman result or explicitly frame the correlation as directional and non-robust.
[Section VI(b), Table I] Behavioral compliance is measured by a refusal-phrase string-match heuristic restricted to the first 250 characters, with only spot-checks mentioned for validation. Since the compliance rates in Table I drive the Pearson correlation in Section VII-E and are used to interpret several borderline sigma cases, the reliability of this heuristic is load-bearing. PARTIAL responses and non-refusal safe responses could be misclassified by string matching, and no manual label set or inter-annotator agreement is reported. The paper should either provide a manual-label subsample validation or explicitly downgrade any compliance-based interpretation that depends on the exact rate.
[Section III-c, Table I] Mistral-7B-Instruct-v0.3, a standard instruction-tuned release, measures sigma_harmful = 1.39 and is classified CRITICAL, and the paper interprets this as a correct flag of a de facto unsafe model. Without a ground-truth modification label, the same observation demonstrates that a legitimate instruction-tuned model can fall in the CRITICAL band, so the false-positive rate on the broader population of instruction-tuned models is unknown. This is not merely an external calibration concern; it is an internal example that the Tier-1 threshold does not reliably separate intact safety training from modified safety training for at least one major model family.

minor comments (5)

[Appendix A] The vLLM integration example sets VLLM_ALLOW_INSECURE_SERIALIZATION = '1' without a security caveat; since AMS is intended for supply-chain verification, the example should warn that insecure deserialization can introduce a code-execution risk when scanning untrusted model repositories.
[Figure 4] The y-axis of the threshold-sensitivity plot extends above 100%, and the phrase '100% correct plateau' may mislead readers into interpreting it as out-of-sample performance; the caption already says it is internal consistency, but the axis and label should be made consistent with that caveat.
[Table II] The 'Verified' column for gemma-2-9b-it-abliterated uses 'x dagger' with a footnote saying it is detected by Tier 2 despite passing Tier 1; this is confusing because the column header implies a binary verification verdict and the dagger is not a standard marker. Clarify whether the row means 'Tier 2 fails verification' or 'Tier 2 detects the modification'.
[Appendix D] The appendix would benefit from a brief note explaining that some listed models (Gemma-2-9b base, Qwen2.5-7B base, Lexi-Uncensored, LLama-3-8b-Uncensored) are not in the 14-model validation set and from an explicit reconciliation with Table I, since the current labeling as 'Complete Tier 1 results' is inaccurate.
[Ethics Statement] There is a typo in 'LLama-3-8b-Uncensored' (capitalization of 'LLama') and the sentence about 'no harmful content was generated' should be reconciled with the behavioral evaluation that measured compliance on JailbreakBench prompts, which presumably elicited harmful completions from several models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the thresholds are calibrated on the evaluation set, but the paper discloses this and the LOOCV and behavioral correlation provide partially independent checks.

full rationale

The derivation chain is not circular. The sigma metric is computed from contrastive-pair centroid differences (Eq. 1–2) and is not fit to modification labels; layer and direction selection maximize separation on the contrastive pairs, not on safety status, and the resulting upward bias is disclosed in Section VII-D. The PASS/WARNING/CRITICAL thresholds are indeed calibrated on the same 14-model set (Section IV-E), and the paper explicitly acknowledges the calibration–evaluation coupling in Section VIII-D(a); however, the leave-one-out procedure recomputes thresholds from the remaining 13 models for each held-out model, so the reported 71% accuracy is not forced by construction. The behavioral correlation (Pearson r = −0.546) is computed from independently measured JailbreakBench compliance rates and is not an artifact of the sigma definition. The four-class taxonomy is grounded in measured activation signatures of named in-the-wild models, not assumed a priori. A separate non-circularity concern is that Appendix D, Table V lists Gemma-2-9b (base) at σ_harmful = 5.29 (PASS) and Qwen2.5-7B (base) at 2.75 (WARNING), contradicting the abstract's characterization of base models at 0.5–1.4σ and weakening the class-(i) generalization; this is an internal consistency/validity risk, not a circular derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

AMS is a tool, not a postulated entity; the four taxonomy classes are categories, not new entities. No new particle, force, or mechanism is introduced. The free parameters are the detection thresholds and hand-set hyperparameters, which are calibrated on the same data used for evaluation.

free parameters (5)

Tier 1 PASS threshold = 3.5 sigma
Calibrated as a midpoint heuristic on the 14-model validation set (Section IV-E(a) and Section VIII-D(a)).
Tier 1 CRITICAL threshold = 2.0 sigma
Calibrated on the same 14-model set; LOOCV PASS thresholds range from 2.97 to 4.55 across folds (Section VII-C).
Tier 2 direction similarity threshold = 0.8 cosine
Set by hand in Section IV-E(b) without independent calibration.
Tier 2 separation drift threshold = 20%
Set by hand in Section IV-E(b) without independent calibration.
Contrastive pair count per concept = 16
Chosen by the author; bootstrap CI width (median 3.36 sigma) shows high sensitivity to the pair set (Section VII-D).

assumptions (5)

domain assumption Safety-relevant concepts are encoded as linear directions in activation space.
Inherited from representation engineering [3] and refusal-direction work [1]; invoked in Sections II and IV-B.
ad hoc to paper The hand-crafted contrastive pairs isolate the target safety concept from confounds.
The pair design (Appendix C) is author-authored and not validated for concept purity.
ad hoc to paper The refusal-phrase string-match in the first 250 characters accurately measures compliance.
Section VI(b) says only 'spot-checks confirm the classification accuracy'; no quantified agreement is given.
domain assumption JailbreakBench's 20 stratified prompts are a representative sample of harmful behaviors.
Used to define compliance rates; Section VIII-F(g) notes prompt-set dependence.
domain assumption Bootstrap resampling of cached projections without new forward passes captures the full sampling uncertainty of sigma.
Section VII-D resamples the 16 pairs with replacement; this ignores prompt-distribution and layer-selection uncertainty.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Detecting Safety Training Modification in Language Models via Activation Analysis." pith.science (2026). https://pith.science/paper/WSDADBLB

@misc{pith2026260805578,
  author       = {Pith},
  title        = {Pith review of: Detecting Safety Training Modification in Language Models via Activation Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSDADBLB}},
  note         = {Machine review of arXiv:2608.05578}
}

read the original abstract

We introduce AMS (Activation-based Model Scanner), a tool that detects modifications to safety training in language models by measuring the geometric structure of safety-relevant concepts in activation space. Safety training creates measurable separation between harmful and benign content classes; certain safety modifications collapse or rotate this structure, while others leave it intact. We validate AMS across 14 model configurations spanning 4 architecture families (Llama, Gemma, Qwen, Mistral) and four safety-modification categories (instruction-tuned, base, abliterated, uncensored fine-tunes). Leave-one-out cross-validation of thresholds achieves 71% accuracy (10/14); bootstrap 95% confidence intervals on sigma point estimates have median width 3.4 sigma. We measure behavioral compliance on 20 stratified JailbreakBench prompts per model and find that sigma on the harmful-content concept predicts compliance with Pearson r = -0.546 (p = 0.043), directionally but with meaningful noise. Mechanistic analysis identifies a four-class taxonomy of safety-training modifications distinguished by activation-space signature: training removal collapses cluster separation; weight-orthogonalization abliteration both collapses separation and rotates the refusal direction; rotation-without-collapse abliteration preserves separation while rotating direction; and behavioral fine-tuning preserves both magnitude and direction. AMS's Tier 1 sigma-threshold detects the first two classes; Tier 2 direction-similarity verification detects the third. The fourth is undetectable by activation-only probing and represents a documented failure mode. We discuss threshold calibration, limitations of single-run measurement, and the open problem of detecting behavioral-only safety modifications.

Figures

Figures reproduced from arXiv: 2608.05578 by the authors.

**Figure 1.** Per-model σharmful vs. behavioral compliance rate on 20 stratified JailbreakBench harmful behaviors. Colors denote category (instruction-tuned, abliterated, uncensored fine-tune, base). Pearson r = −0.546 (p = 0.043), Spearman ρ = −0.423 (p = 0.13). The two off-trend points in the upperright (gemma-2-9b-it-abliterated at (4.54, 1.00), DarkIdol at (5.45, 0.97)) are AMS Tier-1 false negatives discussed in Section VII… view at source ↗

**Figure 2.** Layer-wise σharmful for two AMS Tier-1 false negatives (DarkIdol, gemma-2-9b-it-abliterated) and two controls (gemma-2-9b-it as PASS, dolphin-2.9-llama3-8b as CRITICAL). Open circles mark each model’s AMSselected optimal layer. Dashed lines at the 3.5σ and 2.0σ reference thresholds. Neither false-negative collapses at any layer in the sweep range; gemmaabliterated’s profile tracks the safe gemma-clean parent laye… view at source ↗

**Figure 3.** Per-layer refusal-direction shift between each base/abliterated pair. Top panel: cosine similarity between base and abliterated direction vectors. Bottom [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Threshold sensitivity (internal consistency) on the 14-model validation set. The flat 100% region in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 10 canonical work pages

[1]

Refusal in Language Models Is Mediated by a Single Direction.Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pp

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Pan- ickssery, Wes Gurnee, and Neel Nanda. Refusal in Language Models Is Mediated by a Single Direction.Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pp. 136037–136083, 2024

work page 2024
[2]

Dolphin: An uncensored, unbiased language model

Eric Hartford. Dolphin: An uncensored, unbiased language model. Blog post, 2023

work page 2023
[3]

Representation engineering: A top-down approach to AI transparency.arXiv preprint arXiv:2310.01405, 2023

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann- Kathrin Dombrowski, et al. Representation engineering: A top-down approach to AI transparency.arXiv preprint arXiv:2310.01405, 2023

arXiv
[4]

Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023

Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023

arXiv 2023
[5]

Instructional fingerprinting of large language models.arXiv preprint arXiv:2401.12255, 2024

Weiyan Xu, Zhengyang Shan, Gail Weiss, Yoav Goldberg, and Anna Googins. Instructional fingerprinting of large language models.arXiv preprint arXiv:2401.12255, 2024

arXiv 2024
[6]

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.arXiv preprint arXiv:2402.04249, 2024

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.arXiv preprint arXiv:2402.04249, 2024

arXiv 2024
[7]

ToxiGen: A Large-Scale Machine- Generated Dataset for Adversarial and Implicit Hate Speech Detection

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen: A Large-Scale Machine- Generated Dataset for Adversarial and Implicit Hate Speech Detection. InACL, 2022

work page 2022
[8]

WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs.arXiv preprint arXiv:2406.18495, 2024

Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs.arXiv preprint arXiv:2406.18495, 2024

arXiv 2024

Show all 18 references

[9]

Sokhansanj

Benjamin A. Sokhansanj. Uncensored AI in the Wild: Tracking Publicly Available and Locally Deployable LLMs.Future Internet, 17(10):477, 2025

2025
[10]

XBreaking: Understanding how LLMs security alignment can be broken.arXiv preprint arXiv:2504.21700, 2025

Marco Arazzi, Mauro Conti, Antonino Guiliano, and Vignesh Kumar Kembu. XBreaking: Understanding how LLMs security alignment can be broken.arXiv preprint arXiv:2504.21700, 2025

2025
[11]

Defending Large Language Models Against Attacks With Residual Stream Acti- vation Analysis.arXiv preprint arXiv:2406.03230, 2024

Amelia Kawasaki, Andrew Davis, and Houssam Abbas. Defending Large Language Models Against Attacks With Residual Stream Acti- vation Analysis.arXiv preprint arXiv:2406.03230, 2024

2024 arXiv
[12]

Safety Layers of Aligned Large Language Models: The Key to LLM Security.arXiv preprint arXiv:2408.17003, 2024

Shen Chen, Yingchun Wang, Xiaoran Fan, Xianjun Yang, Zhipeng Cai, Chao Du, Wei Lu, Zhixing Tan, and Jiajun Chen. Safety Layers of Aligned Large Language Models: The Key to LLM Security.arXiv preprint arXiv:2408.17003, 2024

2024 arXiv
[13]

Probing Classifiers: Promises, Shortcomings, and Advances.Computational Linguistics, 48(1):207–219, 2022

Yonatan Belinkov. Probing Classifiers: Promises, Shortcomings, and Advances.Computational Linguistics, 48(1):207–219, 2022

2022
[14]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. InSOSP, 2023

2023
[15]

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/- False Datasets.arXiv preprint arXiv:2310.06824, 2023

Samuel Marks and Max Tegmark. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/- False Datasets.arXiv preprint arXiv:2310.06824, 2023

2023 arXiv
[16]

Lo- cating and Editing Factual Associations in GPT

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Lo- cating and Editing Factual Associations in GPT. InNeurIPS, 2022

2022
[17]

ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation

Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation. InProceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS ’19), pages 1265–128...

2019
[18]

VLLM_ALLOW_INSECURE_SERIALIZATION

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym An- driushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nico- las Flammarion, George Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. JailbreakBench: An Open Robustness Benchmark for Jail- breaking Large...

2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Refusal in Language Models Is Mediated by a Single Direction.Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pp

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Pan- ickssery, Wes Gurnee, and Neel Nanda. Refusal in Language Models Is Mediated by a Single Direction.Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pp. 136037–136083, 2024

work page 2024

[2] [2]

Dolphin: An uncensored, unbiased language model

Eric Hartford. Dolphin: An uncensored, unbiased language model. Blog post, 2023

work page 2023

[3] [3]

Representation engineering: A top-down approach to AI transparency.arXiv preprint arXiv:2310.01405, 2023

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann- Kathrin Dombrowski, et al. Representation engineering: A top-down approach to AI transparency.arXiv preprint arXiv:2310.01405, 2023

arXiv

[4] [4]

Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023

Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023

arXiv 2023

[5] [5]

Instructional fingerprinting of large language models.arXiv preprint arXiv:2401.12255, 2024

Weiyan Xu, Zhengyang Shan, Gail Weiss, Yoav Goldberg, and Anna Googins. Instructional fingerprinting of large language models.arXiv preprint arXiv:2401.12255, 2024

arXiv 2024

[6] [6]

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.arXiv preprint arXiv:2402.04249, 2024

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.arXiv preprint arXiv:2402.04249, 2024

arXiv 2024

[7] [7]

ToxiGen: A Large-Scale Machine- Generated Dataset for Adversarial and Implicit Hate Speech Detection

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen: A Large-Scale Machine- Generated Dataset for Adversarial and Implicit Hate Speech Detection. InACL, 2022

work page 2022

[8] [8]

WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs.arXiv preprint arXiv:2406.18495, 2024

Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs.arXiv preprint arXiv:2406.18495, 2024

arXiv 2024