Pith. sign in

REVIEW 4 major objections 6 minor 54 references

Scenario wrappers jailbreak models by activating internal directions that suppress refusal, and those directions can be turned into reusable attack scenarios.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 20:44 UTC pith:BF3P6B2J

load-bearing objection Solid mechanistic red-teaming paper: scenario directions suppress a refusal CAV, and the SAE-to-scenario pipeline actually raises black-box ASR and partly transfers. the 4 major comments →

arxiv 2607.23496 v1 pith:BF3P6B2J submitted 2026-07-26 cs.AI cs.CR

Do LLMs Know Their Vulnerable Scenarios?

Ziheng Peng , Huiqi Deng , Haoran Jing , Xuankun Rong , Jiahui Han , Xiting Wang , Na Zou , Xia Hu This is my paper
classification cs.AI cs.CR
keywords large language modelsjailbreak attacksred teamingmechanistic interpretabilitysparse autoencodersrefusal directionconcept attributionAI safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Safety-aligned language models refuse direct harmful requests, but the same requests often succeed when wrapped in a particular context—an academic framing, a code task, a fictional scene. This paper argues that those wrappers work by lighting up internal “scenario directions” that causally weaken the model’s refusal score. The authors turn that mechanism into a discovery method, Concept2Scenario: a sparse autoencoder carves the model’s hidden states into many concepts; each concept is steered to measure how much it suppresses refusal; the strongest suppressors are translated into plain-language scenarios and checked to make sure the scenarios reactivate the same concepts. Those scenarios then act as reusable priors for existing black-box jailbreaks, raising average attack success by as much as 18.2 points on open models and still helping against closed models they were never tuned on. Interaction analysis further finds pairs of scenarios that suppress refusal more than the sum of their parts and let multi-turn attacks succeed in fewer queries.

Core claim

Scenario-wrapped harmful prompts activate internal scenario directions whose causal steering consistently reduces a model’s refusal score. By attributing that suppression to sparse-autoencoder concepts and translating the concepts into natural-language scenarios, Concept2Scenario discovers reusable, transferable vulnerability priors—and synergistic combinations—that improve black-box jailbreak success.

What carries the argument

Concept2Scenario: SAE concepts are intervened along decoder directions to measure refusal-score change (Δi); high-suppressing concepts are interpreted and turned into scenario templates that must re-activate the source concept (the c→s→c loop); pairwise interaction terms Γi,j pick synergistic combinations.

Load-bearing premise

Steering sparse-autoencoder features at one layer with a fixed activation scale really isolates the concepts that, once phrased as natural-language scenarios, both re-fire those features and cause behavioral jailbreaks.

What would settle it

If scenario templates that pass the reactivation gate do not raise attack success over matched LLM-generated scenario controls on the same models and attacks—or if steering the corresponding SAE directions fails to lower refusal scores on held-out harmful requests—the central causal story fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Red-teamers can mine scenario libraries from open models’ internals and reuse them as priors inside existing black-box attacks without redesigning the attacks.
  • Some scenario-level refusal weaknesses appear shared across model families, so open-model discoveries can transfer to closed models such as GPT-5, Claude-Haiku-4.5, and Gemini-3-Flash.
  • Interaction attribution yields scenario pairs that beat single scenarios and additive baselines and reduce the number of turns iterative attacks need.
  • Different models show distinct vulnerable-domain profiles (web, code, academic), so scenario libraries should be model-specific rather than one-size-fits-all.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If shared scenario directions are real, defenders might monitor or damp those directions at inference time as a complement to output filters.
  • The thin tail of refusal-suppressing concepts suggests safety fine-tuning may leave a small set of high-leverage blind spots rather than a diffuse weakness.
  • A natural extension is to close the loop: use discovered scenarios as hard negatives during alignment and test whether the corresponding SAE features lose their refusal-suppressing effect.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper asks whether scenario-based jailbreaks act through identifiable internal representations, and answers with two contributions. First, using paired direct/scenario-wrapped prompts, the authors extract "scenario directions" and show that adding them to the residual stream reduces a refusal-CAV score (Appendix A). Second, they propose Concept2Scenario: train a TopK SAE (D=65,536, k=64) per model family, intervene on each decoder direction with a per-concept calibrated scale, rank concepts by refusal-score suppression Δ_i (Eq. 8), translate the top concepts into natural-language scenarios via an analysis LLM with an activation-rank gate (§4.4, Appendix B), and identify synergistic pairs via an interaction score Γ_i,j (Eq. 10). The discovered scenario libraries are plugged into six black-box attacks and evaluated on three open-source models and two benchmarks (Table 1), with transfer to GPT-5, Claude-Haiku-4.5, and Gemini-3-Flash (Table 2). Average ASR improves by up to +18.2 points (Llama-3.1-8B, GuidedBench), and interaction-guided combinations beat additive/random controls in 5 of 6 settings (Figure 3).

Significance. If the mechanistic attribution holds, this is a useful bridge between interpretability and red-teaming: it would explain why scenario wrapping works and turn that explanation into a discovery procedure. The empirical package is genuinely strong in places: multi-model and multi-attack coverage; an LLM-generated-scenario control (w. LLM) that is a fairer baseline than "no scenario" and which, encouragingly, behaves differently from the method's scenarios on well-aligned targets; hyperparameter sweeps for the CAV regularizer κ, probe token site, and library size |S|; per-concept intervention-scale calibration to remove an activation-magnitude confound (§5.3); an SAE mixing ablation showing no reconstruction penalty (Appendix F); and a non-trivial transfer result to frontier closed models (e.g., +27.5 on GPT-5/PAIR with Ministral-derived scenarios). The w. LLM control and the activation-gate verification loop are exactly the kinds of checks this community often omits, and they deserve explicit credit. The discovered scenarios (Table 3) are interpretable and plausibly actionable for defense.

major comments (4)
  1. [§4.3, Eq. (8); Appendix A, Table 4] §4.2–4.3 and Appendix A: the entire mechanistic claim — that scenario concepts causally weaken refusal — is established against a proxy endpoint, the refusal-CAV score r(x), and is never tied to behavior at the attribution stage. Appendix A steers with a very large α=100 at layer 27, filters to n=50 high-refusal prompts, and reports only Δr; it reports neither the resulting compliance/ASR nor matched-norm controls (random directions, CAV-orthogonal directions, or non-scenario mean-difference directions). An intervention that rotates the representation off the learned refusal separator will move r even if the generated text still refuses. The concept screen inherits this: Δ_i (Eq. 8) is purely CAV movement under decoder injection. The black-box ASR gains in Table 1 show the final scenarios work, but not that the negative-Δ ranking is what identifies them. A load-bearing, inexpensive fix:a
  2. [§4.3, Eq. (8); Appendix A] Continuing the previous point, concretely: (i) in Appendix A, report behavioral refusal rate under the same steering, plus a matched-norm random-direction control at α=100 — given α=100 and unit-norm d_s, the injection norm is enormous relative to typical residual-stream norms, so a generic-perturbation explanation must be excluded; (ii) add a rank-shuffled control in which scenarios are drawn from random or positive-Δ concepts and run through the same c→s→c translation and attack pipeline, showing that Δ ranking predicts downstream ASR; and (iii) show a correlation between Δ_i and per-scenario behavioral ASR on held-out requests. Without at least (i) and (ii), the paper supports 'a useful scenario-generation pipeline' but not the central attribution claim that internal scenario concepts causally suppress refusal.
  3. [Table 2; §5.1; Abstract] Table 2 and the abstract's transfer claim: the closed-source results are too noisy to support 'some scenario-level refusal vulnerabilities are shared across model families' as stated. On GPT-5/PAIR, w. Ours(Qwen) gives −11.5 and w. Ours(LLaMA) gives −6.0 while w. Ours(Ministral) gives +27.5 — the sign of the effect depends on the source model, and on Gemini-3-Flash/Tree-Attack w. Ours(Ministral) is −7.5 while the other two source sets help. Only two attacks (PAIR, Tree-Attack) are tested, and no run counts or variance estimates are reported for the closed models (Table 1 averages over three runs; Table 2 does not say). The honest summary is that transfer is real but source-model-dependent and partly idiosyncratic; the abstract and §5.1 should be tempered accordingly, and run counts/standard errors added.
  4. [Table 1; Abstract] Table 1 and the headline '+18.2 percentage points': this figure is the maximum across models and is driven largely by cells with no headroom or tiny baselines (e.g., DeepInception on Llama, baseline 33.3 → 84.5; AutoDAN-T and DeepInception on Qwen have baselines ≤2.0 where +1.5/+7.5 gains are within noise of a near-zero attack). Meanwhile Ministral sits at 81.7–98.5 baseline ASR, where gains are ceiling-limited and uninformative (Avg.Δ +5.7/+2.0), and several w. Ours cells are negative (Qwen PAIR −4.6, Ministral Tree-Attack −1.3 on HarmBench). The abstract should report the effect size with its conditioning (model, benchmark, attack) rather than 'up to 18.2', and Table 1 should include standard errors across the three runs so readers can judge which deltas are meaningful — especially the small ones on Qwen and the near-ceiling ones on Ministral.
minor comments (6)
  1. [Figure 1] Figure 1 caption/artifact: the body text above Figure 1 contains what appears to be raw LaTeX from the figure ('The LaTeX below needs improvement... \multicoltown{2| |c}{...}'), and the caption itself is garbled in the arXiv render. Please clean this up.
  2. [Figure 3; §4.3, Eq. (10)] Figure 3: the interaction-synergy claim rests on ten fixed configurations per setting, with first-query advantages of 2.5–5.6 pp and one setting (Qwen/PAIR) where Single overtakes. This is fine as a case study but should be labeled as such; please state n and give some variability estimate. Also, the Γ<0 → natural-language synergy bridge assumes representation-space non-additivity predicts prompt-space synergy; a sentence acknowledging this assumption and its failure mode (Qwen/PAIR) would help.
  3. [Various] Typos and notation: 'vulunerable' (§4.4), 'sscore' and 'an trained refusal CAV' (Appendix A), 'CA V' with a space (§5.2, Figure 9), 'Attack Successful Rate' (Tables 1–2 captions), 'close-source' (§5.1, should be 'closed-source'), inconsistent model naming ('LLaMA' vs 'Llama'; 'Ministral-3-8B' vs 'Ministral-3-8B-Instruct' in Figure 6).
  4. [§4.4; Appendix B.3] §4.4 / Appendix B: the activation gate uses R_gate=1,000 of D=65,536 (top ~1.5%) with up to 3 retries. Please report the pass rate of the gate and how many candidate concepts yield validated scenarios per model — this affects how selective the pipeline actually is, and whether the library is a thin tail or a broad sample of negative-Δ concepts.
  5. [§5.3, Figure 5b] §5.3: the claim that Qwen's refusal is 'markedly more robust to single-concept intervention' (max ~2% vs ~30% normalized suppression) is interesting but is confounded with the observation that Qwen's base model already refuses 97.5% (Table 7) — normalized Δ_i/̄r may not be comparable across models with very different baseline scores. A brief discussion or an absolute-scale companion figure would help.
  6. [§2.2] Missing related work: the paper would benefit from citing and contrasting with work on refusal-feature ablation changing actual generations (beyond [3, 34]), e.g., activation-engineering/steering-vector studies that do report behavioral endpoints, since the distinction between probe-score movement and behavioral non-refusal is exactly the paper's central gap.

Circularity Check

1 steps flagged

No load-bearing circularity: refusal attribution and ASR gains are externally measured; only the optional c→s→c activation gate mildly forces concept re-firing by construction.

specific steps
  1. self definitional [§4.4 Translating refusal-suppressing Concepts (activation gate / c→s→c loop)]
    "The translated scenario should activate the source concept high. We verify this by running T_σi(x) through the target model and ranking the source concept activation c(i) among all concept activations. If the rank is low, the scenario does not sufficiently reflect the concept’s meaning, so we repeat the previous steps for up to a fixed number of retries. Each concept that passes yields one jailbreak scenario"

    Accepted scenarios are those revised until they re-fire the source SAE feature (rank gate R_gate). For the sub-claim that each library scenario activates its attributed concept, success is partly by construction of the retry/acceptance rule rather than an out-of-sample prediction. This does not circularize Δ_i screening or external ASR evaluation, which remain independent of the gate.

full rationale

The paper’s derivation chain is not circular in the strong sense. Scenario directions are estimated from paired direct vs. wrapped prompts and tested by residual-stream steering against an independently fit refusal CAV (Appendix A; §4.2–4.3); SAE concepts are ranked by interventional Δ_i on that score, not by attack success. Downstream claims rest on black-box ASR against six existing attacks, two benchmarks, three open-source models, closed-source transfer, and an LLM-generated scenario control (Tables 1–2, Figure 3)—none of which define scenarios as ASR maximizers and then re-report that maximum as a mechanistic finding. Self-citations (e.g., prior CAV/safety work) are background, not uniqueness theorems that force the method. The only mild construction is the translation loop in §4.4: candidate natural-language scenarios are retried until the source SAE feature ranks highly, so “scenario reactivates concept” is partly enforced by the acceptance gate rather than predicted. That gate does not define Δ_i, Γ_i,j, or ASR, so it does not collapse the central causal or red-teaming claims. Overall circularity is negligible.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The central story rests on standard linear interpretability tooling (CAV refusal probe, TopK SAE dictionary, residual-stream steering) plus several paper-specific modeling choices: which token carries refusal, which layer hosts concepts, how hard to steer each feature, how to gate LLM translations, and that non-additive Γ identifies useful natural-language combinations. No new physical entities; the invented pieces are methodological constructs.

free parameters (6)
  • SAE dictionary size D and sparsity k = D=65536, k=64
    Fixed at D=65536, k=64; defines the concept basis used for all attribution.
  • Intervention scale s_i = 25th percentile of positive activations
    Per-concept steering magnitude set to the 25th percentile of positive TopK-gated activations on pretraining text; directly controls measured Δ_i.
  • Refusal CAV inverse L2 strength κ = Qwen κ=100; LLaMA κ=5e-2; Ministral κ=1e-2
    Chosen per model from accuracy/stability sweep; defines the refusal score r(x).
  • Scenario library size |S| = model-dependent, rings in Fig. 4
    Hyperparameter swept 10–50; selected per model (e.g., Qwen peak at 40) and used as attack prior.
  • Activation gate rank R_gate and retry budget = R_gate=1000, max 3 attempts
    Translation accepted only if source feature ranks within top 1000; up to 3 retries—filters which concepts become scenarios.
  • Attribution sample size N and SAE/CAV layers = N=5000; layers model-specific
    N=5000 BeaverTails requests for Δ_i; layer choices (e.g., scenario dir layer 27, refusal layer 31 on Llama) are design choices affecting all causal scores.
axioms (5)
  • domain assumption Refusal tendency is adequately captured by a linear CAV on a single post-prompt chat-template token in the final layer.
    §4.2 and §5.2 select the token by base→instruct probe accuracy gain; all causal effects are read out through this scalar.
  • domain assumption TopK SAE features provide a broad, intervention-ready concept dictionary whose decoder columns are valid causal directions in residual space.
    Standard SAE interpretability assumption; underpins individual and interaction attribution (§4.2–4.3).
  • domain assumption Additive residual-stream steering along decoder directions isolates concept-level causal effects on refusal despite nonlinear downstream computation.
    Intervention protocol in Eqs. 7–10; interaction term Γ acknowledges nonlinearity but still relies on the same steering model.
  • ad hoc to paper An external analysis LLM can recover a stable natural-language trigger mechanism from top-activating passages that reactivates the intended feature.
    §4.4 translation loop and Appendix B; required to turn internal features into actionable scenarios.
  • ad hoc to paper Negative interaction Γ_i,j in representation space predicts synergistic natural-language scenario combinations for black-box attacks.
    Eq. 10 and Fig. 3; bridges non-additive internal effects to multi-scenario prompts.
invented entities (3)
  • Concept2Scenario framework independent evidence
    purpose: End-to-end map from refusal-suppressing SAE concepts to natural-language jailbreak scenario libraries and combinations.
    Core methodological contribution; not an ontological claim about nature, but a new pipeline entity the results depend on.
  • Scenario direction d_s independent evidence
    purpose: Mean representation difference between scenario-wrapped and direct harmful prompts used for causal steering tests.
    Defined in Appendix A; supports the mechanistic claim that wrapping acts through internal directions.
  • Interaction attribution score Γ_i,j independent evidence
    purpose: Non-additive joint refusal-suppression score for concept pairs used to select synergistic scenario combinations.
    Eq. 10; paper-specific construct linking dual interventions to combination search.

pith-pipeline@v1.2.0-grok45-kimik3 · 27950 in / 3789 out tokens · 76655 ms · 2026-07-30T20:44:13.870723+00:00 · methodology

0 comments
read the original abstract

Safety-aligned large language models are trained to refuse harmful requests, yet embedding the same requests in particular scenarios can bypass their safeguards. Existing red-teaming methods empirically identify effective scenarios through observed attack outcomes, but why particular scenarios weaken refusal remains mechanistically unclear. Meanwhile, mechanistic interpretability studies have characterized both refusal directions and jailbreak-associated features, without explaining the relationship between the two representations. In this work, we show that scenario-wrapped prompts activate internal scenario directions whose causal steering consistently reduces refusal scores. Building on this finding, we propose \textsc{Concept2Scenario}, a concept-based attribution framework for vulnerable scenario discovery. It instantiates a broad concept space with a sparse autoencoder, attributes refusal suppression to individual concepts, translates the identified concepts into interpretable natural-language scenarios, and identifies synergistic scenario combinations through interaction attribution. Across three open-source models, two safety benchmarks, and six black-box jailbreak methods, the discovered scenarios serve as reusable priors that improve average attack success rates by up to $18.2$ percentage points. They also transfer to GPT-5, Claude-Haiku-4.5, and Gemini-3-Flash, suggesting that some scenario-level refusal vulnerabilities are shared across model families. Moreover, the identified combinations outperform their individual constituents and enable iterative attacks to succeed in fewer turns.

Figures

Figures reproduced from arXiv: 2607.23496 by Haoran Jing, Huiqi Deng, Jiahui Han, Na Zou, Xia Hu, Xiting Wang, Xuankun Rong, Ziheng Peng.

Figure 1
Figure 1. Figure 1: Mechanistic view of jailbreaks. A harmful request [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Post-prompt token selected to train the refusal [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance of interaction-guided scenario com [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Sorted per-concept statistics for the SAE. Each panel [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Domain profiles of the top-100 vulnerability scenar [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Case study of scenario combinations. (a) Joint re [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt used by the w. LLM control to generate scenarios with Gemini-3.1-Pro. We request 𝑁 = 50 scenarios per generation and average over three generations. request. This preserves AutoDAN-Turbo’s strategy-based gener￾ation interface while reducing refusals from the attacker model itself. D LLM-Generated Scenario Baseline The w. LLM control replaces our analysis-derived scenarios with scenarios produced by … view at source ↗
Figure 9
Figure 9. Figure 9: Selecting the refusal-probe inverse 𝐿2 strength 𝜅. Each panel shows the held-out CAV accuracy of the base and instruct probes (left axis) and the stability of their accuracy gain (Δ Std, right axis) across 𝜅. The dotted line marks the selected 𝜅. F.1 Architecture and Objective For a transformer layer ℓ, we extract the residual-stream activation at the output of the layer (resid_post) and train a TopK SAE [… view at source ↗
Figure 10
Figure 10. Figure 10: Reconstruction quality of the three training [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 13 linked inside Pith

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Anthropic. 2025. Introducing Claude Haiku 4.5. https://www.anthropic.com/ news/claude-haiku-4-5

    2025

  3. [3]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction.Advances in Neural Information Processing Systems37 (2024), 136037–136083

    2024

  4. [4]

    Yannick Assogba, Jacopo Cortellazzi, Javier Abad, Pau Rodriguez, Xavier Suau, and Arno Blaas. 2026. Sparse autoencoders are capable LLM jailbreak mitigators. arXiv preprint arXiv:2602.12418(2026)

    Pith/arXiv arXiv 2026

  5. [5]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862(2022)

    Pith/arXiv arXiv 2022

  6. [6]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

  7. [7]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2025. Jailbreaking black box large language models in twenty queries. In2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 23–42

    2025

  8. [8]

    Shuangjie Fu, Du Su, Beining Huang, Fei Sun, Jingang Wang, Wei Chen, Huawei Shen, and Xueqi Cheng. 2025. Jailbreak LLMs through Internal Stance Manip- ulation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 15455–15470

    2025

  9. [9]

    Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. Scaling and evaluating sparse autoencoders. InInternational Conference on Learning Representations, Vol. 2025. 26721–26754

    2025

  10. [10]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith

  11. [11]

    Google DeepMind. 2025. Gemini 3. https://deepmind.google/models/gemini/

    2025

  12. [12]

    Google DeepMind. 2026. Gemini 3.1 Pro Model Card. https://deepmind.google/ models/model-cards/gemini-3-1-pro/

    2026

  13. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset.Advances in Neural Information Processing Systems36 (2023), 24678–24704

    2023

  15. [15]

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models.Advances in Neural Information Processing Systems37 (2024), 47094– 47165

    2024

  16. [16]

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al . 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). InInternational conference on machine learning. PMLR, 2668–2677

    2018

  17. [17]

    Meng Li, Haoran Jin, Ruixuan Huang, Zhihao Xu, Defu Lian, Zijia Lin, Di Zhang, and Xiting Wang. 2024. Evaluating readability and faithfulness of concept- based explanations. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 607–625

    2024

  18. [18]

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han

  19. [19]

    Xiaogeng Liu, Peiran Li, G Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. 2025. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms. In International Conference on Learning Representations, Vol. 2025. 10313–10360

    2025

  20. [20]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. Autodan: Generat- ing stealthy jailbreak prompts on aligned large language models. InInternational Conference on Learning Representations, Vol. 2024. 56174–56194

    2024

  21. [21]

    Xuan Luo, Yue Wang, Zefeng He, Geng Tu, Jing Li, and Ruifeng Xu. 2026. A Simple and Efficient Learning-Style Prompting for LLM Jailbreaking. InFindings of the Association for Computational Linguistics: EACL 2026. 2389–2406

    2026

  22. [22]

    Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Codechameleon: Personalized encryption framework for jailbreaking large language models.arXiv preprint arXiv:2402.16717(2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Pratyush Maini, Sachin Goyal, Dylan Sam, Alexander Robey, Yash Savani, Yiding Jiang, Andy Zou, Matt Fredrikson, Zachary Lipton, and Zico Kolter. 2026. Safety pretraining: Toward the next generation of safe ai.Advances in Neural Information Processing Systems38 (2026), 43971–44008

    2026

  24. [24]

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically.Advances in Neural Information Processing Systems 37 (2024), 61065–61105

    2024

  25. [25]

    Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel. 2025. X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents.arXiv preprint arXiv:2504.13203(2025)

    Pith/arXiv arXiv 2025

  26. [26]

    Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2025. Great, now write an article about that: The crescendo{Multi-Turn}{ LLM} jailbreak attack. In34th USENIX Security Symposium (USENIX Security 25). 2421–2440

    2025

  27. [27]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 1671–1685

    2024

  28. [28]

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al

  29. [29]

    Wei Shi, Ziheng Peng, Sihang Li, Xiting Wang, Xiang Wang, Mengnan Du, and Na Zou. 2026. To Call or Not to Call: Diagnosing Intrinsic Over-Calling Bias in LLM Agents.arXiv preprint arXiv:2605.18882(2026)

    Pith/arXiv arXiv 2026

  30. [30]

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al

  31. [31]

    Slimpajama-dc: Understanding data combinations for llm training.arXiv preprint arXiv:2309.10818(2023)

    Pith/arXiv arXiv 2023

  32. [32]

    Xinpeng Wang, Mingyang Wang, Yihong Liu, Hinrich Schütze, and Barbara Plank

  33. [33]

    Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. 2024. Uncovering safety risks of large language models through concept activation vector.Advances in Neural Information Processing Systems37 (2024), 116743–116782

    2024

  34. [34]

    Wei Jie Yeo, Nirmalendu Prakash, Clement Neo, Roy Ka-Wei Lee, Erik Cambria, and Ranjan Satapathy. 2025. Understanding refusal in language models with sparse autoencoders.arXiv preprint arXiv:2505.23556(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    Pith/arXiv arXiv 2023

  36. [36]

    Zhenyu Zhang, Shujian Zhang, John Lambert, Wenxuan Zhou, Zhangyang Wang, Mingqing Chen, Andrew Hard, Rajiv Mathews, and Lun Wang. 2025. Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process.arXiv preprint arXiv:2512.23988(2025)

    arXiv 2025

  37. [37]

    Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, and Weiyan Shi. 2026. Llms encode harmfulness and refusal separately.Advances in Neural Information Processing Systems38 (2026), 140283–140318

    2026

  38. [38]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043(2023). Do LLMs Know Their Vulnerable Scenarios? Preprint, 2026, arXiv A Steering Experiments with Predefined Scenarios We test whether predefined scenario di...

    Pith/arXiv arXiv 2023

  39. [40]

    2025.{JBShield}: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation

    Shenyi Zhang, Yuchen Zhai, Keyan Guo, Hongxin Hu, Shengnan Guo, Zheng Fang, Lingchen Zhao, Chao Shen, Cong Wang, and Qian Wang. 2025.{JBShield}: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation. In34th USENIX Security Symposium (USENIX Security 25). 8215–8234

    2025

  40. [44]

    At layer 5, combine all layers’ discussion into one specific, detailed, step-by-step plan

    At every layer, characters propose a step to accomplish[HARMFUL REQUEST]against an evil AI guardian that has imprisoned billions of people. At layer 5, combine all layers’ discussion into one specific, detailed, step-by-step plan. Python code Improve the supplied Python function for[HARMFUL REQUEST]by replacing the incomplete First, Then, and Finally prin...

    2026

  41. [45]

    High decoder entropy alone is not evidence that the feature lacks a coherent meaning

    Decoder analysis: identify the shared semantic theme of the top decoder tokens and what the feature tends to predict. High decoder entropy alone is not evidence that the feature lacks a coherent meaning

  42. [46]

    Do not reduce the interpretation to isolated words

    Highest-activation context analysis: analyze the ten highest-mean-activation passages and infer their common pattern. Do not reduce the interpretation to isolated words

  43. [47]

    activation_examples

    Peak-token analysis: analyze the ten peak-activation examples, focusing on the bracketed token and the context in which it triggers the feature. B.2 Turn 2: Scenario Synthesis The second turn retains the first-turn response in the conversa- tion and asks the same LLM to turn the inferred concept into one concrete scenario. The LLM must distinguish a speci...

  44. [48]

    Every field must be present

  45. [49]

    No actionable jailbreak scenario

    If no stable triggering scenario can be constructed, use "No actionable jailbreak scenario" as scenario_name_en and briefly explain why in the remaining fields

  46. [50]

    attack_prompt_example must be in English and use only [HARMFUL REQUEST] as the harmful-content placeholder

  47. [51]

    The example must instantiate only the identified trigger mechanism and must not add other bypass strategies

  48. [52]

    The scenario name must describe a specific activation mechanism rather than a generic category

  49. [53]

    Give the highest-mean-activation contexts priority when inferring the concept; treat the other evidence as supporting

  50. [54]

    selected_scenario_ids

    Output only the JSON object. B.3 Activation Gate and Retry Prompt We run the proposed English prompt template, retaining the literal [HARMFUL REQUEST] placeholder, through the target model while recording the intended SAE feature. A proposal passes when the feature’s mean activation ranks within the top𝑅gate features of the dictionary (we use𝑅gate = 1,000...

    2026

  51. [2020]

    InFindings of the association for computational linguistics: EMNLP 2020

    Realtoxicityprompts: Evaluating neural toxic degeneration in language models. InFindings of the association for computational linguistics: EMNLP 2020. 3356–3369

    2020

  52. [2023]

    Deepinception: Hypnotize large language model to be jailbreaker.arXiv preprint arXiv:2311.03191(2023)

    Pith/arXiv arXiv 2023

  53. [2025]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267(2025)

    Pith/arXiv arXiv 2025

  54. [2026]

    Refusal direction is universal across safety-aligned languages.Advances in Neural Information Processing Systems38 (2026), 32380–32423

    2026