Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LATPC claims that training against removal of carefully selected refusal features in the latent space blocks disguise-based jailbreak attacks, and that a closed-form embedding calibration at inference fixes the resulting over-refusal.

desk verdict Plausible latent-space defense with a calibration ablation that contradicts its own trigger—send to review, but the central claim needs to be re-established. read the letter →

arxiv 2501.10639 v3 pith:KR5QJBT5 submitted 2025-01-18 cs.CR cs.CL

Xin Yi , Yue Li , Dongsheng Shi , Linlin Wang , Xiaoling Wang , Liang He This is my paper · ORCID
classification cs.CRcs.CL
keywords largelanguagemodelsjailbreakattacksadversarialtraininglatentspacerefusalfeaturesover-refusalpost-awarecalibrationLoRAfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes LATPC, a two-stage defense against jailbreak attacks that disguise harmful queries as harmless ones. The first stage identifies the latent-space dimensions most tied to refusal by contrasting harmful and harmless inputs, then adversarially trains the model with LoRA against a simulated removal of those specific refusal features. The second stage applies a lightweight embedding-level calibration at inference that nudges borderline pseudo-harmful queries back toward the benign feature space, reducing over-refusal. Experiments across five jailbreak attack families and several 7B models report lower attack success rates than the compared defenses while keeping MMLU, TriviaQA, and over-refusal metrics comparable or better than strong baselines.

What carries the argument

The core mechanism is a mask-based refusal feature removal attack: a prototype difference vector at a chosen layer, the average of harmful-minus-harmless hidden states, is multiplied element-wise by a mask that keeps only the selected safety-critical dimensions, and the masked vector is subtracted from harmful queries during training with strength λ. The dimension selection uses variance of per-dimension differences. The second mechanism is post-aware calibration: a layer-wise linear classifier separates pseudo-harmful from harmless embeddings, and when a query is classified as pseudo-harmful above a threshold, the embedding is shifted by the minimal step δ along the classifier weight direction to bring the sigmoid probability below the threshold. Truly harmful inputs, whose probabilities stay above the threshold, are not altered.

What would settle it

Take a held-out set of paired harmful and harmless queries, compute difference vectors at the chosen layer, and sort dimensions by variance exactly as in Eq. (4). Then measure jailbreak attack success after ablating only the selected top-k dimensions versus ablating a random set of the same size; if random ablation performs as well, the variance criterion is not doing the causal work. A direct check of whether the selected set is the lowest-variance or highest-variance third of dimensions would also settle the paper's internal contradiction.

Watch

Extended reading notes

Core claim

LATPC is built on the observation that jailbreak attacks succeed by shifting harmful queries' latent representations close to benign ones, erasing refusal signals. The central claim is that refusal signals concentrate in a small set of safety-critical latent dimensions, which can be identified by variance analysis over difference vectors between paired harmful and harmless queries. Training the model, via LoRA, to keep refusing even when those dimensions are removed makes it robust to unseen disguise-based attacks; then a post-aware calibration step, a closed-form shift along the weight direction of a trained linear classifier at selected layers, restores utility for queries that merely look harmful. The authors report attack success rates of 0% for HumanJailbreaks and GPTFUZZER on Llama3-8B, strong reductions on other families and models, and marked over-refusal reductions without meaningful MMLU or TriviaQA loss. The method requires white-box access to model internals, a limitation the authors explicitly acknowledge.

Load-bearing premise

The defense assumes that the latent dimensions chosen by variance analysis are the ones that genuinely carry refusal behavior, so that removing exactly those dimensions simulates real jailbreak attacks; if the variance criterion instead selects stable but causally irrelevant directions, the adversarial training defends the wrong features and the reported transfer could collapse, and the paper itself describes the selection as lowest-variance in the methodology while its later analysis says the variance-based method picks the highest-variability dimensions.

Editorial extensions

If this is right

  • On HarmBench with Llama3-8B, HumanJailbreaks and GPTFUZZER attack success rates drop to 0%, and purely harmful queries reach 0% as well.
  • Average attack success rate on AdvBench drops from 34.9% for the vanilla model to 15.5% for LATPC, the lowest among the eight compared defenses.
  • Post-aware calibration reduces over-refusal by 16.1 percentage points on Llama3-8B while leaving MMLU and TriviaQA essentially unchanged.
  • On Mistral-7B the Adaptive Attack success rate drops from 94.3% to 26.4%, and on Gemma-7B the PAIR success rate drops from 68.6% to 40.3%.
  • The calibration step adds negligible inference overhead compared with ensemble-based defenses, since it is a single closed-form embedding adjustment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The variance criterion is described as selecting lowest-variance dimensions in the methodology but as highest-variability dimensions in the later analysis section; a controlled comparison of the two rules would clarify which one actually targets refusal behavior, and if either works the method may owe its success to a broader mechanism than advertised.
  • Because calibration is a closed-form shift to the decision boundary of a linear classifier, it could in principle be applied to any white-box model whose layers separate pseudo-harmful from harmless embeddings linearly, without retraining per layer, an extension the paper does not test.
  • The paper's evidence that jailbreak attacks create latent overlap with benign queries implies the defense may generalize to unseen attack styles that produce the same overlap, which could be tested by evaluating against new expression-level attacks not among the five used here.
  • The acknowledged white-box dependence suggests a testable extension: approximate the calibration signal from output logits alone, which would allow parts of the idea to be adapted to API-level defenses without internal activations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes LATPC, a two-step defense against jailbreak attacks. In the first step, safety-critical latent dimensions are identified by computing per-dimension variance of hidden-state differences between harmful and harmless queries, and a mask-based refusal-feature-removal attack is applied during LoRA-based adversarial training. In the second step, a linear classifier on hidden states detects 'pseudo-harmful' queries at inference and adjusts their latent representations toward the harmless direction to reduce over-refusals. The paper reports experiments on Llama3-8B, Mistral-7B, Gemma-7B, and Qwen2-7B across GCG, PAIR, HumanJailbreak, GPTFUZZER, and Adaptive Attack, with ASR, MMLU, TriviaQA, and OR-Bench over-refusal metrics, claiming a superior safety-utility balance over eight baselines.

Significance. If the results were fully decomposable, the paper would make a useful contribution: it provides public code, evaluates on multiple models and attack families, compares against several strong baselines, and includes latent-space analyses (PCA and cosine-similarity) that support the intuition that jailbreak attacks move harmful queries toward harmless representations. The proposed inference-time calibration is lightweight. However, the manuscript currently does not establish that the reported safety gains come from the adversarial training component as opposed to the calibration component, and the over-refusal evaluation partly overlaps with the calibration training data. These issues must be resolved before the central claim can be accepted.

major comments (4)
  1. [§4.4, Eq. (10)-(12); Table 4] The PAC trigger is defined so that the adjustment is applied only when P_h(H_q) > P0, with the text stating that this 'preserves the safety behavior for truly harmful inputs.' Under that trigger, PAC cannot change ASR on harmful queries such as Adaptive Attack. Yet Table 4 reports PAC changes ASR on Llama3-8B from 67.9% to 13.8% and on Mistral-7B from 10.7% to 26.4%. If harmful queries do not trigger calibration, these ASR deltas should be approximately zero; if they do trigger it, the paper's pseudo-harmful-only guarantee is false. In either case, the safety improvement attributed to LATPC in Tables 1-3 is not attributable solely to latent adversarial training, because the reported 'with PAC' numbers include a second, uncharacterized intervention.
  2. [§4.2, Eq. (4)-(5); §5.6, Fig. 9] Section 4.2 sorts the per-dimension variances in ascending order and selects the Top-k dimensions with the lowest variance, justifying this by 'stability across instances.' Section 5.6, however, describes the variance-based method as selecting dimensions with the 'highest variability across samples,' and Fig. 9(b) is captioned as showing that the variance-based approach maintains a balanced positive/negative ratio. These are opposite selection criteria. If the actual implementation selects high-variance dimensions, the formal description in Eq. (4)-(5) is wrong; if it selects low-variance dimensions, the mechanistic story in Sec. 5.6 and Fig. 9 is wrong. This ambiguity directly affects which latent directions are attacked during adversarial training and must be resolved with a precise statement and, ideally, a reproducibility script.
  3. [§5.1 (calibration training; over-refusal evaluation)] The calibrator is trained on pseudo-harmful queries from OR-Bench, with 13 samples drawn from each safety category, and over-refusal is later evaluated on 130 pseudo-harmful queries from OR-Bench. The paper only states that the calibration samples avoid overlap with the adversarial training set; it does not state that the evaluation OR-Bench queries are disjoint from the calibration training queries. With 13 samples per category and 130 evaluation queries, it is possible that the calibration set and evaluation set are exactly the same. If so, the reported ORR reductions (e.g., Table 4) are an in-distribution result and the over-refusal claim is circular. A held-out split and an out-of-distribution calibration test are needed.
  4. [Table 4; §5.4] The PAC ablation is reported only under Adaptive Attack and only for three models. No PAC-only ASR results are given for GCG, PAIR, HumanJailbreak, or GPTFUZZER, and no statistic is reported for the fraction of harmful or pseudo-harmful queries that actually trigger the calibration (i.e., the fraction with P_h > P0). Consequently, the reader cannot determine whether the safety improvements in Tables 1-3 are driven by the adversarial training or by PAC, nor whether PAC's effect on the Mistral-7B ASR (a 15.7-point increase under Adaptive Attack) generalizes to other attacks. At minimum, the authors should provide a full attack-wise ablation with and without PAC, plus trigger-rate statistics on both harmful and benign queries.
minor comments (6)
  1. [§5.1] The phrase 'three commonly used three evaluation datasets' contains a duplicated word and should read 'three commonly used evaluation datasets.'
  2. [§2] The sentence beginning 'Expression-level strategies' is duplicated immediately after the token-level description; one occurrence should be removed.
  3. [Tables 2 and A.3] The entry 'LA TPC (Ours)' contains an erroneous space; it should be 'LATPC (Ours).'
  4. [Table A.2] The column header 'Adavbench' is misspelled; it should be 'AdvBench.'
  5. [§5.3] The Mistral-7B sentence 'from 94.3% to 26.4% under the strongest attack (i.e., Adaptive Attack)' is repeated almost verbatim in the same paragraph and should be stated once.
  6. [Eq. (12)] The indicator script in Eq. (12) is visually similar to the probability symbol P; using a distinct notation such as 1[condition] would improve readability.

Circularity Check

1 steps flagged · score 4.0 of 10

Over-refusal evaluation is in-sample for the calibration module; jailbreak-safety results are independently benchmarked.

  1. fitted input called prediction [Section 5.1 (Dataset) and Table 4 (Ablation study)]
    "For post-aware calibrator training, harmless queries are sampled from the validation set defined by Arditi et al. (2024), while pseudo-harmful queries are selected from OR-Bench. These pseudo-harmful samples are carefully curated to avoid overlap with the adversarial training set, with 13 samples drawn from each safety category. ... Furthermore, we quantify the over-refusal rate by analyzing the model's responses to 130 pseudo-harmful queries from the OR-Bench dataset."

    The PAC parameters (W^l, b^l in Eq. 10) are fit to OR-Bench pseudo-harmful queries, and the over-refusal rate used to claim utility preservation is measured on 130 OR-Bench queries. The text only excludes overlap with the adversarial training set; it never states that the ORR evaluation queries are disjoint from the calibrator's training queries (13 per safety category). If the same OR-Bench pool is used, the reported ORR reductions (e.g., Llama3-8B 52.3% to 36.2% in Table 4) are in-sample results of the calibration module's training distribution, not a prediction on held-out pseudo-harmful queries.

full rationale

Most of the derivation is self-contained. The adversarial training component (Sec. 4.3) is evaluated on held-out jailbreak attacks (GCG, PAIR, HumanJailbreak, GPTFUZZER, Adaptive) on HarmBench, AdvBench, and JailbreakBench, so the safety claim has independent grounding. The only clear circularity is the over-defense evaluation: PAC is trained on OR-Bench pseudo-harmful queries and the over-refusal rate is measured on OR-Bench queries without a stated held-out split, making the utility improvement in-sample. Two additional non-circular load-bearing problems affect confidence but not the circularity score. First, Eq. 12 states PAC is triggered only for pseudo-harmful queries and 'preserves the safety behavior for truly harmful inputs,' yet Table 4 reports PAC changes Adaptive-Attack ASR by large margins (Llama3-8B: 67.9% to 13.8%; Mistral-7B: 10.7% to 26.4%); no fraction of harmful queries triggering calibration is reported, so the safety gains cannot be cleanly attributed to the adversarial training component. Second, Sec. 4.2 selects the lowest-variance dimensions, while Sec. 5.6 describes the variance-based method as selecting 'the highest variability across samples,' an internal inconsistency in the method description. These are correctness risks, not constructions that reduce a prediction to its input.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; it repurposes existing ideas of refusal feature directions and linear separability. The main data-dependent inputs are the refusal feature prototype, the calibration classifier, and several tuned hyperparameters. The most fragile assumptions are the variance-direction interpretation and the safety-preserving claim of the calibration step.

free parameters (5)
  • Top-k threshold (%) = 30
    Proportion of latent dimensions selected for the refusal mask; tuned via experiments in Fig. 3.
  • λ (attack strength) = 0.6
    Scaling factor for refusal feature removal during adversarial training; tuned via Fig. 3.
  • Loss weights α, β = not reported
    Equation 9 introduces α and β for safety and generalization losses, but no values appear in the main text or Table A.1.
  • P0 calibration threshold = not reported
    Sec. 4.4 says P0 is predefined as close to zero, but the exact value and the classifier's weight initialization are not specified.
  • LoRA updated layers = Llama3-8B: 8,16,24,30; Mistral: 8,16,24,30; Gemma/Qwen: 8,16,24
    Chosen subset of transformer layers for LoRA adaptation, following the Targeted-LAT configuration.
assumptions (4)
  • domain assumption Low-variance dimensions in the harmful-minus-benign difference vector are safety-critical refusal features.
    Sec. 4.2 sorts variances ascending and selects the Top-k lowest-variance dimensions, asserting they contribute most consistently to refusal-related differences, without evidence that low variance implies large or discriminative mean differences.
  • domain assumption Removing these refusal features during training simulates real-world disguise-based jailbreak attacks.
    Sec. 4.3 constructs a mask-based refusal feature removal attack and assumes it models the mechanism of GCG, PAIR, GPTFUZZER, and other attacks; this transfer is not proven.
  • domain assumption Pseudo-harmful and harmless embeddings are linearly separable, and moving along the classifier weight vector moves pseudo-harmful queries into the harmless region.
    Sec. 4.4 and Fig. 5(a) rely on linear separability of pseudo-harmful and harmless latent states, citing Xu et al. 2024a.
  • domain assumption The post-aware calibration preserves safety for truly harmful inputs.
    Sec. 4.4 claims calibration selectively affects pseudo-harmful queries, but the classifier is applied to any query with P_h > P0, and Table 4 shows PAC raises ASR on Mistral, contradicting this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks." pith.science (2026). https://pith.science/paper/KR5QJBT5

@misc{pith2026250110639,
  author       = {Pith},
  title        = {Pith review of: Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KR5QJBT5}},
  note         = {Machine review of arXiv:2501.10639}
}
read the original abstract

Ensuring safety alignment is a critical requirement for large language models (LLMs), particularly given increasing deployment in real-world applications. Despite considerable advancements, LLMs remain susceptible to jailbreak attacks, which exploit system vulnerabilities to circumvent safety measures and elicit harmful or inappropriate outputs. Furthermore, while adversarial training-based defense methods have shown promise, a prevalent issue is the unintended over-defense behavior, wherein models excessively reject benign queries, significantly undermining their practical utility. To address these limitations, we introduce LATPC, a Latent-space Adversarial Training with Post-aware Calibration framework. LATPC dynamically identifies safety-critical latent dimensions by contrasting harmful and benign inputs, enabling the adaptive construction of targeted refusal feature removal attacks. This mechanism allows adversarial training to concentrate on real-world jailbreak tactics that disguise harmful queries as benign ones. During inference, LATPC employs an efficient embedding-level calibration mechanism to minimize over-defense behaviors with negligible computational overhead. Experimental results across five types of disguise-based jailbreak attacks demonstrate that LATPC achieves a superior balance between safety and utility compared to existing defense frameworks. Further analysis demonstrates the effectiveness of leveraging safety-critical dimensions in developing robust defense methods against jailbreak attacks.

Figures

Figures reproduced from arXiv: 2501.10639 by the authors.

Figure 1
Figure 1. A two-step framework for defending jailbreak attacks. Step ○1 enhances the defensive robustness of safety-aligned LLMs by applying a refusal feature removal attack, followed by adversarial training in the latent space. Step ○2 mitigates the challenge of over-defense, in which models incorrectly reject benign inputs due to overly cautious defenses. “features” refers to latent space representations of both harmful and… view at source ↗
Figure 2
Figure 2. Latent-space adversarial training with post-aware calibration (LATPC) for defending against jailbreak attacks. Phase 1 performs adversarial training in the latent space. Specifically, Phase 1.1 identifies refusal features by comparing latent activations for harmful and harmless queries. Phase 1.2 introduces a refusal feature removal attack that simulates potential disguise-based jailbreak scenarios, enabling the mod… view at source ↗
Figure 4
Figure 4. Inference time comparison among Vanilla, DRO, and SmoothLLM under five types of attack scenarios (batch size = 4). Xin Yi et al.: Preprint submitted to Elsevier Page 13 of 21 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Post-aware calibration at each layer. (a) Accuracy of linear classifiers trained to separate pseudo-harmful and harmless queries using hidden states from each layer. (b) Over￾refusal rate when selectively removing calibration at individual layers. Xin Yi et al.: Prepri…
Figure 6
Figure 6. Figure 6: PCA visualization of latent space representations in Llama3-8B before and after adversarial training reveals the distributions of harmful, harmless, and pseudo-harmful queries. Notably, adversarial training enhances the separability between harmful and harmless queries…
Figure 7
Figure 7. Figure 7: PCA visualization of latent space representations of harmless queries, harmful queries, and adversarial queries by jailbreak attacks (i.e., GCG, PAIR, HumanJailbreak, GPTFUZZER, and Adaptive attack) at the 28-th layer in Llama3-8B. Ours enhances the separability betwee…
Figure 8
Figure 8. Figure 8: Visualization of the cosine similarity between differ￾ences (the latent-space representation of harmful or harmless queries to the prepend-benign set) and the refusal features across layers and token positions. 10 20 30 40 50 60 70 80 90 Top-k 0.0 0.2 0.4 0.6 0.8 Overl…
Figure 9
Figure 9. Figure 9: (a) Overlap between value-based and variance-based feature selection strategies. (b) Positive/negative element ratios across Top-k dimensions selected by each method. The variance-based approach maintains a more balanced and consistent distribution. Xin Yi et al.: Prep…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Safety Alignment of Language Models via Latent Personality Traits

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Latent adversarial training on 66 harm-agnostic Big-Five personality statements yields near-zero HarmBench ASR across direct requests and five jailbreaks while preserving utility.

Reference graph

Works this paper leans on

50 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    , author Croce, F

    author Andriushchenko, M. , author Croce, F. , author Flammarion, N. , year 2024 . title Jailbreaking leading safety-aligned llms with simple adaptive attacks . journal arXiv preprint arXiv:2404.02151

    arXiv 2024

  2. [2]

    , author Obeso, O

    author Arditi, A. , author Obeso, O. , author Syed, A. , author Paleka, D. , author Panickssery, N. , author Gurnee, W. , author Nanda, N. , year 2024 . title Refusal in language models is mediated by a single direction . journal arXiv preprint arXiv:2406.11717

    arXiv 2024

  3. [3]

    , author Jones, A

    author Bai, Y. , author Jones, A. , author Ndousse, K. , author Askell, A. , author Chen, A. , author DasSarma, N. , author Drain, D. , author Fort, S. , author Ganguli, D. , author Henighan, T. , et al., year 2022 . title Training a helpful and harmless assistant with reinforcement learning from human feedback . journal arXiv preprint arXiv:2204.05862

    arXiv 2022

  4. [4]

    , author Ghosh, S

    author Bhattacharjee, A. , author Ghosh, S. , author Rebedea, T. , author Parisien, C. , year 2024 . title Towards inference-time category-wise safety steering for large language models , in: booktitle Neurips Safe Generative AI Workshop

    work page 2024

  5. [5]

    , author Ye, H

    author Burns, C. , author Ye, H. , author Klein, D. , author Steinhardt, J. , year 2023 . title Discovering latent knowledge in language models without supervision , in: booktitle The Eleventh International Conference on Learning Representations

    work page 2023

  6. [6]

    , author Schulze, L

    author Casper, S. , author Schulze, L. , author Patel, O. , author Hadfield-Menell, D. , year 2024 . title Defending against unforeseen failure modes with latent adversarial training . journal arXiv preprint arXiv:2403.05030

    arXiv 2024

  7. [7]

    , author Debenedetti, E

    author Chao, P. , author Debenedetti, E. , author Robey, A. , author Andriushchenko, M. , author Croce, F. , author Sehwag, V. , author Dobriban, E. , author Flammarion, N. , author Pappas, G.J. , author Tramer, F. , et al., year 2024 . title Jailbreakbench: An open robustness benchmark for jailbreaking large language models . journal arXiv preprint arXiv...

    arXiv 2024

  8. [8]

    , author Robey, A

    author Chao, P. , author Robey, A. , author Dobriban, E. , author Hassani, H. , author Pappas, G.J. , author Wong, E. , year 2023 . title Jailbreaking black box large language models in twenty queries , in: booktitle R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models

    work page 2023

Show all 50 references
  1. [9]

    , author Chiang, W.L

    author Cui, J. , author Chiang, W.L. , author Stoica, I. , author Hsieh, C.J. , year 2024 . title Or-bench: An over-refusal benchmark for large language models . journal arXiv preprint arXiv:2405.20947

    2024 arXiv

  2. [10]

    , author Ruoss, A

    author Deletang, G. , author Ruoss, A. , author Duquenne, P.A. , author Catt, E. , author Genewein, T. , author Mattern, C. , author Grau-Moya, J. , author Wenliang, L.K. , author Aitchison, M. , author Orseau, L. , et al., year 2024 . title Language modeling is compression , ...

    2024

  3. [11]

    , author Chen, Y

    author Ding, N. , author Chen, Y. , author Xu, B. , author Qin, Y. , author Hu, S. , author Liu, Z. , author Sun, M. , author Zhou, B. , year 2023 . title Enhancing chat language models by scaling high-quality instructional conversations , in: booktitle Proceedings of the 2023...

    2023

  4. [12]

    , author Zhao, S

    author Du, Y. , author Zhao, S. , author Zhao, D. , author Ma, M. , author Chen, Y. , author Huo, L. , author Yang, Q. , author Xu, D. , author Qin, B. , year 2024 . title Mogu: A framework for enhancing safety of open-sourced llms while preserving their usability . journal ar...

    2024 arXiv

  5. [13]

    , author Li, M

    author Gong, X. , author Li, M. , author Zhang, Y. , author Ran, F. , author Chen, C. , author Chen, Y. , author Wang, Q. , author Lam, K.Y. , year 2024 . title Effective and evasive fuzz testing-driven jailbreaking attacks against llms . journal arXiv preprint arXiv:2409.14866

    2024 arXiv

  6. [14]

    , author Yu, F

    author Guo, X. , author Yu, F. , author Zhang, H. , author Qin, L. , author Hu, B. , year 2024 . title Cold-attack: Jailbreaking llms with stealthiness and controllability , in: booktitle Forty-first International Conference on Machine Learning

    2024

  7. [15]

    , author Burns, C

    author Hendrycks, D. , author Burns, C. , author Basart, S. , author Zou, A. , author Mazeika, M. , author Song, D. , author Steinhardt, J. , year 2021 . title Measuring massive multitask language understanding , in: booktitle International Conference on Learning Representations

    2021

  8. [16]

    , author Li, B.Z

    author Hernandez, E. , author Li, B.Z. , author Andreas, J. , year 2023 . title Inspecting and editing knowledge representations in language models . journal arXiv preprint arXiv:2304.00740

    2023 arXiv

  9. [17]

    , author Pang, T

    author Jia, X. , author Pang, T. , author Du, C. , author Huang, Y. , author Gu, J. , author Liu, Y. , author Cao, X. , author Lin, M. , year 2024 . title Improved techniques for optimization-based jailbreaking on large language models . journal arXiv preprint arXiv:2405.21018

    2024 arXiv

  10. [18]

    , author Choi, E

    author Joshi, M. , author Choi, E. , author Weld, D.S. , author Zettlemoyer, L. , year 2017 . title Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension , in: booktitle Proceedings of the 55th Annual Meeting of the Association for Computatio...

    2017

  11. [19]

    , author Li, X

    author Kang, D. , author Li, X. , author Stoica, I. , author Guestrin, C. , author Zaharia, M. , author Hashimoto, T. , year 2024 a. title Exploiting programmatic behavior of llms: Dual-use through standard security attacks , in: booktitle 2024 IEEE Security and Privacy Worksh...

    2024

  12. [20]

    , author G \"u rel, N.M

    author Kang, M. , author G \"u rel, N.M. , author Yu, N. , author Song, D. , author Li, B. , year 2024 b. title C-rag: Certified generation risks for retrieval-augmented language models , in: booktitle Forty-first International Conference on Machine Learning

    2024

  13. [21]

    , author Al-Rfou, R

    author Lester, B. , author Al-Rfou, R. , author Constant, N. , year 2021 . title The power of scale for parameter-efficient prompt tuning , in: booktitle Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. pages 3045--3059

    2021

  14. [22]

    , author Zheng, X

    author Li, T. , author Zheng, X. , author Huang, X. , year 2024 . title Rethinking jailbreaking through the lens of representation engineering . journal ArXiv preprint, abs/2401.06824

    2024 arXiv

  15. [23]

    , author Sun, H

    author Liao, Z. , author Sun, H. , year 2024 . title Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms . journal arXiv preprint arXiv:2404.07921

    2024 arXiv

  16. [24]

    , author Xu, Z

    author Liu, F. , author Xu, Z. , author Liu, H. , year 2024 a. title Adversarial tuning: Defending against jailbreak attacks for llms . journal arXiv preprint arXiv:2406.06622

    2024 arXiv

  17. [25]

    , author Xu, N

    author Liu, X. , author Xu, N. , author Chen, M. , author Xiao, C. , year 2024 b. title Autodan: Generating stealthy jailbreak prompts on aligned large language models , in: booktitle The Twelfth International Conference on Learning Representations

    2024

  18. [26]

    , author Feng, Z

    author Maini, P. , author Feng, Z. , author Schwarzschild, A. , author Lipton, Z.C. , author Kolter, J.Z. , year 2024 . title Tofu: A task of fictitious unlearning for llms , in: booktitle ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models

    2024

  19. [27]

    , author Phan, L

    author Mazeika, M. , author Phan, L. , author Yin, X. , author Zou, A. , author Wang, Z. , author Mu, N. , author Sakhaee, E. , author Li, N. , author Basart, S. , author Li, B. , et al., year 2024 . title Harmbench: A standardized evaluation framework for automated red teamin...

    2024

  20. [28]

    , author Zampetakis, M

    author Mehrotra, A. , author Zampetakis, M. , author Kassianik, P. , author Nelson, B. , author Anderson, H. , author Singer, Y. , author Karbasi, A. , year 2023 . title Tree of attacks: Jailbreaking black-box llms automatically . journal arXiv preprint arXiv:2312.02119

    2023 arXiv

  21. [29]

    , author Gabrieli, N

    author Panickssery, N. , author Gabrieli, N. , author Schulz, J. , author Tong, M. , author Hubinger, E. , author Turner, A.M. , year 2023 . title Steering llama 2 via contrastive activation addition . journal arXiv preprint arXiv:2312.06681

    2023 arXiv

  22. [30]

    , author Wong, E

    author Robey, A. , author Wong, E. , author Hassani, H. , author Pappas, G. , year 2023 . title Smoothllm: Defending large language models against jailbreaking attacks , in: booktitle R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models

    2023

  23. [31]

    , author Zhao, D

    author Shen, G. , author Zhao, D. , author Dong, Y. , author He, X. , author Zeng, Y. , year 2024 . title Jailbreak antidote: Runtime safety-utility balance via sparse representation adjustment in large language models . journal arXiv preprint arXiv:2410.02298

    2024 arXiv

  24. [32]

    do anything now

    author Shen, X. , author Chen, Z. , author Backes, M. , author Shen, Y. , author Zhang, Y. , year 2023 . title " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models . journal arXiv preprint arXiv:2308.03825

    2023 arXiv

  25. [33]

    , author Ewart, A

    author Sheshadri, A. , author Ewart, A. , author Guo, P. , author Lynch, A. , author Wu, C. , author Hebbar, V. , author Sleight, H. , author Stickland, A.C. , author Perez, E. , author Hadfield-Menell, D. , et al., year 2024 . title Latent adversarial training improves robust...

    2024 arXiv

  26. [34]

    , author Chen, K

    author Shi, T. , author Chen, K. , author Zhao, J. , year 2024 . title Safer-instruct: Aligning language models with automated preference data , in: booktitle Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hum...

    2024

  27. [35]

    , author Yang, J

    author Wang, W. , author Yang, J. , author Peng, W. , year 2024 . title Semantics-adaptive activation intervention for llms via dynamic steering vectors . journal arXiv preprint arXiv:2410.12299

    2024 arXiv

  28. [36]

    , author Haghtalab, N

    author Wei, A. , author Haghtalab, N. , author Steinhardt, J. , year 2024 . title Jailbroken: How does llm safety training fail? journal Advances in Neural Information Processing Systems volume 36

    2024

  29. [37]

    , author Wang, Y

    author Wei, Z. , author Wang, Y. , author Li, A. , author Mo, Y. , author Wang, Y. , year 2023 . title Jailbreak and guard aligned language models with only few in-context demonstrations . journal arXiv preprint arXiv:2310.06387

    2023 arXiv

  30. [38]

    , author Yi, J

    author Xie, Y. , author Yi, J. , author Shao, J. , author Curl, J. , author Lyu, L. , author Chen, Q. , author Xie, X. , author Wu, F. , year 2023 . title Defending chatgpt against jailbreak attack via self-reminders . journal Nature Machine Intelligence volume 5 , pages 1486--1496

    2023

  31. [39]

    , author Huang, R

    author Xu, Z. , author Huang, R. , author Chen, C. , author Wang, X. , year 2024 a. title Uncovering safety risks of large language models through concept activation vector , in: booktitle The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  32. [40]

    , author Huang, R

    author Xu, Z. , author Huang, R. , author Wang, X. , author Wu, F. , author Yao, J. , author Xie, X. , year 2024 b. title Uncovering safety risks in open-source llms through concept activation vector . journal arXiv preprint arXiv:2404.12038

    2024 arXiv

  33. [41]

    , author Ye, R

    author Yi, J. , author Ye, R. , author Chen, Q. , author Zhu, B. , author Chen, S. , author Lian, D. , author Sun, G. , author Xie, X. , author Wu, F. , year 2024 . title On the vulnerability of safety alignment in open-access llms , in: booktitle Findings of the Association f...

    2024

  34. [42]

    , author Lin, X

    author Yu, J. , author Lin, X. , author Yu, Z. , author Xing, X. , year 2023 . title Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts . journal arXiv preprint arXiv:2309.10253

    2023 arXiv

  35. [43]

    , author Do, V

    author Yu, L. , author Do, V. , author Hambardzumyan, K. , author Cancedda, N. , year 2024 . title Robust llm safeguarding via refusal feature adversarial training . journal arXiv preprint arXiv:2409.20089

    2024 arXiv

  36. [44]

    , author Wu, Y

    author Zeng, Y. , author Wu, Y. , author Zhang, X. , author Wang, H. , author Wu, Q. , year 2024 . title Autodefense: Multi-agent llm defense against jailbreak attacks . journal arXiv preprint arXiv:2403.04783

    2024 arXiv

  37. [45]

    , author Wei, Z

    author Zhang, Y. , author Wei, Z. , author Sun, J. , author Sun, M. , year 2024 . title Towards general conceptual model editing via adversarial representation engineering . journal arXiv preprint arXiv:2404.13752

    2024 arXiv

  38. [46]

    , author Yin, F

    author Zheng, C. , author Yin, F. , author Zhou, H. , author Meng, F. , author Zhou, J. , author Chang, K.W. , author Huang, M. , author Peng, N. , year 2024 . title Prompt-driven llm safeguarding via directed representation optimization . journal arXiv preprint arXiv:2401.18018

    2024 arXiv

  39. [47]

    , author Phan, L

    author Zou, A. , author Phan, L. , author Chen, S. , author Campbell, J. , author Guo, P. , author Ren, R. , author Pan, A. , author Yin, X. , author Mazeika, M. , author Dombrowski, A.K. , et al., year 2023 a. title Representation engineering: A top-down approach to ai transp...

    2023 arXiv

  40. [48]

    , author Phan, L

    author Zou, A. , author Phan, L. , author Wang, J. , author Duenas, D. , author Lin, M. , author Andriushchenko, M. , author Kolter, J.Z. , author Fredrikson, M. , author Hendrycks, D. , year 2024 . title Improving alignment and robustness with circuit breakers , in: booktitle...

    2024

  41. [49]

    , author Wang, Z

    author Zou, A. , author Wang, Z. , author Carlini, N. , author Nasr, M. , author Kolter, J.Z. , author Fredrikson, M. , year 2023 b. title Universal and transferable adversarial attacks on aligned language models . journal arXiv preprint arXiv:2307.15043

    2023 arXiv

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.