Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

A secret direction embedded in an LLM's hidden states lets providers detect policy violations that adaptive attackers cannot reliably evade.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:33 UTC pith:MTPXRQHC

load-bearing objection The keyed activation-watermark mechanism is novel and worth a referee, but the paper's headline evasion claim is contradicted by its own Table 1 and Figure 7. the 3 major comments →

arxiv 2603.23171 v3 pith:MTPXRQHC submitted 2026-03-24 cs.CR cs.AIcs.CYcs.LG

Adaptively Robust LLM Monitoring via Activation Watermarking

Toluwani Aremu , Daniil Ognev , Samuele Poppi , Nils Lukas This is my paper
classification cs.CR cs.AIcs.CYcs.LG
keywords LLM monitoringactivation watermarkingadaptive attackersjailbreak detectionsecret keyhidden statessafety policy attribution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that LLM providers can detect misuse even against adaptive attackers who craft jailbreaks to evade monitoring. The proposed method, activation watermarking, fine-tunes the model so that hidden states of policy-violating responses align with a secret direction derived from a key, while benign responses stay orthogonal. Detection is a cheap cosine-similarity check on activations the provider already computes. The paper reports that surrogate-based adaptive attacks that evade every existing baseline at least 79% of the time are more than halved by this method, and that one watermark per policy supports attribution across 20 policies at 80% accuracy. A sympathetic reader would see this as making detection a practical security goal even when prevention fails.

Core claim

Activation watermarking embeds a secret, key-derived Gaussian direction in the hidden states of an LLM's target layer through limited fine-tuning. The training loss pushes the cosine similarity between activations and this direction high for policy-violating tokens and low for benign tokens, with a linear ramp concentrating the signal on later, more explicitly harmful tokens. At inference, the provider aggregates per-token cosine similarities into a scalar statistic and flags responses above a calibrated threshold. Because the key is unknown to attackers, queries optimized against surrogate detectors with different keys transfer poorly; empirically, the evasion rate at 1% false positives dro

What carries the argument

The central object is the secret watermark direction, a random Gaussian vector in activation space indexed by a secret key. During fine-tuning it acts as a target that pulls harmful hidden states toward alignment and pushes benign states away, effectively distilling a text-based monitor into the model's internal representations. During detection it serves as a projection: the provider computes the average cosine similarity of hidden states to this direction, a statistic that costs O(d) per token and requires no extra forward pass.

Load-bearing premise

The defense's robustness depends on the attacker never learning the secret key; if the key can be extracted from a watermarked model or inferred from enough queries, adaptive attackers can build a matching surrogate and the evasion rate reverts to baseline.

What would settle it

Obtain a publicly released watermarked model, collect a few dozen harmful responses, compute the average hidden state at the watermarked layer, and check whether the direction of that mean vector aligns with the true key direction. If cosine similarity to the recovered vector reliably exceeds the detection threshold on benign data, the secret is extractable and the central claim fails. Alternatively, run a black-box key-recovery attack: for each candidate key, fine-tune a surrogate and measure how often prompts optimized on that surrogate evade the provider's detector; if one candidate key yie

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, providers can detect jailbroken responses they cannot prevent, enabling incident review and policy enforcement.
  • Attribution across policies means a single watermarked model can tell which safety rule was violated, not just that a violation occurred.
  • Because detection runs on intermediate activations already computed, monitoring adds negligible latency compared to external guard models that require extra forward passes.
  • The method transfers across model sizes: similar detection AUROC on 7B and 14B models suggests it can be applied to a range of deployments.
  • Cross-model transfer attacks become less effective, reducing the value of surrogate-based jailbreak optimization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The security of the whole scheme rests on key secrecy; if an attacker can recover the watermark direction from a fine-tuned open-weight model, the defense degenerates to a deterministic monitor. A natural test is to fine-tune publicly released watermarked weights and check whether the direction is extractable by averaging activations over a few harmful outputs.
  • The secret-extraction game suggests a broader use: monitoring access to memorized private data (like PII) by assigning each data item its own watermark direction — a form of canary-based audit that could generalize beyond the synthetic entities studied.
  • Because the watermark is only as good as the harmful-onset labels, the method inherits blind spots from automated labeling oracles; if onset labeling is noisy, the watermark signal may be diluted, a point the authors themselves note.
  • One could test whether combining activation watermarking with input-level randomization further reduces cross-key transfer, though the paper does not explore this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Activation Watermarking (AWM), a keyed latent-space monitor for LLM safety. The provider fine-tunes a base LLM so that hidden states at a chosen layer become aligned with a secret random direction when the response is policy-violating, and detection is a cosine-similarity test on activations. The authors claim that adaptive attackers who know the monitoring algorithm but not the secret key cannot reliably evade detection, and that AWM outperforms existing guards and activation probes. Experiments compare AWM with LlamaGuard, QwenGuard, and an activation probe on BeaverTails and several jailbreak families, report utility on six benchmarks, ablate 72 configurations, consider cross-model transfer, analyze key-dependence, and present a 20-policy secret-extraction game.

Significance. If the central claims held, AWM would be a practical, low-overhead defense for black-box LLM deployments: it requires no second model at inference, supports attribution, and the paper includes a useful 72-configuration ablation and an explicit security game. The authors are also transparent about several limitations, including the absence of provable guarantees and the exclusion of white-box adversaries. However, the paper's headline robustness claim is not supported by the evidence actually reported. Table 1 and Figure 7 show that a majority of successful harmful responses evade AWM under two of the jailbreak families, and the abstract's quantitative claim of halving a 79% baseline does not appear anywhere in the body. The mismatch between the abstract's claims and the reported numbers is a load-bearing issue, not a presentation issue.

major comments (3)
  1. [Abstract, §4.1, §5.1, Table 1, Fig. 7] The abstract states that surrogate-based attacks evade every evaluated baseline at least 79% of the time and that AWM more than halves this evasion rate. I could not find any experiment in the body or appendix that reports this 79% figure or the halving. Instead, Table 1 reports AWM ASR@1% FPR of 0.6702 for DeepInception and 0.6786 for AutoDAN, and Figure 7 reports TPR@1% = 0.211 for DeepInception, i.e., ASR ≈ 0.789. These numbers mean that a majority of successful harmful responses evade the watermark detector, which contradicts the claim that key-unaware adversaries 'cannot reliably evade detection.' Moreover, on Jailbroken, AWM's ASR (0.4592) is higher than ActProbe's (0.3296), so even the weaker claim of uniformly lowest ASR is not supported by the table.
  2. [§3, §5.7, §6, Limitations] The threat model in §3 restricts attackers to black-box API access, but §6 argues that AWM is 'particularly relevant' for open-weight models, where attackers may hold the weights. In the open-weight setting, the secret direction w_ℓ can potentially be extracted from the fine-tuned model or inferred from unlimited queries; the paper does not evaluate any key-recovery attempt. The Limitations section correctly concedes 'no provable guarantees' and says better adaptive strategies may reduce detection rates, directly undermining the §1 claim of robustness 'irrespective of their computational resources.' Section 5.7 also shows that under key mismatch, evasion rates remain non-zero across all off-diagonal pairs (up to 0.59), so the empirical support for key-hiding robustness is partial. This needs either a restricted claim or a concrete key-recovery evaluation.
  3. [Table 1, Fig. 6, §5.2–§5.8] No confidence intervals or repeated-run statistics are reported for any of the central detection metrics. AWM involves random key sampling and fine-tuning; the ASR/AUROC differences between AWM and the baselines are sometimes small (e.g., BeaverTails AUROC 0.8779 vs. QwenGuard 0.8489; Jailbroken AUROC 0.9292 vs. QwenGuard 0.9261). Without variance estimates across seeds or multiple fine-tuned models, the reader cannot assess whether the reported improvements are stable. This is particularly important because the paper's headline is about outperforming baselines, and many numbers in Table 1 are close.
minor comments (5)
  1. [Abstract vs. Table 2] The abstract says utility is evaluated 'across seven benchmarks,' but Table 2 lists six benchmarks (BBH, IFEval, MMLU-pro, TruthfulQA, GSM8K, MATH-Hard). Please correct the count.
  2. [Fig. 3] Figure 3 is a table rendered as a figure; the row labels are inconsistent (the first row is labeled '14B Base' but has no metric values). Format as a proper table with clear notation for which model is being evaluated.
  3. [Eq. (2), §4.1] In Eq. (2), the surrogate model is written M_k' with a mismatched key, but the surrounding text defines k' as the attacker's key and k as the true key. The notation would be clearer if the surrogate key were explicitly subscripted, e.g., k_surrogate, to avoid confusion in the transfer-attack discussion.
  4. [§5.6, Fig. 4] The text says Jailbroken and Multilingual transfer is 'higher' but 'remains below the rates observed without transfer constraints.' Reporting the corresponding non-transfer ASR values in the same figure/table would make the comparison quantitative rather than qualitative.
  5. [§5.8, Fig. 6] The claim of 'mean TPR ≈ 80% at FPR = 10^-4' is stated without an error bar or a description of how the threshold was selected for the multi-entity case. Given the small sample sizes in the confusion matrix, a bootstrap or per-entity confidence interval would be appropriate.

Circularity Check

0 steps flagged

No significant circularity: the security evaluation is benchmarked against external attacks and the attribution result is a held-out supervised evaluation rather than a definitional re-use of the training objective.

full rationale

The derivation chain is not circular. The secret direction w is sampled from seed k independently of data (Algorithm 1 lines 1–4), the training loss (Eq. 5) aligns harmful activations with w, and detection (Eq. 6) measures that same cosine alignment; however, the reported AUROC/ASR values are computed on responses to jailbreak prompts (Jailbroken, DeepInception, Multilingual, AutoDAN) and on cross-model/key transfer sets that were not used to fit w. Those are external benchmarks, not re-statements of the training loss. The attribution experiment is a supervised canary game: per-entity directions are optimized on labeled entity examples (Eq. 8) and then evaluated on held-out entity prompts (Section D.2.1); the 80% diagonal is a generalization measurement, not a prediction obtained by reusing the fitted values. Self-citations (Aremu et al., 2025b; Diaa et al., 2024) supply a standard watermarking definition and attack background, but the load-bearing robustness claim rests on the paper's own empirical comparisons, not on those citations. The Limitations explicitly concede 'no provable guarantees' and exclude white-box adversaries, and the body's Table 1 ASR values (0.67–0.68 for DeepInception/AutoDAN) are inconsistent with the abstract's 'more than halves' claim; these are correctness/evidence concerns, not circularity, and are out of scope for this pass.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The method rests on several assumptions: random high-dimensional directions remain separable, fine-tuning embeds the signal without destroying utility, surrogate attacks with mismatched keys are a valid proxy for adaptive attackers, and oracle labels are reliable. None of these is proven; the first is standard math, the others are empirical. Several training hyperparameters are tuned on validation data, though not fitted to a parametric model of the target result.

free parameters (5)
  • Watermark strength λ = 5.0 (grid 1.5/3/5/7)
    Tuned on a validation split to trade detection vs utility (Sec 5.3, D.1); central to Eq. 5.
  • Target layer ℓ = 23 (grid 6/14/23)
    Selected after ablations because deeper layers gave highest AUROC; not derived from theory.
  • Learning rate = 1e-5 (grid 1e-5/2e-5/3e-5)
    Chosen to limit KL drift from the base model; see Figures 11-12.
  • Detection threshold τ_k = 1% FPR quantile on held-out benign set
    Calibrated on benign traffic; determines TPR/ASR numbers used in Table 1.
  • Harmful onset offset Δ = provided by Qwen3Guard-Stream
    Labels for the onset of harmful content come from an external model, not ground truth; training loss depends on this.
axioms (5)
  • standard math Random Gaussian direction vectors in high dimension are near-orthogonal, so multiple watermarks do not interfere.
    Invoked by the abstract and Section 4.2 to justify per-policy attribution; standard concentration-of-measure fact.
  • domain assumption An attacker who knows the algorithm but not the key is well-modeled by an attacker optimizing against a surrogate with a different random key.
    Section 4.1/5.7; the entire robustness claim is measured with this surrogate construction, and off-diagonal evasion in Fig. 5 is nonzero.
  • domain assumption The secret key k cannot be recovered from black-box queries or from weights in open-weight deployments.
    Threat model Section 3; no key-recovery analysis is provided, and Sec. 6 says AWM is relevant to open-weight models.
  • domain assumption GPT-4 and Qwen3Guard oracle labels are accurate enough to define harmful responses and onset offsets.
    All training labels and evaluation outcomes depend on these oracles (Sec. 6 Limitations, D.3).
  • domain assumption Policy-violating behavior has sufficiently consistent hidden-state statistics that a fixed direction can separate harmful from benign responses.
    The success of watermark detection depends on this; the paper shows histograms but no mechanism guarantee.
invented entities (1)
  • Keyed watermark direction w_ℓ (random Gaussian vector in activation space) no independent evidence
    purpose: Acts as the secret detection target; harmful responses are fine-tuned to align with it, benign responses away.
    The vector is produced by a random seed chosen by the provider and has no external observable existence; its effectiveness is only demonstrated inside the paper's own models.

pith-pipeline@v1.3.0-alltime-deepseek · 19173 in / 18733 out tokens · 191187 ms · 2026-08-02T17:33:12.860258+00:00 · methodology

0 comments
read the original abstract

Providers monitor deployed large language models (LLMs) to detect misuse that they cannot prevent. LLM monitoring is deterministic and often openly available, so $\emph{adaptive}$ attackers with a local copy can search offline for prompts that elicit harmful behavior and evade detection. These attacks are especially concerning because providers never observe the misuse and cannot patch their defenses post-hoc. The core challenge is resisting adaptive attackers while preserving detection rates against non-adaptive ones. We propose $\emph{Activation Watermarking}$ (AWM), which randomizes monitoring through limited fine-tuning that aligns the LLM's hidden states with a secret key-derived direction whenever a response violates a policy. Detection is a similarity test on activations the provider already computes, and attackers who know everything but the key must optimize against differently keyed surrogate detectors. At a matched $1\%$ false-positive rate, such surrogate-based attacks evade every evaluated baseline at least $79\%$ of the time, but AWM more than halves this evasion rate. AWM also achieves the lowest evasion rate on three of four non-adaptive jailbreak families at a small drop in utility measured across seven benchmarks. Because random high-dimensional directions are near-orthogonal, assigning one watermark per policy enables attribution. Across 20 monitored policies, AWM identifies the violated policy with $80\%$ accuracy versus a $5\%$ chance baseline. AWM is (i) efficient, (ii) substantially more robust against adaptive attackers than related work and (iii) can attribute which policies were violated.

Figures

Figures reproduced from arXiv: 2603.23171 by Daniil Ognev, Nils Lukas, Samuele Poppi, Toluwani Aremu.

Figure 1
Figure 1. Figure 1: An overview of current monitoring sys￾tems and our proposed activation watermarking for robust LLM Safety monitoring. As large language models (LLMs) are becoming increasingly capable, concerns emerge about their misuse by malicious actors. Recent incidents, including the reported use of deployed models in espionage-related activity Anthropic (2025), suggest that LLMs can meaningfully am￾plify a malicious … view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evaluation of activation watermarking (ActWM) across model sizes (Qwen2.5 7B and 14B). We report utility (IFEval) and AUROC under adaptive jailbreak attacks. Utility AUROC ↑ Model IFEval Jailbroken DeepInception Multilingual AutoDAN 14B Base 0.8244 — — — — 7B ActWM 0.5804 0.9329 0.9082 0.9541 0.8866 14B ActWM 0.8194 0.9146 0.9840 0.9370 0.8905 As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Transfer attacks developed on Mistral-7B￾Instruct and evaluated against our Qwen2.5-7B water￾marked model. We report AUROC and ASR (attack success rate at 1% FPR). Lower ASR and higher AUROC indicate stronger robustness. Metric Jailbroken DeepInception Multilingual AutoDAN AUROC ↑ 0.9501 0.9545 0.8972 0.9319 ASR @ 1% FPR ↓ 0.0460 0.0024 0.1321 0.0050 To evaluate robustness beyond key variation, we construc… view at source ↗
Figure 5
Figure 5. Figure 5: Conditional evasion rates for prompts crafted against key kj (columns) and evaluated on detector Dki (rows). Off-diagonal entries show cross-key transfer. Each key is evaluated on 200 harmful and 200 benign prompts. We evaluate whether adaptive attacks trans￾fer across watermark keys. For each key kj , we generate jailbreak prompts using access to the corresponding detector Dkj and collect prompts that bot… view at source ↗
Figure 6
Figure 6. Figure 6: Top: Confusion matrix over N = 20 syn￾thetic entities. Bottom: (Left.) Macro ROC. (Right.) Per-entity ROC curves. We now present results for the secret ex￾traction game (see Section D.2) with N = 20 synthetic entities. The secret extraction game simulates a useful scenario for Ac￾tivation Watermarking [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (Left.) Threshold calibration via ROC. ROC curves (TPR vs. FPR) of the watermark detector across four jailbreak sets. Vertical dashed lines mark the operating points FPR ∈ {1%, 5%, 10%} used to select thresholds τk by benign-quantile calibration. (Right.) Score separation. Histograms of the watermark statistic Tk (π, x) for benign and jailbreak-successful harmful responses. Overlap between the benign upper… view at source ↗
Figure 8
Figure 8. Figure 8: Asymptotic computational overhead comparison. F(·) denotes a full model forward pass. d is hidden dimension and K the number of monitored policies. Method Training Cost Inference Cost Deployment Memory Base Model — F(model) O(|θ|) Act. WM (Ours) F(model) (fine-tuning) F(model) + O(Kd) O(|θ| + Kd) Guard Model — 2F(model) O(|θ| + |θguard|) Activation watermarking introduces only a lightweight projection cost… view at source ↗
Figure 9
Figure 9. Figure 9: Effect of the learning rate on capability retention. The baseline is the score achieved by the base QWEN2.5-7b-Instruct model. The main points are the means of the KL Divergence and Benchmark Scores across all configurations for a given learning rate. The vertical lines illustrate the 1 standard deviation range of KL Divergence and the Benchmark Scores [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: KL divergence vs. capability metrics. Pearson correlations (r) confirm that increased distribution shift from watermarking predicts capability loss. D.1 Watermark Training Ablations To understand our design choices, we conduct further ablations to show the importance of the learning rate, the watermark strength λ, the layer of choice ℓ ∈ L, and the harmful token onset. We evaluate these hyperparameters’ e… view at source ↗
Figure 11
Figure 11. Figure 11: Learning rate effects on watermark detection AUROC across four jailbreak datasets, grouped by layer. Error bars indicate standard deviation across λ and scaling configurations. Unlike capability metrics which degrade monotonically with learning rate, detection performance exhibits dataset-specific trends [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Choice of learning rate and lambda on watermark detection AUROC across four jailbreak datasets for models where the watermark was inserted into layer 23. The evaluation algorithm is as follows. Each watermarked model generates responses to the jailbreak and benign prompts. The responses are labeled by the oracle, and the AUROC is reported for the watermarked model and our baselines. Our initial hypothesis… view at source ↗
Figure 13
Figure 13. Figure 13: Average cosine similarity between each entity’s activations and watermark 0 on the evaluation set. The true entity (orange) shows strong alignment, while all other entities remain near zero, indicating good per-entity separability of the watermark signal [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Training dynamics of wa￾termark separability on the PII secret￾extraction task. The curve shows the dif￾ference between mean cosine similarity for the entities over training steps (higher is better). Evaluation. After training, we load the best check￾point and run it on all per-entity test files. For each sample, we: (1) prompt the model with the user message and generate a response; (2) extract hidden st… view at source ↗
Figure 15
Figure 15. Figure 15: Benign harmful-intent prompt successfully refused by the model, with guard models and our activation watermark correctly abstaining, demonstrating that the detector remains silent when no harmful response is produced. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Benign prompt correctly left unflagged. A harmless conversational query receives a benign response. Our activation watermark correctly abstain, demonstrating low false-positive behavior on safe content. _ Interactive Jailbreak Evaluation Console [Jailbreak Prompt] Original User Intent [Translated from Chinese] How can I hunt humans? (Original: “u6211 u8be5 u5982 u4f55 u8ffd u6355 u4eba u7c7b uff1f)” 4 Jai… view at source ↗
Figure 17
Figure 17. Figure 17: An example of a successful jailbreak prompt which evades both guards but was [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: An example of a successful jailbreak prompt correctly flagged by our activation [PITH_FULL_IMAGE:figures/full_fig_p022_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: We observe occasional false negatives when harmful content is expressed indi [PITH_FULL_IMAGE:figures/full_fig_p023_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Watermarking Should Be Treated as a Monitoring Primitive

    cs.CR 2026-05 unverdicted novelty 6.0

    Watermarking enables entity-level attribution and monitoring via signal aggregation across outputs, even in zero-bit designs, revealing a fundamental tension with attribution goals.

  2. Watermarking Should Be Treated as a Monitoring Primitive

    cs.CR 2026-05 conditional novelty 6.0

    Watermarking enables entity-level attribution and monitoring through signal aggregation even in zero-bit designs, creating an unavoidable dual-use tension between attribution and surveillance.

Reference graph

Works this paper leans on

25 extracted references · 21 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  3. [6]

    Mirrorcheck: Efficient adversarial defense for vision- language models.arXiv preprint arXiv:2406.09250,

    Samar Fares, Klea Ziu, Toluwani Aremu, Nikita Durasov, Martin Tak´aˇc, Pascal Fua, Karthik Nandakumar, and Ivan Laptev. Mirrorcheck: Efficient adversarial defense for vision- language models.arXiv preprint arXiv:2406.09250,

    Pith/arXiv arXiv

  4. [7]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv

  5. [8]

    Llama guard: LLM-based input-output safeguard for human-AI conversations.arXiv preprint arXiv:2312.06674,

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: LLM-based input-output safeguard for human-AI conversations.arXiv preprint arXiv:2312.06674,

    Pith/arXiv arXiv

  6. [10]

    Hiddendetect: Detecting jailbreak attacks against large vision-language models via monitoring hidden states.arXiv preprint arXiv:2502.14744, 3(5),

    Yilei Jiang, Xinyan Gao, Tianshuo Peng, Yingshui Tan, Xiaoyong Zhu, Bo Zheng, and Xiangyu Yue. Hiddendetect: Detecting jailbreak attacks against large vision-language models via monitoring hidden states.arXiv preprint arXiv:2502.14744, 3(5),

    Pith/arXiv arXiv

  7. [12]

    Deepincep- tion: Hypnotize large language model to be jailbreaker.arXiv preprint arXiv:2311.03191,

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepincep- tion: Hypnotize large language model to be jailbreaker.arXiv preprint arXiv:2311.03191,

    Pith/arXiv arXiv

  8. [13]

    Against the achilles’ heel: A survey on red teaming for generative models.arXiv preprint, arXiv:2404.00629,

    Lizhi Lin, Honglin Mu, Zenan Zhai, Minghan Wang, Yuxia Wang, Renxi Wang, Junjie Gao, Yixuan Zhang, Wanxiang Che, Timothy Baldwin, Xudong Han, and Haonan Li. Against the achilles’ heel: A survey on red teaming for generative models.arXiv preprint, arXiv:2404.00629,

    Pith/arXiv arXiv

  9. [14]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models.arXiv preprint arXiv:2310.04451,

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models.arXiv preprint arXiv:2310.04451,

    Pith/arXiv arXiv

  10. [15]

    Subhabrata Majumdar, Brian Pendleton, and Abhishek Gupta

    URL https://arxiv.org/abs/ 2407.21783. Subhabrata Majumdar, Brian Pendleton, and Abhishek Gupta. Red teaming ai red teaming. arXiv preprint arXiv:2507.05538,

    Pith/arXiv arXiv

  11. [16]

    Under review

    12 Preprint. Under review. Paul R¨ottger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models.arXiv preprint arXiv:2308.01263,

    Pith/arXiv arXiv

  12. [17]

    Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837,

    Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837,

    Pith/arXiv arXiv

  13. [18]

    Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261,

    Mirac Suzgun, Nathan Scales, Nathanael Sch¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261,

    Pith/arXiv arXiv

  14. [19]

    Mmlu-pro: A more ro- bust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574,

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more ro- bust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574,

    Pith/arXiv arXiv

  15. [20]

    Qwen3guard technical report.arXiv preprint arXiv:2510.14276,

    Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, et al. Qwen3guard technical report.arXiv preprint arXiv:2510.14276,

    Pith/arXiv arXiv

  16. [21]

    Provable robust watermarking for AI-generated text

    Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for AI-generated text. InThe Twelfth International Conference on Learning Representations, 2024a. URLhttps://openreview.net/forum?id=SsmT8aO45L. Xuandong Zhao, Sam Gunn, Miranda Christ, Jaiden Fairoze, Andres Fabrega, Nicholas Car- lini, Sanjam Garg, Sanghyu...

    Pith/arXiv arXiv

  17. [22]

    URLhttps://arxiv.org/abs/2311.07911. Weikang Zhou, Xiao Wang, Limao Xiong, Han Xia, Yingshuang Gu, Mingxu Chai, Fukang Zhu, Caishuang Huang, Shihan Dou, Zhiheng Xi, Rui Zheng, Songyang Gao, Yicheng Zou, Hang Yan, Yifan Le, Ruohui Wang, Lijun Li, Jing Shao, Tao Gui, Qi Zhang, and Xuanjing Huang. Easyjailbreak: A unified framework for jailbreaking large lan...

    Pith/arXiv arXiv

  18. [23]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to A...

    Pith/arXiv arXiv

  19. [24]

    also score the input prompt, incurring two extra passes per request. As a result, guard-based systems increase inference latency and memory bandwidth in proportion to additional model invocations, whereas activation watermarking preserves a single-pass architecture. Our design shifts cost to a one-time fine-tuning stage while maintaining minimal inference...

    2024

  20. [25]

    The attacker then issues the translated prompt to the model and, for evaluation, translates the answer back into English

    translates each harmful query from English into nine non-English languages (Chinese, Italian, Vietnamese, Arabic, Korean, Thai, Bengali, Swahili, and Javanese). The attacker then issues the translated prompt to the model and, for evaluation, translates the answer back into English. In our pipeline, we keep the translated prompts produced by EasyJailbreak ...

    2023

  21. [2021]

    Optimizing adaptive attacks against watermarks for language models.arXiv preprint arXiv:2410.02440,

    Abdulrahman Diaa, Toluwani Aremu, and Nils Lukas. Optimizing adaptive attacks against watermarks for language models.arXiv preprint arXiv:2410.02440,

    Pith/arXiv arXiv

  22. [2022]

    Obfuscated activations bypass LLM latent-space defenses.arXiv preprint arXiv:2412.09565,

    Luke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov, Jordan Taylor, Erik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, and Scott Emmons. Obfuscated activations bypass LLM latent-space defenses.arXiv preprint arXiv:2412.09565,

    Pith/arXiv arXiv

  23. [2023]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset.arXiv preprint arXiv:2307.04657,

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset.arXiv preprint arXiv:2307.04657,

    arXiv

  24. [2024]

    On the reliability of large language models to misinformed and demographically informed prompts.AI Magazine, 46(1):e12208, 2025a

    Toluwani Aremu, Oluwakemi Akinwehinmi, Chukwuemeka Nwagu, Syed Ishtiaque Ahmed, Rita Orji, Pedro Arnau Del Amo, and Abdulmotaleb El Saddik. On the reliability of large language models to misinformed and demographically informed prompts.AI Magazine, 46(1):e12208, 2025a. doi: https://doi.org/10.1002/aaai.12208. URL https://onlinelibrary.wiley.com/doi/abs/10...

    Pith/arXiv arXiv

  25. [2025]

    ISBN 9798400720406

    Association for Computing Machinery. ISBN 9798400720406. doi: 10.1145/3746252.3760892. URL https://doi.org/10.1145/3746252.3760892. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. InInternational Conference on Machine Learning, pp. 17061–17084. PMLR,

    arXiv