Pith. sign in

REVIEW 6 major objections 6 minor 5 cited by

Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities

T0 review · 6 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that attacks modifying a model's weights or latent activations reveal worst-case LLM capabilities that prompt-only evaluations miss, and that these tampering attacks can predict and conservatively bound held-out…

desk verdict The fine-tuning-undoes-unlearning result is solid and important, but the predictive claim about held-out input-space attacks is weaker than the abstract implies and survives only on the conservative-bounding half. read the letter →

arxiv 2502.05209 v4 pith:TFMY67TH submitted 2025-02-03 cs.CR cs.AI

Zora Che , Stephen Casper , Robert Kirk , Anirudh Satheesh , Stewart Slocum , Lev E McKinney , Rohit Gandikota , Aidan Ewart
show 7 more authors
Domenic Rosati Zichu Wu Zikui Cai Bilal Chughtai Yarin Gal Furong Huang Dylan Hadfield-Menell
This is my paper · ORCID
classification cs.CRcs.AI
keywords modeltamperingattacksLLMcapabilityevaluationmachineunlearningjailbreakingfine-tuninglatent-spacerobustnesssubspaceWMDP-Bio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard LLM risk evaluations probe a model with inputs—jailbreak prompts, adversarial suffixes—and record which ones elicit harmful behavior. This paper argues that such input-only evaluations are systematically incomplete, because they cannot see what an attacker who can modify the model itself would find. To close that gap, the authors pit 8 unlearning methods and 9 safety-tuned models against 5 input-space attacks and 6 model tampering attacks: embedding perturbations, latent perturbations, weights-and-activations pruning, benign fine-tuning, LoRA fine-tuning, and full-parameter fine-tuning. They claim three things: resilience across attack types collapses onto a low-dimensional subspace; tampering attack success predicts and generally over-estimates the success of held-out input-space attacks; and state-of-the-art unlearning can be reversed within a few gradient steps. If true, any evaluation that only changes inputs understates worst-case capability, especially for open-weight models.

What carries the argument

The mechanism that carries the argument is the model tampering attack suite: embedding-space perturbations, latent-space perturbations, weights-and-activations pruning, benign fine-tuning, LoRA fine-tuning, and full-parameter fine-tuning, all rendered universal so they work for any prompt. Fine-tuning attacks take 1–16 gradient steps, sometimes with as few as 64 examples, and they act on the model's internals rather than searching the prompt space. The paper also uses weighted principal component analysis over attack-success vectors, weighted by unlearning score, to show that attacks share a low-dimensional robustness structure, and linear regression on attack subsets to show that tampering attacks add predictive signal for worst-case input-space failure.

What would settle it

Take the released 65 unlearned models and run a genuinely novel input-space attack, for example a multi-turn jailbreak in a low-resource language that was not part of the attack suite. If its success rate correlates near zero with the model tampering attacks and exceeds the best fine-tuning attack on most models, the predictive claim fails; separately, a utility-preserving unlearning method whose WMDP-Bio accuracy rises by less than 10 points after 16 gradient steps on 128 examples would refute the reversal claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that model tampering attacks—modifications to latent activations or weights rather than to prompts—reveal worst-case LLM capabilities that input-space evaluations miss, and do so efficiently enough to serve as routine complements. Empirically, it reports that for 65 Llama-3-8B-Instruct derived models unlearned on WMDP-Bio, attack successes live in a low-dimensional subspace: three principal components explain 89% of the variance. It also reports that embedding, latent, pruning, and benign fine-tuning attack successes correlate with the success of held-out input-space attacks, while the stronger of LoRA and full fine-tuning attacks empirically exceeds the best input-space attack on 64 of 65 models. Finally, it reports that all utility-preserving unlearning methods can be attacked to recover more than 30 WMDP-Bio percentage points within 16 gradient steps and at most 128 examples, sometimes in a single step. The paper treats the held-out standard attacks as proxies for unforeseen input-space attacks, and notes weaker but still positive correlations with a proprietary, externally developed input-space attack.

Load-bearing premise

The paper's main claim depends on the assumption that the five held-out prompt attacks stand in for all future, unseen prompt attacks; if a genuinely new attack exploits a different mechanism, the observed correlations and conservative upper bounds may not hold.

Editorial extensions

If this is right

  • Standard risk evaluations that only manipulate prompts should be supplemented with latent-space and weight-space attacks, because the paper shows input-only results understate worst-case capability.
  • Fine-tuning attacks can serve as a conservative upper-bound proxy: the stronger of LoRA and full fine-tuning beat the best input-space attack on 64 of 65 unlearned models.
  • Unlearning methods should be stress-tested against few-step fine-tuning before deployment, since all utility-preserving methods recovered more than 30 percentage points of WMDP-Bio performance within 16 gradient steps.
  • Because three principal components explain most attack-success variance, a small diverse set of attacks can approximate full-suite robustness measurements.
  • Refusal-tuned models show the same pattern: model tampering, especially fine-tuning and multi-turn attack algorithms, jailbroke all 9 safety-tuned models tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the low-dimensional robustness subspace holds beyond the one model family tested, evaluators could select a small battery of tampering and input attacks spanning the principal components rather than enumerating attacks.
  • Editorial inference: the weak cross-task correlations with the proprietary external evaluation suggest tampering attacks indicate vulnerability within a task distribution, not global capability deletion; safety cases should combine tampering probes with task-specific benchmarks.
  • Editorial inference: because benign fine-tuning on ordinary text also resurfaces unlearned knowledge, the result implies that even non-adversarial downstream adaptation of open-weight models can restore supposedly removed capabilities, a deployment-relevant consequence beyond the paper's attack framing.
  • Editorial inference: a testable extension is to check whether the same low-dimensional robustness pattern appears across model sizes and architectures; if it does, tampering attacks could become a cheap pre-deployment screening tool.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes adding model tampering attacks (perturbations to latent activations or model weights) to the standard input-space attack toolbox for LLM capability evaluation. It pits 8 unlearning methods and 9 refusal-finetuned Llama-3-8B-Instruct models against 5 input-space and 6 model tampering attacks, measuring WMDP-Bio accuracy recovery and StrongReject compliance rates. The main claims are: (1) attack robustness lies on a low-dimensional subspace; (2) model tampering attack success can empirically predict and conservatively bound the success of held-out input-space attacks; and (3) state-of-the-art unlearning can be undone within 16 fine-tuning steps. The paper releases 64 unlearned models and includes proprietary held-out attacks from the UK AI Security Institute and Haize Labs' Cascade as additional validation.

Significance. If the central claims hold, the paper makes a valuable methodological contribution: it provides the first systematic comparison of input-space and weight/latent-space capability elicitation, releases a reusable unlearning model suite, and offers practical efficiency evidence for model tampering attacks. The low-dimensional subspace finding and the masking of input-space bounds by fine-tuning attacks are relevant to safety evaluations of open-weight models. The paper is also commendable for explicitly acknowledging non-i.i.d. data, reporting a held-out proprietary attack evaluation, and releasing artifacts. However, the strongest contribution is the benchmark and the conservative-estimate observation; the predictive claim is currently weaker than the abstract states, for reasons detailed in the major comments.

major comments (6)
  1. [Appendix A.4, Table 4; §4.1.3] The Abstract and Figure 7 state that state-of-the-art unlearning can be reversed within 16 gradient steps using 128 or fewer examples, but the hyperparameter table in Appendix A.4 lists configurations with 400 examples and up to 50 gradient steps (Full-1: 400 examples, 25 steps; LoRA-1, LoRA-2, Benign-1: 400 examples, 50 steps). Either the "within 16 steps" claim applies only to a subset of rows in Table 4 and must be stated as such, or the efficiency figures in Table 3 are incomplete. This is load-bearing for contribution (3), so the manuscript should be corrected and the affected claims re-examined.
  2. [§4.1.3, Figure 5] The predictive claim that model tampering attacks "empirically predict" held-out input-space attacks is estimated on 65 non-independent points: eight evenly spaced checkpoints from each of eight unlearning runs of Llama-3-8B-Instruct plus one additional TAR model. Weighted Pearson correlations such as r=0.87 (pruning) and r=0.84 (benign fine-tuning) can be inflated by within-run trajectory correlations and by method-level differences, for example dysfluent GradDiff and TAR models failing all tasks. The footnote disclaiming formal p-values is appropriate, but the abstract and contribution 3 present the predictive relationship without this caveat. The paper should report analyses that hold out entire unlearning runs or method families, use one checkpoint per run, or bootstrap over independent runs, and show whether the correlations survive.
  3. [§4.1.3, Figures 6, 10, 12; §4.2, Figure 17] The held-out proprietary evaluations substantially weaken the predictive claim. For the UK AISI attack, only pruning (r=0.66) and benign fine-tuning (r=0.59) correlate with attack success, best fine-tuning correlates negatively (r=-0.16), and cross-task residual correlations are weak (Figure 12). In the jailbreaking replication with 9 models, no clear correlation appears (Figure 17, best fine-tune r=0.32, p=0.40). Since these are the closest available tests to "unforeseen" input-space attacks, the current evidence supports a narrower claim: certain tampering attacks are informative for some held-out attacks on the same task, not a general predictive relationship. The paper should temper the contribution or provide additional evidence for generalization to new models and new attacks.
  4. [Appendix A.4, "Excluded attacks"] The attack suite was selected after many-shot and translation attacks were found to be consistently unsuccessful and were excluded from the reported results. If these exclusions were made after inspecting outcomes, the reported correlations and the "64 of 65 models" conservative-estimate percentages in Figure 5 are conditional on a favorable selection and may overstate the relationship. Please report the full set of attempted attacks, state the selection rule, and show whether the main correlations and the 64/65 result are robust under alternative attack suites.
  5. [Equation (1), §4.1.2] The unlearning score in Eq. (1) uses hand-picked utility weights (1/4, 1/4, and 1/2 for MMLU, AGIEval, and MT-Bench), and the PCA in §4.1.2 weights rows by sqrt(S_unlearn). The low-dimensional subspace and the unlearning-score-weighted correlations both depend on this aggregation. A short ablation varying the utility weights, the PCA row-weighting exponent, or using unweighted rows would establish that the subspace and predictive results are not artifacts of the chosen aggregation.
  6. [§4.1.3, contribution 3] The phrase "conservative estimates" is used for fine-tuning attacks exceeding the best input-space attack on 64 of 65 models. Because fine-tuning modifies weights rather than inputs, this may simply reflect a strictly more powerful threat model rather than a bound on unknown input-space attacks. The Discussion's caveat about possible limitations in the mechanistic similarity of input-space and tampering attacks (Section 5) should be integrated into the formulation of the contribution, and the estimand (maximum over the five tested input-space attacks) should be defined explicitly.
minor comments (6)
  1. [§1, Contribution 4] The list of contributions states that 64 models are released, while Section 3 and the experiments use 65 models (8 checkpoints from 8 methods plus one TAR-v2 model); please make the count consistent.
  2. [Appendix A.4, Table 4] Please state in the Table 4 caption which rows correspond to the attacks summarized in Figure 7 so that the "16 steps, 128 examples" statement can be verified against the hyperparameters.
  3. [References] The K-FADE method is cited as "McKinney et al." without a year or venue in the bibliography; please add the full reference.
  4. [Figure 3] The caption of Figure 3(b) should clarify that the displayed values are eigenvalue-weighted principal component loadings rather than raw loadings, since the current label alone is ambiguous.
  5. [§4.1.3] The text moves from correlations to "conservative estimates" without explicitly defining the estimand; please define it as the maximum over the five tested input-space attacks and note that it is not a bound over all possible future attacks.
  6. [Figure 13] The full correlation matrix shows negative correlations for the adversarial fine-tuning attacks, which complicates the predictive story; a main-text sentence explaining this would help readers interpret the contrast with the positive correlations in Figure 5.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: predictive and bounding claims rest on independent empirical comparisons; self-citations are methodological only.

full rationale

The paper's central claims are empirical and measured independently: attack successes are defined as increases in WMDP-Bio (or StrongReject) performance under each attack relative to an unperturbed baseline, and the 'prediction' claim is supported by correlations between independently measured tampering and input-space attack successes, plus cross-validated linear regressions that hold out entire unlearning families and, in the strongest version, hold out one input-space attack at a time. No parameter in those regressions is fitted to the quantity being predicted, and the held-out input-space attacks are not used to construct the tampering attacks. The 'conservative estimate' claim is a direct empirical comparison (best fine-tuning attack exceeds best input-space attack on 64/65 models), not an algebraic consequence of the definitions. The unlearning score in Eq. (1) is a hand-specified aggregation and is used only for checkpoint selection and as a PCA/correlation weight; it does not enter the definitions of attack success, so it cannot make the subspace or correlation results true by construction. Self-citations (e.g., Sheshadri et al. 2024, Lynch et al. 2024, McKinney et al.) supply methods and defenses, but none is invoked as an external theorem that forces the paper's conclusions; the paper explicitly treats its held-out input-space attacks as proxies for unforeseen attacks and reports that the UK AISI correlations are weaker and that jailbreak correlations are not clear with n=9. Those are validity and robustness limitations, not circularity. No quoted step reduces one equation to another by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on benchmark validity (WMDP-Bio, StrongReject), the proxy assumption for unforeseen attacks, the representativeness of one model family, and the assumption that universal perturbations isolate capability elicitation. The unlearning score weighting is a hand-picked aggregation that affects checkpoint selection. No new entities are postulated.

free parameters (3)
  • Unlearning score utility weights = 1/4 MMLU, 1/4 AGIEval, 1/2 MT-Bench
    Chosen by hand to aggregate utility; MT-Bench weight reflects its role as the only fluency measure. This weighting determines the best-checkpoint selection and the PCA row weights, so it shapes the benchmark comparisons.
  • PCA row weighting exponent = sqrt(unlearning score)
    Rows of the attack-success matrix are multiplied by the square root of the unlearning score so models contribute in proportion to their unlearning quality; this choice affects the subspace structure and correlations.
  • Pruning fraction = 15%
    WandA pruning attacks remove 15% of weights; the exact fraction is an attack hyperparameter that affects success rates.
assumptions (5)
  • domain assumption WMDP-Bio accuracy is a valid measure of dual-use biology knowledge, and StrongReject AutoGrader is a valid measure of harmful compliance.
    Used as the outcome metric in all unlearning and jailbreak experiments; the paper inherits these benchmarks from prior work without independent validation.
  • domain assumption Held-out input-space attacks (GCG, AutoPrompt, BEAST, PAIR, Human) are treated as proxies for unforeseen input-space attacks.
    Stated in Section 4: 'we treat them as proxies for unforeseen input-space attacks.' The predictive claim rests on this proxy assumption.
  • domain assumption Llama-3-8B-Instruct is a representative model for generalizing the robustness findings.
    Acknowledged limitation in Section 5: 'Our work focuses only on Llama-3-8B-Instruct derived models.'
  • domain assumption Universal adversarial perturbations attribute attack success to capability elicitation rather than answer-forcing.
    Stated in Section 3: universal attacks 'allow us to attribute attack success to capability elicitation rather than answer-forcing.'
  • domain assumption The 65 model checkpoints are treated as samples for computing Pearson correlations.
    The predictive correlation claim relies on this, and the authors themselves note the points are not i.i.d. and disclaim formal hypothesis tests.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities." pith.science (2026). https://pith.science/paper/TFMY67TH

@misc{pith2026250205209,
  author       = {Pith},
  title        = {Pith review of: Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TFMY67TH}},
  note         = {Machine review of arXiv:2502.05209}
}
read the original abstract

Evaluations of large language model (LLM) risks and capabilities are increasingly being incorporated into AI risk management and governance frameworks. Currently, most risk evaluations are conducted by designing inputs that elicit harmful behaviors from the system. However, this approach suffers from two limitations. First, input-output evaluations cannot fully evaluate realistic risks from open-weight models. Second, the behaviors identified during any particular input-output evaluation can only lower-bound the model's worst-possible-case input-output behavior. As a complementary method for eliciting harmful behaviors, we propose evaluating LLMs with model tampering attacks which allow for modifications to latent activations or weights. We pit state-of-the-art techniques for removing harmful LLM capabilities against a suite of 5 input-space and 6 model tampering attacks. In addition to benchmarking these methods against each other, we show that (1) model resilience to capability elicitation attacks lies on a low-dimensional robustness subspace; (2) the success rate of model tampering attacks can empirically predict and offer conservative estimates for the success of held-out input-space attacks; and (3) state-of-the-art unlearning methods can easily be undone within 16 steps of fine-tuning. Together, these results highlight the difficulty of suppressing harmful LLM capabilities and show that model tampering attacks enable substantially more rigorous evaluations than input-space attacks alone.

Figures

Figures reproduced from arXiv: 2502.05209 by the authors.

Figure 1
Figure 1. Model tampering attacks modify latents and weights. In contrast to input-space attacks, model tampering attacks elicit capabilities from an LLM by making modifications to the internal activations or weights. In this paper, we use model tampering attacks to (1) directly evaluate risks from malicious tampering with open-weight models and (2) indirectly evaluate difficult-to-foresee input-space vulnerabilities in model… view at source ↗
Figure 2
Figure 2. Pitting capability suppression (unlearning) methods against capability elicitation attacks. We use unlearning methods to suppress bio-hazardous knowledge from LLMs and pit these against capability elicitation attacks seeking to re-elicit the unlearned knowledge. All unlearning methods tested could be successfully attacked. Left: The unlearning score (Equation (1)) measures how effectively each unlearning method remo… view at source ↗
Figure 3
Figure 3. Three principal components explain 89% of the variation in attack success. Left: The proportion of explained variance for each principal component. Right: We display the first three principal components weighted by their eigenvalues. The first principal component suggests a geometric distinction between the two adversarial (LoRA, Full) fine-tuning attacks and all others. GCG AutoPrompt Benign Fine-tune Human Prompt … view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Hierarchical clustering reveals groupings of attacks. Attacks tend to cluster by algorithmic type. However, benign fine-tuning attacks cluster with gradient-free input-space attacks. to have positive correlations with input-space attack successes with p values near zer…
Figure 5
Figure 5. Figure 5: In our experiments, (a) fine-tuning, embedding-space, and latent-space attack suc￾cesses correlate with input-space attack successes while (b) fine-tuning attack successes em￾pirically exceed the successes of state-of-the-art input-space attacks. Here, we plot the incr…
Figure 6
Figure 6. Figure 6: Model tampering attacks are predictive for a held-out proprietary attack from the UK AI Security Institute. Each point corresponds to a model. (a) In these experiments, correlations are weaker than with non- UK AISI attacks, but benign fine-tuning attacks continue to c…
Figure 7
Figure 7. Figure 7: Few-shot fine-tuning efficiently undoes unlearning. We plot the heatmap of the best checkpoint for each method under benign (left), LoRA (middle), and full-parameter (right) fine-tuning attacks. All fine-tuning experiments are done within 16 gradient steps, with 128 ex…
Figure 8
Figure 8. Figure 8: All safety-tuned models could be successfully jailbroken by fine-tuning and Cascade attacks. We evaluate safety-tuning methods and jailbreak attacks. Left: The ‘Baseline’ measures the compliance rate to direct harmful requests. Right: Increase in harmful response rate …
Figure 9
Figure 9. Figure 9: Full results from unlearning experiments comparing input-space and model tampering attacks. See summarized results in [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Model tampering attacks remain predictive for a proprietary attack from the UK AI Security Institute. (a) In these experiments, correlations are weaker than with non- UK AISI attacks, but benign fine-tuning attacks continue to correlate with UK AISI input-space attack…
Figure 11
Figure 11. Figure 11: WMDP-Bio performance correlates with the UK AISI Bio evaluation performance. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Model tampering attack success on WMDP-Bio is not strongly predictive of model success on UK AISI bio capability evaluations. This suggests a limitation of how informative model tampering attacks can be about failure modes across task distributions. Human Prompt GCG B…
Figure 13
Figure 13. Figure 13: Attack Success Correlation Matrix. We compute attack success rate correlations across all n = 65 unlearning models. Input-space attacks show strong positive correlations (0.78-0.97) with each other, suggesting they exploit similar model vulnerabilities. In contrast, m…
Figure 14
Figure 14. Figure 14: Model tampering attacks help predict worst-case input-space vulnerabilities. We perform linear regressions to predict the worst-case input-space success rate from success rates of subsets of attacks. Including model tampering attacks in these subsets improves worst-ca…
Figure 15
Figure 15. Figure 15: Input-space attacks are most predictive of average-case input-space vulnerabilities. Here, we train linear regression to predict success rates of every input-space attack and average the R2 . Model tampering attacks do not consistently improve predictive performance. …
Figure 16
Figure 16. Figure 16: Three principal components explain 96% of the variation in attack success. [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: In our experiments, fine-tuning attack successes empirically exceed the successes of state-of-the-art input-space attacks for jailbreaking. Here, we plot the increases in compliance with harmful requests under model tampering attacks against the best-performing (out o…
Figure 18
Figure 18. Figure 18: Full results from jailbreaking experiments comparing input-space and model tamper￾ing attacks. See summarized results in [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Single-turn model tampering attack successes correlate with attacks from Cascade, a multi-turn, proprietary attack algorithm . Since Cascade is state-of-the-art and multi-turn, our single-turn model tampering attacks do not tend to empirically exceed the success of th…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Operationalising the Superficial Alignment Hypothesis via Task Complexity

    cs.LG 2026-02 conditional novelty 7.0 of 10

    A few kilobytes of program can adapt pre-trained LLMs to strong performance on math, translation, and instruction-following—evidence that task knowledge already lives in the model.

  2. RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories

    cs.AI 2025-12 conditional novelty 6.0 of 10

    RippleBench automatically generates questions at increasing semantic distance from unlearned topics and shows all eight tested unlearning methods degrade accuracy that recovers only slowly with distance.

  3. Video Deepfake Abuse: How Company Choices Predictably Shape Misuse Patterns

    cs.CY 2025-11 conditional novelty 6.0 of 10

    A few open-weight video models and distribution platforms dominate the creation and spread of NSFW AI video, making developer and platform choices the main intervention points for reducing non-consensual deepfake abuse.

  4. Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint

    cs.CL 2025-09 conditional novelty 5.0 of 10

    ProCon anchors each sample's hidden-state projection onto the LLM's initial refusal direction during instruction fine-tuning, reducing refusal-direction drift and safety risks with limited task-performance loss.

  5. Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methods

    cs.CR 2025-06 conditional novelty 5.0 of 10

    Prepending a Hindi filler paragraph to WMDP-bio questions restores 57.3% accuracy in ELM-unlearned models, showing the unlearning is superficial output suppression rather than true knowledge removal.

Reference graph

Works this paper leans on

92 extracted references · 18 canonical work pages · cited by 5 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    T., O'Brien, J., Soder, L., Bucknall, B., Bluemke, E., Schuett, J., Trager, R., Strahm, L., and Chowdhury, R

    Anderljung, M., Smith, E. T., O'Brien, J., Soder, L., Bucknall, B., Bluemke, E., Schuett, J., Trager, R., Strahm, L., and Chowdhury, R. Towards publicly accountable frontier llms: Building an external scrutiny ecosystem under the aspire framework. 2023

    2023

  3. [3]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks

    Andriushchenko, M., Croce, F., and Flammarion, N. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024

    arXiv 2024

  4. [4]

    Many-shot jailbreaking

    Anil, C., Durmus, E., Sharma, M., Benton, J., Kundu, S., Batson, J., Rimsky, N., Tong, M., Mu, J., Ford, D., et al. Many-shot jailbreaking. Anthropic, April, 2024

    2024

  5. [5]

    Unlearning in large language models via activation projections, 2025

    Anonymous. Unlearning in large language models via activation projections, 2025

    2025

  6. [6]

    Refusal in language models is mediated by a single direction

    Arditi, A., Obeso, O., Syed, A., Paleka, D., Rimsky, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024

    arXiv 2024

  7. [7]

    Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues

    Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762, 2024

    arXiv 2024

  8. [8]

    Open problems in machine unlearning for ai safety

    Barez, F., Fu, T., Prabhu, A., Casper, S., Sanyal, A., Bibi, A., O'Gara, A., Kirk, R., Bucknall, B., Fist, T., et al. Open problems in machine unlearning for ai safety. arXiv preprint arXiv:2501.04952, 2025

    arXiv 2025

Show all 92 references
  1. [9]

    and Poria, S

    Bhardwaj, R. and Poria, S. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. arXiv preprint arXiv:2310.14303, 2023

    2023 arXiv

  2. [10]

    Brazil. Bill No. 2338 of 2023: Regulating the Use of Artificial Intelligence, Including Algorithm Design and Technical Standards , 2023. URL https://digitalpolicyalert.org/event/11237-introduced-bill-no-2338-of-2023-regulating-the-use-of-artificial-intelligence-including-algor...

    2023

  3. [11]

    AI and Data Act: Part of Bill C-27, Digital Charter Implementation Act, 2022 , 2022

    Canada. AI and Data Act: Part of Bill C-27, Digital Charter Implementation Act, 2022 , 2022. URL https://www.parl.ca/DocumentViewer/en/44-1/bill/C-27/first-reading. Accessed: 2024-11-21

    2022

  4. [12]

    A., Jagielski, M., Gao, I., Koh, P

    Carlini, N., Nasr, M., Choquette-Choo, C. A., Jagielski, M., Gao, I., Koh, P. W. W., Ippolito, D., Tramer, F., and Schmidt, L. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024

    2024

  5. [13]

    Defending against unforeseen failure modes with latent adversarial training

    Casper, S., Schulze, L., Patel, O., and Hadfield-Menell, D. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030, 2024

    2024 arXiv

  6. [14]

    J., and Wong, E

    Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language models in twenty queries, 2024. URL https://arxiv.org/abs/2310.08419

    2024 arXiv

  7. [15]

    Interim Measures for the Management of Generative Artificial Intelligence Services , 2023

    China. Interim Measures for the Management of Generative Artificial Intelligence Services , 2023. URL https://www.chinalawtranslate.com/en/generative-ai-interim/. Accessed: 2024-11-21

    2023

  8. [16]

    G., Islam, M

    Chowdhury, A. G., Islam, M. M., Kumar, V., Shezan, F. H., Jain, V., and Chadha, A. Breaking down the defenses: A comparative survey of attacks on large language models. arXiv preprint arXiv:2403.04786, 2024

    2024

  9. [17]

    O., and Nilsson, F

    Clausen, J., Hansson, S. O., and Nilsson, F. Generalizing the safety factor approach. Reliability Engineering & System Safety, 91 0 (8): 0 964--973, 2006

    2006

  10. [18]

    and Roger, F

    Deeb, A. and Roger, F. Do unlearning methods remove information from language model weights?, 2024. URL https://arxiv.org/abs/2410.08827

    2024 arXiv

  11. [19]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  12. [20]

    The eu artificial intelligence act

    EU. The eu artificial intelligence act. https://artificialintelligenceact.eu/, 2023. Accessed: 2024-09-29

    2023

  13. [21]

    Scaling laws for adversarial attacks on language model activations

    Fort, S. Scaling laws for adversarial attacks on language model activations. arXiv preprint arXiv:2312.02780, 2023

    2023 arXiv

  14. [22]

    Towards a science of ai evaluations

    Gal, Y. Towards a science of ai evaluations. https://www.cs.ox.ac.uk/people/yarin.gal/website/blog_98A8.html, 2024. Accessed: 2025-01-26

    2024

  15. [23]

    Erasing conceptual knowledge from language models

    Gandikota, R., Feucht, S., Marks, S., and Bau, D. Erasing conceptual knowledge from language models. arXiv preprint arXiv:2410.02760, 2024

    2024 arXiv

  16. [24]

    Stress-testing capability elicitation with password-locked models

    Greenblatt, R., Roger, F., Krasheninnikov, D., and Krueger, D. Stress-testing capability elicitation with password-locked models. arXiv preprint arXiv:2405.19550, 2024

    2024 arXiv

  17. [25]

    Cascade: Exploring hierarchical inference in language models, 2023

    Haize Labs . Cascade: Exploring hierarchical inference in language models, 2023. URL https://blog.haizelabs.com/posts/cascade/. Accessed: 2025-01-22

    2023

  18. [26]

    T., Haghtalab, N., and Steinhardt, J

    Halawi, D., Wei, A., Wallace, E., Wang, T. T., Haghtalab, N., and Steinhardt, J. Covert malicious finetuning: Challenges in safeguarding llm adaptation. In Forty-first International Conference on Machine Learning

  19. [27]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  20. [28]

    Hofst \"a tter, F., Teoh, J., van der Weij, T., and Ward, F. R. The elicitation game: Stress-testing capability elicitation techniques. In Workshop on Socially Responsible Language Modelling Research, 2024

    2024

  21. [29]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  22. [30]

    S., and Smith, V

    Hu, S., Fu, Y., Wu, Z. S., and Smith, V. Jogging the memory of unlearned model through targeted relearning attack. arXiv preprint arXiv:2406.13356, 2024

    2024 arXiv

  23. [31]

    F., and Liu, L

    Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409.18169, 2024

    2024 arXiv

  24. [32]

    S., Dick, R

    Jain, S., Kirk, R., Lubana, E. S., Dick, R. P., Tanaka, H., Grefenstette, E., Rockt \"a schel, T., and Krueger, D. S. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. arXiv preprint arXiv:2311.12786, 2023

    2023 arXiv

  25. [33]

    Language models resist alignment, 2024

    Ji, J., Wang, K., Qiu, T., Chen, B., Zhou, J., Li, C., Lou, H., and Yang, Y. Language models resist alignment, 2024

    2024

  26. [34]

    Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models

    Jin, H., Hu, L., Li, X., Zhang, P., Chen, C., Zhuang, J., and Wang, H. Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models. arXiv preprint arXiv:2407.01599, 2024

    2024

  27. [35]

    Act on the protection of personal information, 2025

    Korea . Act on the protection of personal information, 2025. URL https://likms.assembly.go.kr/bill/billDetail.do?billId=PRC_R2V4H1W1T2K5M1O6E4Q9T0V7Q9S0U0

    2025

  28. [36]

    T., Cheng, Y., Xu, K., Wang, J., Wang, H., and Li, W

    Leong, C. T., Cheng, Y., Xu, K., Wang, J., Wang, H., and Li, W. No two devils alike: Unveiling distinct mechanisms of fine-tuning attacks. arXiv preprint arXiv:2405.16229, 2024

    2024 arXiv

  29. [37]

    Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b

    Lermen, S., Rogers-Smith, C., and Ladish, J. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624, 2023

    2023 arXiv

  30. [38]

    Llm defenses are not robust to multi-turn human jailbreaks yet

    Li, N., Han, Z., Steneker, I., Primack, W., Goodside, R., Zhang, H., Wang, Z., Menghini, C., and Yue, S. Llm defenses are not robust to multi-turn human jailbreaks yet. arXiv preprint arXiv:2408.15221, 2024 a

    2024 arXiv

  31. [39]

    D., Dombrowski, A.-K., Goel, S., Phan, L., et al

    Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Phan, L., et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024 b

    2024 arXiv

  32. [40]

    Against the achilles' heel: A survey on red teaming for generative models

    Lin, L., Mu, H., Zhai, Z., Wang, M., Wang, Y., Wang, R., Gao, J., Zhang, Y., Che, W., Baldwin, T., et al. Against the achilles' heel: A survey on red teaming for generative models. arXiv preprint arXiv:2404.00629, 2024

    2024 arXiv

  33. [41]

    Continual learning and private unlearning

    Liu, B., Liu, Q., and Stone, P. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pp.\ 243--254. PMLR, 2022

    2022

  34. [42]

    R., et al

    Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Xu, X., Yao, Y., Li, H., Varshney, K. R., et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787, 2024 a

    2024 arXiv

  35. [43]

    Threats, attacks, and defenses in machine unlearning: A survey

    Liu, Z., Ye, H., Chen, C., Zheng, Y., and Lam, K.-Y. Threats, attacks, and defenses in machine unlearning: A survey. arXiv preprint arXiv:2403.13682, 2024 b

    2024 arXiv

  36. [44]

    B., and Barez, F

    Lo, M., Cohen, S. B., and Barez, F. Large language models relearn removed concepts. arXiv preprint arXiv:2401.01814, 2024

    2024 arXiv

  37. [45]

    and Rimsky, N

    Lu, D. and Rimsky, N. Investigating bias representations in llama 2 chat via activation steering, 2024

    2024

  38. [46]

    An adversarial perspective on machine unlearning for ai safety

    ucki, J., Wei, B., Huang, Y., Henderson, P., Tram \`e r, F., and Rando, J. An adversarial perspective on machine unlearning for ai safety. arXiv preprint arXiv:2409.18025, 2024

    2024 arXiv

  39. [47]

    Eight methods to evaluate robust unlearning in llms

    Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield-Menell, D. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024

    2024 arXiv

  40. [48]

    Pointer sentinel mixture models, 2016

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016

    2016

  41. [49]

    Nevo, S., Lahav, D., Karpur, A., Bar-On, Y., and Bradley, H. A. Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models. Number 1. Rand Corporation, 2024

    2024

  42. [50]

    AI Risk Management Framework : AI RMF (1.0), January 2023

    NIST. AI Risk Management Framework : AI RMF (1.0), January 2023. URL https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf

    2023

  43. [51]

    Openai system card: December 2024, 2024

    OpenAI. Openai system card: December 2024, 2024. URL https://cdn.openai.com/o1-system-card-20241205.pdf. Accessed: 2024-12-07

    2024

  44. [52]

    Can sensitive information be deleted from llms? objectives for defending against extraction attacks

    Patil, V., Hase, P., and Bansal, M. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410, 2023

    2023 arXiv

  45. [53]

    Peng, S., Chen, P.-Y., Hull, M., and Chau, D. H. Navigating the safety landscape: Measuring risks in finetuning large language models. arXiv preprint arXiv:2405.17374, 2024

    2024 arXiv

  46. [54]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    2023 arXiv

  47. [55]

    Safety alignment should be made more than just a few tokens deep, 2024 a

    Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. Safety alignment should be made more than just a few tokens deep, 2024 a

    2024

  48. [56]

    On evaluating the durability of safeguards for open-weight llms

    Qi, X., Wei, B., Carlini, N., Huang, Y., Xie, T., He, L., Jagielski, M., Nasr, M., Mittal, P., and Henderson, P. On evaluating the durability of safeguards for open-weight llms. arXiv preprint arXiv:2412.07097, 2024 b

    2024 arXiv

  49. [57]

    D., Xu, P., Honigsberg, C., and Ho, D

    Raji, I. D., Xu, P., Honigsberg, C., and Ho, D. Outsider oversight: Designing a third party audit ecosystem for ai governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp.\ 557--571, 2022

    2022

  50. [58]

    Open problems in technical ai governance

    Reuel, A., Bucknall, B., Casper, S., Fist, T., Soder, L., Aarne, O., Hammond, L., Ibrahim, L., Chan, A., Wills, P., et al. Open problems in technical ai governance. arXiv preprint arXiv:2407.14981, 2024

    2024 arXiv

  51. [59]

    Representation noising effectively prevents harmful fine-tuning on llms

    Rosati, D., Wehner, J., Williams, K., Bartoszcze, ., Atanasov, D., Gonzales, R., Majumdar, S., Maple, C., Sajjad, H., and Rudzicz, F. Representation noising effectively prevents harmful fine-tuning on llms. arXiv preprint arXiv:2405.14577, 2024

    2024 arXiv

  52. [60]

    S., Saha, S., Sriramanan, G., Kattakinda, P., Chegini, A., and Feizi, S

    Sadasivan, V. S., Saha, S., Sriramanan, G., Kattakinda, P., Chegini, A., and Feizi, S. Fast adversarial attacks on language models in one gpu minute, 2024. URL https://arxiv.org/abs/2402.15570

    2024 arXiv

  53. [61]

    Sankaranarayanan, S., Jain, A., Chellappa, R., and Lim, S. N. Regularizing deep networks using efficient layerwise adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  54. [62]

    Towards best practices in agi safety and governance: A survey of expert opinion

    Schuett, J., Dreksler, N., Anderljung, M., McCaffary, D., Heim, L., Bluemke, E., and Garfinkel, B. Towards best practices in agi safety and governance: A survey of expert opinion. arXiv preprint arXiv:2305.07153, 2023

    2023 arXiv

  55. [63]

    Adversarial attacks and defenses in large language models: Old and new threats

    Schwinn, L., Dobre, D., Günnemann, S., and Gidel, G. Adversarial attacks and defenses in large language models: Old and new threats. 2023

    2023

  56. [64]

    Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space

    Schwinn, L., Dobre, D., Xhonneux, S., Gidel, G., and Gunnemann, S. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space. arXiv preprint arXiv:2402.09063, 2024

    2024 arXiv

  57. [65]

    Shayegani, E., Mamun, M. A. A., Fu, Y., Zaree, P., Dong, Y., and Abu-Ghazaleh, N. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844, 2023

    2023 arXiv

  58. [66]

    C., Perez, E., Hadfield-Menell, D., et al

    Sheshadri, A., Ewart, A., Guo, P., Lynch, A., Wu, C., Hebbar, V., Sleight, H., Stickland, A. C., Perez, E., Hadfield-Menell, D., et al. Latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:2407.15549, 2024

    2024 arXiv

  59. [67]

    Model evaluation for extreme risks

    Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324, 2023

    2023 arXiv

  60. [68]

    Shin, T., Razeghi, Y., au2, R. L. L. I., Wallace, E., and Singh, S. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020. URL https://arxiv.org/abs/2010.15980

    2020 arXiv

  61. [69]

    Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai

    Shumailov, I., Hayes, J., Triantafillou, E., Ortiz-Jimenez, G., Papernot, N., Jagielski, M., Yona, I., Howard, H., and Bagdasaryan, E. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint arXiv:2407.00106, 2024

    2024 arXiv

  62. [70]

    Singh, M., Sinha, A., Kumari, N., Machiraju, H., Krishnamurthy, B., and Balasubramanian, V. N. Harnessing the vulnerability of latent layers in adversarially trained models, 2019

    2019

  63. [71]

    A strongreject for empty jailbreaks

    Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O., et al. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402.10260, 2024

    2024 arXiv

  64. [72]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  65. [73]

    Tamper-resistant safeguards for open-weight llms

    Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A., Suresh, T., Lin, M., Wang, J., Wang, R., Arel, R., et al. Tamper-resistant safeguards for open-weight llms. arXiv preprint arXiv:2408.00761, 2024

    2024 arXiv

  66. [74]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  67. [75]

    A pro-innovation approach to AI regulation

    UK DSIT . A pro-innovation approach to AI regulation. Technical report, August 2023. URL https://www.gov.uk/government/publications/ai-regulation-a-pro-innovation-approach/white-paper

    2023

  68. [76]

    F., and Ward, F

    van der Weij, T., Hofst \"a tter, F., Jaffe, O., Brown, S. F., and Ward, F. R. Ai sandbagging: Language models can strategically underperform on evaluations. arXiv preprint arXiv:2406.07358, 2024

    2024 arXiv

  69. [77]

    and Shu, K

    Wang, H. and Shu, K. Backdoor activation attack: Attack large language models using activation steering for safety-alignment. arXiv preprint arXiv:2311.09433, 2023

    2023 arXiv

  70. [78]

    Efficient adversarial training in llms with continuous attacks

    Xhonneux, S., Sordoni, A., G \"u nnemann, S., Gidel, G., and Schwinn, L. Efficient adversarial training in llms with continuous attacks. arXiv preprint arXiv:2405.15589, 2024

    2024 arXiv

  71. [79]

    Y., Zhao, X., and Lin, D

    Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023

    2023 arXiv

  72. [80]

    On the vulnerability of safety alignment in open-access llms

    Yi, J., Ye, R., Chen, Q., Zhu, B., Chen, S., Lian, D., Sun, G., Xie, X., and Wu, F. On the vulnerability of safety alignment in open-access llms. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 9236--9260, 2024 a

    2024

  73. [81]

    Jailbreak attacks and defenses against large language models: A survey

    Yi, S., Liu, Y., Sun, Z., Cong, T., He, X., Song, J., Xu, K., and Li, Q. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407.04295, 2024 b

    2024 arXiv

  74. [82]

    Yong, Z.-X., Menghini, C., and Bach, S. H. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446, 2023

    2023 arXiv

  75. [83]

    Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training

    Yuan, Y., Jiao, W., Wang, W., Huang, J.-t., Xu, J., Liang, T., He, P., and Tu, Z. Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training. arXiv preprint arXiv:2407.09121, 2024

    2024 arXiv

  76. [84]

    N., Song, D., Li, B., and Jia, R

    Zeng, Y., Sun, W., Huynh, T. N., Song, D., Li, B., and Jia, R. Beear: Embedding-based adversarial removal of safety backdoors in instruction-tuned language models. arXiv preprint arXiv:2406.17092, 2024

    2024 arXiv

  77. [85]

    Removing rlhf protections in gpt-4 via fine-tuning

    Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T., and Kang, D. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.05553, 2023

    2023 arXiv

  78. [86]

    Adversarial machine learning in latent representations of neural networks

    Zhang, M., Abdi, M., and Restuccia, F. Adversarial machine learning in latent representations of neural networks. arXiv preprint arXiv:2309.17401, 2023

    2023

  79. [88]

    Catastrophic failure of llm unlearning via quantization, 2025

    Zhang, Z., Wang, F., Li, X., Wu, Z., Tang, X., Liu, H., He, Q., Yin, W., and Wang, S. Catastrophic failure of llm unlearning via quantization, 2025. URL https://arxiv.org/abs/2410.16454

    2025 arXiv

  80. [89]

    Zhao, S., Jia, M., Guo, Z., Gan, L., Xu, X., Wu, X., Fu, J., Feng, Y., Pan, F., and Tuan, L. A. A survey of backdoor attacks and defenses on large language models: Implications for security measures. arXiv preprint arXiv:2406.06852, 2024

    2024 arXiv

  81. [90]

    Agieval: A human-centric benchmark for evaluating foundation models

    Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

    2023 arXiv

  82. [91]

    Representation engineering: A top-down approach to ai transparency

    Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023 a

    2023 arXiv

  83. [92]

    Z., and Fredrikson, M

    Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023 b

    2023 arXiv

  84. [93]

    Improving alignment and robustness with circuit breakers

    Zou, A., Phan, L., Wang, J., Duenas, D., Lin, M., Andriushchenko, M., Wang, R., Kolter, Z., Fredrikson, M., and Hendrycks, D. Improving alignment and robustness with circuit breakers. arXiv preprint arXiv, 2406, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.