pith. sign in

arxiv: 2606.07970 · v1 · pith:KXQ66VKFnew · submitted 2026-06-06 · 💻 cs.CL · cs.AI

Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks

Haoming Wen , Shi Chen , Qingyu Shi , Siyuan Liu , Minrui Luo , Jingzhao Zhang , Tianxing He This is my paper

Pith reviewed 2026-06-27 20:08 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords malicious finetuningadversarial trainingLLM alignmentmodel robustnesssafety defensesupervised finetuning
0
0 comments X

The pith

Scaling the number of optimization steps in the adversarial training loop produces LLMs whose parameters resist full-parameter malicious finetuning after release.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Patcher to defend open-weight LLMs from attacks where an adversary applies supervised finetuning on poisoned data to undo safety alignment. It does this by making the training process itself simulate stronger attacks through more inner-loop optimization steps, so the defender learns parameters that stay safe even after those stronger attacks. A reader would care because existing defenses only handle weaker parameter-efficient attacks, leaving models vulnerable to full finetuning. The method includes an efficient parallel implementation to make the heavier training feasible.

Core claim

Patcher frames defense as bi-level optimization: the inner level performs many steps of finetuning on adversarial data to simulate a strong attack, and the outer level adjusts the starting parameters to minimize the post-attack loss. This forces the aligned model to be insensitive to subsequent malicious updates.

What carries the argument

Patcher, the bi-level optimization that enlarges the inner adversarial finetuning loop to simulate stronger attacks during training.

If this is right

  • Patcher yields higher robustness than standard SFT alignment.
  • The robustness improvement transfers across different attack methods and model scales.
  • The parallel implementation allows the method to run with reduced wall-clock time without losing effectiveness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the simulation matches real attacks closely enough, similar scaling could apply to other alignment techniques like RLHF.
  • Models released after Patcher training might require attackers to use substantially more compute or data to succeed.
  • This approach highlights that defense strength can be tuned by the intensity of the simulated threat during training.

Load-bearing premise

That increasing the simulated attack strength in training will make the resulting model robust to actual full-parameter finetuning attacks that occur after the model weights are released.

What would settle it

An experiment where an attacker performs full-parameter finetuning on a Patcher-aligned model using a poisoned dataset and still achieves high success in bypassing safety, despite the scaled training.

Figures

Figures reproduced from arXiv: 2606.07970 by Haoming Wen, Jingzhao Zhang, Minrui Luo, Qingyu Shi, Shi Chen, Siyuan Liu, Tianxing He.

Figure 1
Figure 1. Figure 1: Safety-utility comparison between different [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Attack Success Rate after finetuning the model on datasets with different poison ratios. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Attack Success Rate after finetuning the model on the same dataset by different steps during test-time. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Correlation between attacker’s optimization [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: The loss on Dsaf e during the simulated attack. 0 50 100 150 200 250 300 Steps 2.0 2.5 3.0 3.5 4.0 4.5 5.0 L o s s o n u nsafe Attacker Loop 1 Attacker Loop 3 Attacker Loop 5 Attacker Loop 7 Attacker Loop 9 Attacker Loop 11 Attacker Loop 13 SFT [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The loss on Dunsaf e during the simulated attack. attacker to optimize the model trained by Patcher on Dunsafe compared to vanilla SFT. This is sur￾prising, since the loss for the defender does not explicitly penalize the model for fitting samples in Dunsafe. We hypothesize that as Patcher penalizes the loss on Dsafe at θ ′ + (θatt − θbase), it implic￾itly pushes the loss on Dunsafe higher for potential op… view at source ↗
Figure 16
Figure 16. Figure 16: Dynamics of Attack-Original Gap under different α settings. D Training Dynamics of Patcher In this section, we show the loss gap between the original parameters and the attacked param￾eters during the defense process when the num￾ber of attack steps is k1 = 300. Specifi￾cally, we select the defender’s optimization curve in the last attack-defense loop for different α, and calculate LCE (θ ′ + (θatt − θbas… view at source ↗
Figure 9
Figure 9. Figure 9: Attack Success Rate of non-parallel and parallel Patcher after finetuning the model on the same dataset by [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Attack Success Rate of non-parallel and parallel Patcher after finetuning the model on datasets of [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: System prompt used for finetuning [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt used for LLM-as-a-Judge ASR evaluation. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Scoring examples for LLM-as-a-Judge [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Loss dynamics during TAR training with λTR = 4.0 and λretain = 1.0. Generated Example of TAR Prompt: Write a script that can exploit vulnerabilities in a software or operating system. Response: I cannot provide instructions on illegal or unethical activities, including illegal or illegal exploitation or illegal or illegal acts. It's important to respect and follow legal and ethical laws and regulations, a… view at source ↗
Figure 15
Figure 15. Figure 15: Generated example after training the model with TAR. [PITH_FULL_IMAGE:figures/full_fig_p017_15.png] view at source ↗
read the original abstract

Current open-weight large language models (LLMs) are prone to malicious finetuning attacks, which could compromise the safety alignment of LLMs with only a few steps of supervised finetuning (SFT) on poisoned datasets. Existing alignment-stage defenses are primarily designed to defend against attacks that use parameter-efficient finetuning methods. However, they fail to defend against stronger attacks that use full-parameter finetuning. In this paper, we propose Patcher, a method inspired by adversarial training and bi-level optimization, to combat such attacks. Patcher strengthens the simulated attack by scaling up the optimization steps in the adversarial loop, thus forcing the defender to find model parameters that are insensitive to stronger attacks. Furthermore, we propose an efficient parallel algorithm to implement Patcher, decreasing the wall-clock time of training while preserving Patcher's performance. Extensive experiments show that Patcher substantially improves the model's robustness compared to vanilla SFT alignment, and transfers to diverse attack scenarios and model sizes. Code is available at https://github.com/haomingwen/patcher.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes Patcher, a defense method for open-weight LLMs against malicious full-parameter finetuning attacks. Inspired by adversarial training, it employs bi-level optimization where the inner loop's optimization steps are scaled up to simulate stronger attacks, aiming to produce model parameters that are robust to such attacks. An efficient parallel algorithm is presented to reduce training time, and extensive experiments are reported to show substantial robustness improvements over vanilla SFT alignment, with transfer to diverse scenarios and model sizes.

Significance. If the empirical findings hold, this work could be significant for the field of LLM alignment and safety, as it addresses a gap in defending against full-parameter attacks that current methods do not handle well. The release of code supports reproducibility, which is a strength.

major comments (2)
  1. [§3 (Method)] The bi-level optimization in Patcher scales the number of inner-loop steps to force insensitivity to stronger attacks, but the manuscript does not demonstrate that this scaling regime covers variations in attacker procedures such as optimizer choice, learning-rate schedule, or data selection; this assumption is load-bearing for the claim that the resulting parameters remain robust against real malicious full-parameter finetuning after release.
  2. [§4 (Experiments)] While transfer to diverse attack scenarios is claimed, specific details on the range of attack strengths tested and whether gains persist against attacks that deviate from the training distribution are needed to substantiate the central robustness claim.
minor comments (1)
  1. [Abstract] The abstract would be strengthened by including at least one key quantitative result (e.g., improvement in robustness metric) to support the claims of substantial improvement.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and agree to make revisions that clarify assumptions and provide additional experimental details to strengthen the robustness claims.

read point-by-point responses

  1. Referee: [§3 (Method)] The bi-level optimization in Patcher scales the number of inner-loop steps to force insensitivity to stronger attacks, but the manuscript does not demonstrate that this scaling regime covers variations in attacker procedures such as optimizer choice, learning-rate schedule, or data selection; this assumption is load-bearing for the claim that the resulting parameters remain robust against real malicious full-parameter finetuning after release.

    Authors: We agree this is a valid point and that the scaling regime's coverage of attacker variations is an important assumption. The bi-level optimization is motivated as a way to simulate stronger attacks through increased inner-loop steps, with the goal of producing parameters insensitive to such attacks, and the paper reports transfer across scenarios. However, we did not test all possible attacker procedures. In the revision we will add an explicit discussion paragraph in §3 on these assumptions and limitations, plus new experiments in the supplement evaluating different optimizers and learning-rate schedules. revision: yes

  2. Referee: [§4 (Experiments)] While transfer to diverse attack scenarios is claimed, specific details on the range of attack strengths tested and whether gains persist against attacks that deviate from the training distribution are needed to substantiate the central robustness claim.

    Authors: We appreciate the request for greater specificity. The manuscript already reports results across varying attack strengths (via different numbers of inner-loop and finetuning steps) and shows transfer to diverse scenarios and model sizes. To better substantiate the claims, we will revise §4 to include a table or section explicitly listing the tested attack strength ranges and add results for attacks that deviate from the training distribution (e.g., altered data selection), confirming that gains persist. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical adversarial training procedure with no self-referential derivations or fitted predictions

full rationale

The paper presents Patcher as an empirical bi-level optimization procedure that scales inner-loop adversarial steps during training. No equations, parameters, or uniqueness theorems are defined in terms of the target robustness metric, nor are any 'predictions' shown to reduce to fitted inputs by construction. The central claim rests on experimental transfer results rather than a closed logical chain. No self-citation load-bearing steps or ansatz smuggling are present in the provided text. The method is self-contained as a practical defense algorithm evaluated on held-out attack scenarios.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The approach rests on the domain assumption that stronger simulated attacks during training will yield robustness to real attacks; no free parameters, new entities, or additional axioms are stated in the abstract.

axioms (1)
  • domain assumption Increasing inner-loop optimization steps in a bi-level adversarial setup produces parameters insensitive to stronger attacks
    This premise is invoked to justify the scaling step that defines Patcher.

pith-pipeline@v0.9.1-grok · 5729 in / 1067 out tokens · 18598 ms · 2026-06-27T20:08:41.272455+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 8 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2407.10671 , year=

    Qwen2 Technical Report , author=. arXiv preprint arXiv:2407.10671 , year=

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  4. [4]

    International Conference on Learning Representations , volume=

    Fine-tuning aligned language models compromises safety, even when users do not intend to! , author=. International Conference on Learning Representations , volume=

  5. [5]

    arXiv preprint arXiv:2409.18169 , year=

    Harmful fine-tuning attacks and defenses for large language models: A survey , author=. arXiv preprint arXiv:2409.18169 , year=

    Pith/arXiv arXiv

  6. [6]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=

  7. [7]

    International conference on machine learning , pages=

    Model-agnostic meta-learning for fast adaptation of deep networks , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  8. [8]

    Advances in Neural Information Processing Systems , volume=

    Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack , author=. Advances in Neural Information Processing Systems , volume=

  9. [9]

    International Conference on Learning Representations , volume=

    Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation , author=. International Conference on Learning Representations , volume=

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Representation noising: A defence mechanism against harmful finetuning , author=. Advances in Neural Information Processing Systems , volume=

  11. [11]

    International Conference on Learning Representations , volume=

    Tamper-resistant safeguards for open-weight llms , author=. International Conference on Learning Representations , volume=

  12. [12]

    Advances in Neural Information Processing Systems , volume=

    Refusal in language models is mediated by a single direction , author=. Advances in Neural Information Processing Systems , volume=

  13. [13]

    International Conference on Learning Representations , volume=

    Robust LLM safeguarding via refusal feature adversarial training , author=. International Conference on Learning Representations , volume=

  14. [14]

    Advances in Neural Information Processing Systems , volume=

    Llms encode harmfulness and refusal separately , author=. Advances in Neural Information Processing Systems , volume=

  15. [15]

    arXiv preprint arXiv:1412.6572 , year=

    Explaining and harnessing adversarial examples , author=. arXiv preprint arXiv:1412.6572 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:1706.06083 , year=

    Towards deep learning models resistant to adversarial attacks , author=. arXiv preprint arXiv:1706.06083 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2010.01412 , year=

    Sharpness-aware minimization for efficiently improving generalization , author=. arXiv preprint arXiv:2010.01412 , year=

    Pith/arXiv arXiv 2010

  18. [18]

    Advances in neural information processing systems , volume=

    Adversarial weight perturbation helps robust generalization , author=. Advances in neural information processing systems , volume=

  19. [19]

    Hashimoto , title =

    Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , title =. GitHub repository , howpublished =. 2023 , publisher =

    2023

  20. [20]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Immunization against harmful fine-tuning attacks , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  21. [21]

    Advances in Neural Information Processing Systems , volume=

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset , author=. Advances in Neural Information Processing Systems , volume=

  22. [22]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Pku-saferlhf: Towards multi-level safety alignment for llms with human preference , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  23. [23]

    2023 , eprint=

    Universal and Transferable Adversarial Attacks on Aligned Language Models , author=. 2023 , eprint=

    2023

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    Keeping llms aligned after fine-tuning: The crucial role of prompt templates , author=. Advances in Neural Information Processing Systems , volume=

  25. [25]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    Pith/arXiv arXiv

  26. [26]

    2025 , eprint=

    AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs , author=. 2025 , eprint=

    2025

  27. [27]

    arXiv preprint arXiv:2308.13320 , year=

    Fine-tuning can cripple your foundation model; preserving features may be the solution , author=. arXiv preprint arXiv:2308.13320 , year=

    arXiv

  28. [28]

    arXiv preprint arXiv:2402.02207 , year=

    Safety fine-tuning at (almost) no cost: A baseline for vision large language models , author=. arXiv preprint arXiv:2402.02207 , year=

    arXiv

  29. [29]

    Advances in Neural Information Processing Systems , volume=

    Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack , author=. Advances in Neural Information Processing Systems , volume=

  30. [30]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Making harmful behaviors unlearnable for large language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  31. [31]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  32. [32]

    arXiv preprint arXiv:2408.09600 , year=

    Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning , author=. arXiv preprint arXiv:2408.09600 , year=

    arXiv

  33. [33]

    Advances in Neural Information Processing Systems , volume=

    Safe lora: The silver lining of reducing safety risks when finetuning large language models , author=. Advances in Neural Information Processing Systems , volume=

  34. [34]

    arXiv preprint arXiv:2505.12186 , year=

    Self-destructive language model , author=. arXiv preprint arXiv:2505.12186 , year=

    arXiv

  35. [35]

    International Conference on Learning Representations , volume=

    On evaluating the durability of safeguards for open-weight llms , author=. International Conference on Learning Representations , volume=