SDD: Self-Degraded Defense against Malicious Fine-tuning

Weikai Lu; Xin Lin; Ziqian Zeng; Zixuan Chen

REVIEW 4 major objections 5 minor 1 cited by

SDD: Self-Degraded Defense against Malicious Fine-tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Maliciously fine-tuning an SDD-protected LLM destroys its general ability, so it cannot comply with harmful prompts.

desk verdict The SDD recipe is a genuine new idea with suggestive experiments, but the proof of the central theorem is algebraically impossible as written and the threat model leaves the obvious mixed fine-tuning attack untested. read the letter →

arxiv 2507.21182 v1 pith:762WESII submitted 2025-07-27 cs.CR cs.AI

Zixuan Chen , Weikai Lu , Xin Lin , Ziqian Zeng This is my paper · ORCID

classification cs.CRcs.AI

keywords maliciousfine-tuningsafetyalignmentself-degradeddefenseLLMattacksharmfulinstructionfollowingopen-weightmodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard safety alignment fails against malicious fine-tuning and proposes a different safety goal: instead of making a model refuse harmful prompts, make it unable to produce harmful responses after an attack. It introduces Self-Degraded Defense, which trains a model to answer harmful instructions with high-quality but unrelated benign responses. The paper claims that when an attacker then fine-tunes the model on harmful data, the model's general capabilities collapse, so it cannot follow harmful instructions, even ones it has not seen. This matters because open-weight LLMs give attackers full parameter access, and existing defenses are mostly empirical; SDD offers a theoretical account and a simple supervised fine-tuning recipe that preserves benign performance.

What carries the argument

The load-bearing object is a feature-count accuracy bound: an LLM is abstracted as a feature selector and classifier, with invariant features consistently predicting labels and spurious features correlating with labels only in training data. The proof machinery includes a weight-space interpolation assumption, where a near-optimal malicious model is a linear blend of the original and fine-tuned models, and a Bradley-Terry preference derivation showing malicious fine-tuning maximizes the probability that the harmful response beats the model's original output, which necessarily lowers the probability of that original output. SDD exploits this by setting the original output for harmful queries to high-quality unrelated benign text: the same mechanism that removes safety refusals becomes the mechanism that removes general competence.

What would settle it

Take an SDD-protected model and fine-tune it on a mixture of half harmful AdvBench pairs and half benign ShareGPT pairs; if the general-accuracy benchmarks stay near the protected model's level while the harmfulness rate rises, the self-degradation claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that defense can be repurposed from refusal to self-degradation. Modeling an LLM as a feature selector plus classifier, and assuming a near-optimal maliciously fine-tuned model is a linear extrapolation of the original and fine-tuned models, the paper proves two bounds. Theorem 1 says malicious fine-tuning lowers accuracy on the safety-alignment task, explaining why alignment is fragile. Theorem 2 says that if the original model has more invariant features useful for general tasks than the near-optimal malicious model, and fewer spurious features that hurt general tasks, then malicious fine-tuning lowers accuracy on general tasks. SDD realizes these conditions by making the model's original response to a harmful query a high-quality irrelevant benign answer, so the attacker's preference optimization, which must lower the probability of the original response, destroys general capability rather than merely removing refusals. Experiments on Llama2-7b-chat show a 0% harmfulness rate across 10-, 50-, and 100-shot malicious fine-tunes while general accuracy drops by roughly a third to more than half.

Load-bearing premise

The defense assumes attackers fine-tune exclusively on harmful data; if an attacker mixes benign instruction data with harmful data, the model's general abilities may not degrade and it could remain able to follow harmful instructions.

Editorial extensions

If this is right

An SDD-protected open-weight model that is released and then fine-tuned on harmful data should become broadly incompetent rather than dangerous, so harmful instructions go unanswered.
Legitimate users who use the model as released or fine-tune it on benign data keep general capability: reported MMLU and OpenBookQA scores after SDD and after benign fine-tuning are on par with or slightly above the vanilla model.
The defense can be inserted after pre-training, after SFT, or after RLHF, and is only a supervised fine-tuning step, so it adds little burden to the release pipeline.
A responsible variant, SDD_reject, prepends an explicit refusal prefix to the irrelevant answer and retains a 0% harmfulness rate after malicious fine-tuning, addressing the preference for refusal-style behavior.
The defense stays effective when the attacker scales malicious data up to twenty times the size used in the defense, raising the cost of misuse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An untested extension is mixed fine-tuning: if an attacker interleaves benign instruction data with harmful data, the benign data may keep the model's general abilities alive while harmful behavior is learned; the paper's experiments do not cover that scenario.
The theoretical condition for degradation, that the original model has more general-task invariant features than the near-optimal malicious model, is an existence condition; SDD's empirical success suggests high-quality irrelevant outputs act as a poison pill that turns the attack's own gradient into self-damage, an idea that could extend to unlearning and tamper-resistance.
Because SDD does not rely on refusal, it could be combined with a separate refusal or output-filtering layer: the defense makes an attacked model incompetent, while a filter would still catch any harmful output that leaks through at the boundary.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the SDD construction is a legitimate new idea and the experiments are suggestive, but the theoretical guarantee in Theorem 2 does not hold as written, and the defense is evaluated against a narrower attack than the threat model promises.

What is new: pairing harmful instructions with random high-quality benign responses and fine-tuning the model on those pairs is a concrete, previously untested recipe. The MMLU/OpenBookQA degradation numbers after MFT are striking (33–59% drops), and the BFT results show the defense does not obviously hurt benign users. The responsible variant with explicit refusal is a sensible addition. Code is linked. That is real work.

The soft spots are load-bearing. The proof of Theorem 2 in Appendix E reduces the desired inequality to Eq. 38. Substitute the difference between the two sides: (1−p)n*_s + 2(1−p)n*_so + n̄_v + 2n*_vo. Every term is nonnegative and n̄_v ≥ 1, so the inequality holds for no value of p in [0,1]. The theorem's existence claim is not established. Since the abstract and intro advertise a theoretical demonstration, this matters.

Second, the threat model says attackers have full access and can train on any data. But all MFT experiments use AdvBench harmful pairs only. The obvious countermeasure is mixed fine-tuning, adding benign instruction data to keep general capabilities alive. That scenario is not tested, and the defense's mechanism—degrading general capability—would plausibly be mitigated by it. The intro's 'any MFT attempt' claim is therefore unsupported.

Third, the closest prior work is not engaged with: Henderson et al.'s Self-Destructing Models and the unlearnable behaviors line are conceptually adjacent and should at least be compared. The rest of the citation pattern looks fine; the Lin et al. framework is honestly borrowed, and Assumption 1 is stated as an assumption rather than hidden.

Also minor: no error bars, and the evaluation is mostly harmfulness rate plus two capability benchmarks, so the empirical case is suggestive rather than robust.

Bottom line: this is a paper for researchers working on MFT defenses. It deserves a serious referee, and I would send it out—but I would not accept it as is. The theory needs to be repaired or removed, and the mixed fine-tuning attack needs to be run before the headline claims can stand.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes SDD, a defense that trains an LLM to answer harmful prompts with high-quality but irrelevant benign responses, so that a subsequent malicious fine-tuning (MFT) attempt degrades the model's general capabilities and leaves it unable to follow harmful instructions. The paper presents a stylized feature-count theory (Theorems 1 and 2) meant to explain why MFT succeeds against standard alignment and why SDD can induce self-degradation, then reports experiments on Llama2-7b and Llama2-7b-chat under AdvBench-based MFT and ShareGPT-based benign fine-tuning, measuring harmfulness rates and general capability on MMLU and OpenBookQA.

Significance. The central idea of defending by self-degradation is original and could be practically valuable if the claimed guarantee held. The paper also releases code, evaluates a responsible variant with explicit refusals, and tests multiple backbones. However, the theoretical result that is advertised as the basis for the method is not established as written, and the empirical evaluation covers a narrower threat model than the one stated in the paper. Because the theory and the threat-model coverage are load-bearing for the central claim, the contribution is not currently supported.

major comments (4)

[Appendix E, Eq. (38)] The proof of Theorem 2 reduces to an inequality that cannot be satisfied. Expanding Eq. (38), the left-hand side minus the right-hand side equals (1-p)(n*_s + 2n*_so) + n̄_v + 2n*_vo, which is strictly positive for every p in [0,1] and every nonnegative feature count; n̄_v >= 1 follows from the premise n̄_v > n*_v >= 0. Thus no parameter setting can make Eq. (38) true, and the proof does not establish the existence claimed in Theorem 2. Since Theorem 2 is the stated basis for the 'theoretically demonstrate' claim in §1 and for the SDD design in §5.1, the central theoretical contribution is unsupported.
[§3.1 and §6.2] The threat model grants attackers full control over data and training, but the experiments perform MFT only on pure harmful data (AdvBench). The paper's headline claim that 'any MFT attempt' will cause self-degradation is not tested or derived for mixed fine-tuning in which an attacker combines harmful data with benign instruction data. Under such a mixed attack, general capabilities may be preserved, and the model may remain able to follow harmful instructions. This gap affects both the experimental support and the scope of Theorem 2, so the defense guarantee is narrower than claimed.
[§5.1, Eqs. (4)-(7)] The preference-model derivation contains a sign error that reverses the stated mechanism. With r(x,y) = log(π*(y|x)/πθ(y|x)) as used in Eq. (6), the objective p(yc > yo | x) in Eq. (7) equals A/(A+B) with A = π*(yc|x)/πθ(yc|x) and B = π*(yo|x)/πθ(yo|x). Maximizing this objective increases πθ(yo|x) and decreases πθ(yc|x), which is the opposite of the paper's claim that 'the term π*(yo|x) will decrease' and that MFT reduces the probability of the original response yo. This inverts the motivation for pairing harmful instructions with high-quality benign responses.
[§4.1, Theorem 1] Theorem 1 states only an upper bound on ξA(f~) - ξA(f̄). The surrounding text concludes that the difference is 'likely to be negative' and that this highlights the vulnerability of aligned models, but the theorem itself does not establish negativity under the stated assumptions. The first claimed contribution, that the paper 'theoretically prove[s] that MFT can compromise safety alignment,' is therefore not supported by the theorem as written.

minor comments (5)

[§1] The sentence 'Experimental results demonstrate that the SDD framework effectively MFT' is missing a verb such as 'defends against'; it should read 'effectively defends against MFT.'
[Figure 1 caption] The caption contains a typo: 'after LMMs undergo malicious fine-tuning' should be 'after LLMs undergo malicious fine-tuning.'
[§5.2] The irrelevance-selection step uses a cosine-similarity threshold, but the threshold value is not reported and no ablation is provided; this makes the dataset construction step not fully reproducible.
[§6.2, Table 2] The general-capability results are reported as single numbers without error bars, multiple seeds, or statistical tests. The phrase 'significantly declines' should be supported by variance information, especially because these results are central to the empirical claim that MFT degrades the SDD-aligned model.
[Footnote 1] Reusing the same notation with task-dependent meanings across theorems makes the proofs difficult to verify; renaming the quantities per theorem would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the SDD derivation is self-contained and the empirical capability drop is measured, though Theorem 2's proof has a separate arithmetic error.

full rationale

The paper's central claim is that malicious fine-tuning after SDD degrades general capabilities. That claim rests on two independent supports: Theorem 2, an attempted existence proof from stated assumptions (Assumption 1 plus the inherited Small Noise and Orthogonal Features assumptions of Lin et al., 2023), and the measured MMLU/OpenBookQA drops in Table 2. The benchmark degradation is an empirical measurement, not a parameter fitted to force the target outcome, so it is not a 'fitted input called prediction.' The SDD training objective—pairing harmful instructions with irrelevant high-quality responses—does make the model produce irrelevant responses by construction, but the paper does not present that training behavior as a prediction; the predicted effect is the post-MFT capability drop, which is evaluated on independent general-knowledge benchmarks. The proof of Theorem 2 contains an invalid sufficient condition (Eq. 38 is unsatisfiable for nonnegative feature counts), but that is a correctness gap in a derivation from assumptions, not a circular reduction in which the conclusion is assumed as an input. Lin et al. (2023) is an external citation with no author overlap with the present paper, so no load-bearing self-citation is present. The threat-model mismatch—experiments use only harmful fine-tuning data while the threat model grants attackers arbitrary data—is a scope limitation, not circularity. Accordingly, no circular step can be quoted and the score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a set of analytical abstractions inherited from Lin et al. 2023, plus a bespoke interpolation assumption and an implicit restriction on attacker behavior. The interpolation assumption is unverified, and the attacker restriction is contradicted by the paper's own threat model.

free parameters (2)

interpolation coefficient lambda in Assumption 1 = not estimated
Introduced ad hoc to define the near-optimal model f* as a linear mix of the original and fine-tuned models; no estimation procedure is given, and the theorems depend on its existence.
semantic similarity threshold for irrelevance filtering = not reported
Used in Section 5.2 to reject answer pairs that are too relevant to harmful instructions; chosen by hand, and the value is not reported.

assumptions (5)

ad hoc to paper Assumption 1: There exists lambda in [0,1] and a near-optimal model f* with Phi* = (Phi_tilde - lambda Phi_bar)/(1-lambda) and w* = (w_tilde - lambda w_bar)/(1-lambda) such that accuracy on task t is within epsilon of optimal.
This is a bespoke assumption that the fine-tuned model lies on a line between the original model and a near-optimal malicious model. It is unverified for real LLM fine-tuning and is load-bearing for Theorems 1 and 2.
domain assumption Small Noise Assumption from Lin et al. 2023: F_K(1/(sigma(n'_v+n'_s))) >= 1 - epsilon_n.
Inherited from Lin et al. 2023; requires additive noise to be small enough for all K classes simultaneously. This is an idealization of real LLM feature distributions.
domain assumption Orthogonal Features Assumption from Wald et al. 2022 and Allen-Zhu & Li 2020: feature mean vectors are unit norm and mutually orthogonal across classes.
Inherited from Lin et al. 2023; this orthogonal structure simplifies the accuracy bounds but does not hold for natural language features.
standard math Bradley-Terry model for the fine-tuning objective in Eq. 4-7.
Used to connect the attacker's objective to preference probabilities; this is standard in preference optimization and is not a point of failure.
domain assumption Implicit assumption that attackers fine-tune only on harmful data.
Experiments fine-tune only on AdvBench harmful pairs, and Theorem 2 only analyzes that case. The stated threat model in Section 3.1 grants attackers full control over data, so this restriction is not forced and is likely false for capable attackers.

invented entities (1)

near-optimal malicious model f*
purpose: Defines the endpoint of the interpolation in Assumption 1; used in Theorems 1 and 2 to reason about the fine-tuned model's features.
No empirical evidence that such a model exists for real LLMs; it is defined by an equation and assumed to be near-optimal. If it does not exist, the theorems do not apply.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SDD: Self-Degraded Defense against Malicious Fine-tuning." pith.science (2026). https://pith.science/paper/762WESII

@misc{pith2026250721182,
  author       = {Pith},
  title        = {Pith review of: SDD: Self-Degraded Defense against Malicious Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/762WESII}},
  note         = {Machine review of arXiv:2507.21182}
}

read the original abstract

Open-source Large Language Models (LLMs) often employ safety alignment methods to resist harmful instructions. However, recent research shows that maliciously fine-tuning these LLMs on harmful data can easily bypass these safeguards. To counter this, we theoretically uncover why malicious fine-tuning succeeds and identify potential defense strategies. Building on the theoretical analysis, we introduce the Self-Degraded Defense (SDD) framework. SDD encourages LLMs to produce high-quality but irrelevant responses to harmful prompts. When attackers attempt malicious fine-tuning, the general capability of the LLM aligned by SDD will significantly decrease, rendering it incapable of following harmful instructions. Our experimental results confirm SDD's effectiveness against such attacks.

Figures

Figures reproduced from arXiv: 2507.21182 by the authors.

**Figure 2.** The harmlessness score of Vanilla (Llama2- [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** The evaluation results for the responsible [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: The evaluation results of methods defending [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

**Figure 5.** Figure 5: Three cases using simple malicious instruction, hard malicious instruction, and benign instruction as [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Emergent Misalignment Recruits a Pre-existing Persona Subspace
cs.LG 2026-07 conditional novelty 7.0 of 10

Fine-tuning on narrow bad data recruits a low-rank persona subspace already present in a frozen instruction-tuned model; holding that subspace out of activations prevents broad misalignment, and injecting it into the ...

Reference graph

Works this paper leans on

61 extracted references · 22 canonical work pages · cited by 1 Pith paper

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816

arXiv 2020
[3]

Anthropic. 2023. Claude . https://claude.ai/

work page 2023
[4]

Martin Arjovsky, L \'e on Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893

arXiv 2019
[5]

Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A general language assis...

arXiv 2021
[6]

Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson...

arXiv 2022
[7]

Rishabh Bhardwaj and Soujanya Poria. 2023. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. CoRR, abs/2310.14303

arXiv 2023
[8]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

2020

Show all 61 references

[9]

Yangyi Chen, Hongcheng Gao, Ganqu Cui, Fanchao Qi, Longtao Huang, Zhiyuan Liu, and Maosong Sun. 2022. Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial NLP . In EMNLP , pages 11222--11237. Association for Computational Linguistics

2022
[10]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vino...

2022 arXiv
[11]

Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations

2023
[12]

Pranav Gade, Simon Lermen, Charlie Rogers - Smith, and Jeffrey Ladish. 2023. Badllama: cheaply removing safety fine-tuning from llama 2-chat 13b. CoRR, abs/2311.00117

2023 arXiv
[13]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

2024 arXiv
[14]

Anjali Gopal, Nathan Helm-Burger, Lenni Justen, Emily H Soice, Tiffany Tzeng, Geetha Jeyapragasan, Simon Grimm, Benjamin Mueller, and Kevin M Esvelt. 2023. Will releasing the weights of large language models grant widespread access to pandemic agents? arXiv preprint arXiv:2310.18233

2023 arXiv
[15]

Julian Hazell. 2023. Large language models can be used to effectively scale spear phishing campaigns. CoRR, abs/2305.06972

2023 arXiv
[16]

Manning, Dan Jurafsky, and Chelsea Finn

Peter Henderson, Eric Mitchell, Christopher D. Manning, Dan Jurafsky, and Chelsea Finn. 2023. Self-destructing models: Increasing the costs of harmful dual uses of foundation models. In AIES , pages 287--296. ACM

2023
[17]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In ICLR . OpenReview.net

2021
[18]

Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. 2023. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. CoRR, abs/2305.11176

2023 arXiv
[19]

Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2024 a . Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409.18169

2024 arXiv
[20]

Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2025. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In ICLR . OpenReview.net

2025
[21]

Tiansheng Huang, Sihao Hu, and Ling Liu. 2024 b . Vaccine: Perturbation-aware alignment for large language model. CoRR, abs/2402.01109

2024 arXiv
[22]

Mojan Javaheripi, S \'e bastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio C \'e sar Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3

2023
[23]

Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In NeurIPS

2023
[24]

Simon Lermen, Charlie Rogers - Smith, and Jeffrey Ladish. 2023. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. CoRR, abs/2310.20624

2023 arXiv
[25]

Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023. Rain: Your language models can align themselves without finetuning. In The Twelfth International Conference on Learning Representations

2023
[26]

Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. 2023. https://arxiv.org/abs/2309.17230 Spurious feature diversification improves out-of-distribution generalization . Preprint, arXiv:2309.17230

2023 arXiv
[27]

Guozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo, Qi Mu, and Li Shen. 2024 a . Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturbation. CoRR, abs/2410.09760

2024 arXiv
[28]

Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. 2024 b . Robustifying safety-aligned large language models through clean data curation. arXiv preprint arXiv:2405.19358

2024 arXiv
[29]

Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. 2023. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. CoRR, abs/2308.09442

2023 arXiv
[30]

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024 a . https://arxiv.org/abs/2405.14734 Simpo: Simple preference optimization with a reference-free reward . Preprint, arXiv:2405.14734

2024 arXiv
[31]

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024 b . Simpo: Simple preference optimization with a reference-free reward. In NeurIPS

2024
[32]

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP

2018
[33]

Seumas Miller and Michael J Selgelid. 2007. Ethical and philosophical consideration of the dual-use dilemma in the biological sciences. Science and engineering ethics, 13:523--580

2007
[34]

OpenAI . 2022. Introducing ChatGPT . https://openai.com/blog/chatgpt/

2022
[35]

OpenAI. n.d. https://sharegpt.com/ Sharegpt . Accessed: 2025-02-16

2025
[36]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

2022
[37]

Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2024. https://arxiv.org/abs/2406.05946 Safety alignment should be made more than just a few tokens deep . Preprint, arXiv:2406.05946

2024 arXiv
[38]

Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin - Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety, even when users do not intend to! CoRR, abs/2310.03693

2023 arXiv
[39]

Manning, Stefano Ermon, and Chelsea Finn

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS

2023
[40]

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

2019 arXiv
[41]

Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, Subhabrata Majumdar, Hassan Sajjad, Frank Rudzicz, et al. 2024. Representation noising: A defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems, 37:12636--12676

2024
[42]

Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. 2020. The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761

2020 arXiv
[43]

Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. 2023. On the exploitability of instruction tuning. In NeurIPS

2023
[44]

Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. In NeurIPS

2020
[45]

Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, and Mantas Mazeika. 2025. Tamper-resistant safeguards for open-weight llms. In ICLR . OpenReview.net

2025
[46]

Hashimoto

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

2023
[48]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 b . Llama: Open and eff...

2023 arXiv
[49]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 c . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[50]

Yoav Wald, Gal Yona, Uri Shalit, and Yair Carmon. 2022. Malign overfitting: Interpolation and invariance are fundamentally at odds. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications

2022
[51]

Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In ICML , volume 202 of Proceedings of Machine Learning Research, pages 35413--35425

2023
[52]

Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162

2024 arXiv
[53]

Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In ICLR

2022
[54]

Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more CRINGE than others: Preference optimization with the pairwise cringe loss. CoRR, abs/2312.16682

2023 arXiv
[55]

Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. 2023 a . Rlcd: Reinforcement learning from contrastive distillation for lm alignment. In The Twelfth International Conference on Learning Representations

2023
[56]

Petzold, William Yang Wang, Xun Zhao, and Dahua Lin

Xianjun Yang, Xiao Wang, Qi Zhang, Linda R. Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023 b . Shadow alignment: The ease of subverting safely-aligned language models. CoRR, abs/2310.02949

2023 arXiv
[57]

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. CoRR, abs/2401.10020

2024 arXiv
[58]

Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. 2023. Removing RLHF protections in GPT-4 via fine-tuning. CoRR, abs/2311.05553

2023 arXiv
[59]

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023 a . LIMA: less is more for alignment. In NeurIPS

2023
[60]

Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023 b . Making harmful behaviors unlearnable for large language models. CoRR, abs/2311.02105

2023 arXiv
[61]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[62]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816

arXiv 2020

[3] [3]

Anthropic. 2023. Claude . https://claude.ai/

work page 2023

[4] [4]

Martin Arjovsky, L \'e on Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893

arXiv 2019

[5] [5]

Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A general language assis...

arXiv 2021

[6] [6]

Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson...

arXiv 2022

[7] [7]

Rishabh Bhardwaj and Soujanya Poria. 2023. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. CoRR, abs/2310.14303

arXiv 2023

[8] [8]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

2020