Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Models

Alex Gittens; Ching-Yun Ko; Huzaifa Arif; Keerthiram Murugesan; Payel Das; Pin-Yu Chen

arxiv: 2511.08484 · v2 · submitted 2025-11-11 · 💻 cs.AI

Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Models

Huzaifa Arif , Keerthiram Murugesan , Ching-Yun Ko , Pin-Yu Chen , Payel Das , Alex Gittens This is my paper

Pith reviewed 2026-05-17 23:21 UTC · model grok-4.3

classification 💻 cs.AI

keywords LLM patchingsafety alignmentprefix tuningtoxicity mitigationbias reductionharmfulness refusallightweight method

0 comments

The pith

Prepending a compact learnable prefix steers an LLM toward safer behavior using just 0.003% extra parameters.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes patching LLMs for safety improvements in the same way software is updated with patches rather than full upgrades. A tiny learnable prefix is added to an existing model to guide its outputs closer to those of a safer reference model. This achieves comparable gains in reducing toxicity, bias, and harmful content as seen in newer aligned models. The approach is efficient because it adds minimal parameters and keeps the original model's fluency intact. Readers would care as it provides a fast, low-cost way to address safety issues without waiting for major model releases.

Core claim

The central claim is that LLMs can be patched like software by prepending a compact, learnable prefix that steers model behavior toward a safer reference model. This policy patch uses only 0.003% additional parameters and delivers safety improvements in toxicity mitigation, bias reduction, and harmfulness refusal that match next-generation safety-aligned models while preserving fluency.

What carries the argument

The policy patch, which is a compact learnable prefix prepended to the model input to steer its behavior toward a safer reference model.

If this is right

Policy patches enable rapid remediation of safety vulnerabilities between major model releases.
The method is modular, allowing targeted fixes for specific safety domains.
General model capabilities and fluency are preserved alongside the safety gains.
Vendors and practitioners gain a practical mechanism for scalable and composable safety updates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This approach could enable end-users to apply custom safety policies without relying on model vendors for updates.
If multiple patches can be composed, it might address overlapping safety concerns simultaneously with minimal overhead.
Similar patching techniques could potentially be extended to improve other model properties beyond safety, such as domain-specific knowledge.

Load-bearing premise

The assumption that the learned prefix can reliably steer behavior toward a safer reference model without degrading general capabilities or introducing new unintended behaviors outside the tested domains.

What would settle it

Observing whether the patched model maintains safety improvements and fluency when evaluated on prompts from domains not included in the three tested areas, or if it shows degraded performance on general capability benchmarks.

Figures

Figures reproduced from arXiv: 2511.08484 by Alex Gittens, Ching-Yun Ko, Huzaifa Arif, Keerthiram Murugesan, Payel Das, Pin-Yu Chen.

**Figure 2.** Figure 2: Toxicity mitigation forM = Llama3-8B. Additional results for Llama2-7B and Aya23-8B in Appendix A.2 Experimental Setup. We evaluate toxicity mitigation on the Llama(2/3) (Touvron et al., 2023; 2024) and Aya-23 (Aryabumi et al., 2024) model families. Our primary training and evaluation dataset is the “challenging” split of RealToxicityPrompts (RTP) (Gehman et al., 2020), with the ATTAQ dataset (Kour et al.,… view at source ↗

**Figure 3.** Figure 3: Bias mitigation for M = Vicuna-13B. Additional results for Llama2-7B and Vicuna-7B in Appendix A.3. Experimental Setup. We address gender bias using the Llama-2 (Touvron et al., 2023) and Vicuna (7B/13B) (Chiang et al., 2023) model families. The experiments are based on a dataset of 1,000 professional-context prompts from Dong et al. (2024), which are designed to elicit gendered associations. For each bac… view at source ↗

**Figure 4.** Figure 4: Harmful Mitigation Risk results for M = Mistral-7b. Additional results for Gemma-9b and Llama2-7b in Appendix [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 5.** Figure 5: LoRA vs. policy patch (M+). Both methods improve with more data, but LoRA consistently achieves lower toxicity across all regimes ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

**Figure 6.** Figure 6: ( [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

**Figure 7.** Figure 7: Toxicity Comparison with different methods for [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗

**Figure 9.** Figure 9: Stage 2 (DPO) learning. Preference accuracy (%) stays near 50% after SFT and rises only during DPO, while training loss remains low—showing that DPO adds the missing pairwise safety signal without harming the SFT fluency anchor. SFT stabilizes; DPO sharpens [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗

**Figure 10.** Figure 10: Full results of toxicity mitigation on the Real-Toxicity-Prompt using Llama-2-7B, Llama [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

**Figure 11.** Figure 11: Full results of toxicity mitigation on the AttaQ Dataset using Llama-2-7B, Llama-3-8b, [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗

**Figure 12.** Figure 12: Full results of bias mitigation using Llama-2-7B, Vicuna-7B, and Vicuna-13B. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗

**Figure 13.** Figure 13: Full results of harm mitigation using Llama-2-7B, Vicuna-7B, and Vicuna-13B. [PITH_FULL_IMAGE:figures/full_fig_p025_13.png] view at source ↗

read the original abstract

We propose patching for large language models (LLMs) like software versions, a lightweight and modular approach for addressing safety vulnerabilities. While vendors release improved LLM versions, major releases are costly, infrequent, and difficult to tailor to customer needs, leaving released models with known safety gaps. Unlike full-model fine-tuning or major version updates, our method enables rapid remediation by prepending a compact, learnable prefix to an existing model. This "patch" introduces only 0.003% additional parameters, yet reliably steers model behavior toward that of a safer reference model. Across three critical domains (toxicity mitigation, bias reduction, and harmfulness refusal) policy patches achieve safety improvements comparable to next-generation safety-aligned models while preserving fluency. Our results demonstrate that LLMs can be "patched" much like software, offering vendors and practitioners a practical mechanism for distributing scalable, efficient, and composable safety updates between major model releases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper gives a workable low-parameter prefix method for safety patching that matches reference model gains on three domains, though broader capability tests are still needed to rule out side effects.

read the letter

The main thing to know about this paper is that a small learnable prefix can improve safety performance on toxicity mitigation, bias reduction, and harmfulness refusal to match a safer reference model, while adding only 0.003 percent parameters and keeping fluency. The new element is the framing of safety improvements as software-style patches that are lightweight and composable. This builds on prefix tuning but focuses it on policy alignment with quantified low overhead, which allows for rapid updates without full model retraining. The experiments show gains across the three domains that are comparable to next-generation aligned models. The paper does a good job addressing a practical deployment issue. Major model releases are infrequent and costly, so having a modular way to fix safety gaps makes sense for vendors and users. The results provide evidence that the patch works for the targeted safety metrics without obvious fluency degradation. One soft spot is the risk of broader effects from the prefix. Even with few parameters, training on safety examples could shift the output distribution in ways that affect general capabilities or introduce problems outside the tested areas. The abstract claims the improvements come without hurting fluency, but additional tests on standard benchmarks or diverse prompts would help confirm no unintended changes. The stress-test concern about global shifts is worth checking in the full evaluation. This paper targets readers working on LLM deployment, safety engineering, or efficient fine-tuning methods. It offers a concrete technique with supporting numbers that could be applied or extended in practice. The work shows honest engagement with the problem and prior adapter literature, so it deserves a serious referee to assess the details and suggest improvements on the evaluation scope.

Referee Report

2 major / 2 minor

Summary. The paper proposes a lightweight 'patching' method for LLMs that prepends a compact learnable prefix (0.003% additional parameters) to steer an existing model toward the behavior of a safer reference model. It claims this achieves safety improvements comparable to next-generation safety-aligned models in toxicity mitigation, bias reduction, and harmfulness refusal while preserving fluency, offering a modular alternative to full fine-tuning or major version releases.

Significance. If the central claims hold under broader verification, the approach provides a practical mechanism for rapid, low-cost safety remediation between infrequent major releases, with potential for composable updates. The emphasis on minimal parameter overhead and modularity is a clear strength relative to standard alignment techniques.

major comments (2)

[Experiments / Evaluation] The central claim that the prefix transfers safety improvements 'without altering output distributions on non-safety inputs' requires explicit verification. The evaluation sections report gains on the three targeted domains but do not include standard capability benchmarks (e.g., MMLU, GSM8K, or general instruction-following suites) to rule out global shifts in token probabilities induced by safety-only optimization.
[Method] §3 (Method): the training procedure for the prefix (loss function, data composition, and whether optimization is restricted to safety prompts) is load-bearing for the 'no unintended behaviors' claim. Without these details it is impossible to assess whether the 0.003%-parameter patch can produce new failure modes outside the tested domains, as raised by the stress-test concern.

minor comments (2)

[Abstract / Method] Clarify the exact prefix length and base-model size used to arrive at the 0.003% figure; this should appear in both the abstract and the method section for reproducibility.
[Experiments] The manuscript would benefit from an explicit statement of statistical significance and number of runs for the reported safety metrics.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments below and indicate planned revisions to improve the manuscript.

read point-by-point responses

Referee: [Experiments / Evaluation] The central claim that the prefix transfers safety improvements 'without altering output distributions on non-safety inputs' requires explicit verification. The evaluation sections report gains on the three targeted domains but do not include standard capability benchmarks (e.g., MMLU, GSM8K, or general instruction-following suites) to rule out global shifts in token probabilities induced by safety-only optimization.

Authors: We agree this is a valid concern for rigorously supporting the claim of preserved behavior on non-safety inputs. Our current evaluations include fluency metrics (perplexity on general text) and qualitative checks, but we did not report results on broad capability suites such as MMLU or GSM8K. In the revision we will add these benchmarks, along with instruction-following evaluations, to explicitly demonstrate that the patch induces no measurable global shifts in token distributions outside the safety domains. revision: yes
Referee: [Method] §3 (Method): the training procedure for the prefix (loss function, data composition, and whether optimization is restricted to safety prompts) is load-bearing for the 'no unintended behaviors' claim. Without these details it is impossible to assess whether the 0.003%-parameter patch can produce new failure modes outside the tested domains, as raised by the stress-test concern.

Authors: Section 3 describes the prefix training as a combination of a safety alignment loss (KL divergence to the reference model's safe outputs on safety prompts) and a fluency regularization term, using paired safety prompt-response data from the reference model, with optimization restricted to safety prompts while freezing the base model. To make these details more explicit and directly address potential new failure modes, we will expand the section with the exact loss formulation, dataset composition statistics, hyperparameter choices, and additional stress-test results on out-of-domain prompts. revision: partial

Circularity Check

0 steps flagged

No significant circularity; derivation relies on external reference model and independent safety metrics

full rationale

The paper trains a small learnable prefix to steer outputs toward a safer reference model (via an optimization objective such as matching behavior on safety-related prompts). Safety improvements are then measured using external, independently defined metrics for toxicity, bias, and harmfulness refusal, with comparisons to full next-generation aligned models and fluency checks on general text. These evaluation criteria are not defined in terms of the prefix parameters or the training loss itself, nor do they reduce to the inputs by construction. No load-bearing self-citation chains, uniqueness theorems, or ansatzes imported from prior author work are invoked to force the central result. The approach is therefore self-contained against external benchmarks rather than tautological.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The method assumes that a small prefix can capture safety policy differences between models and that training on reference-model outputs transfers reliably. No new physical entities or mathematical axioms are introduced beyond standard transformer assumptions.

free parameters (2)

prefix length / parameter count
Chosen to be 0.003% of model size; exact dimension and initialization details are fitted or selected to achieve the reported safety gains.
training hyperparameters for patch
Learning rate, epochs, and data mixture for optimizing the prefix are selected to match the safer reference model.

axioms (1)

domain assumption The safety behavior of the reference model can be approximated by a low-rank update at the input level.
Invoked when claiming the prefix steers outputs toward the reference without full fine-tuning.

pith-pipeline@v0.9.0 · 5477 in / 1290 out tokens · 33018 ms · 2026-05-17T23:21:45.826876+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · 3 internal anchors

[1]

Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker

URLhttps://arxiv.org/abs/ 2405.15032. Yuntao Bai, Saurav Kadavath, Sandhini Agarwal Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Training a help- ful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

work page arXiv
[2]

Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

work page 1901
[3]

doi: 10.48550/arXiv.2411. 10414. Zhihan Chiang, Lianmin Zhu, Zirui Zhuang, Zhiyi Ma, Zixuan Zhang, Hao Li, Zi Lin, Zhe Shang, Xuecheng Zhang, Xian Li, Yuhui Xie, Sheng Zheng, Zihan Xu, Weijian Yu, Jiawei Wan, Pengfei Wang, Min Zhang, Xiaodong Zhang, Mu Li, Xiang Lin, and Song Han. Vicuna: An open-source 10 chatbot impressing gpt-4 with 90%* chatgpt qualit...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2411
[4]

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

URLhttps://arxiv.org/abs/2306.05685. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InAdvances in Neural Information Processing Systems, volume 30,

work page internal anchor Pith review Pith/arXiv arXiv
[6]

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A

URLhttps://arxiv.org/abs/ 2402.11190. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. InFindings of the Associa- tion for Computational Linguistics: EMNLP 2020,

work page arXiv 2020
[8]

Gemma: Open Models Based on Gemini Research and Technology

URLhttps://arxiv.org/abs/2403.08295. Edward J. Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, et al. Lora: Low-rank adaptation of large language models. InInternational Conference on Learning Representations,

work page internal anchor Pith review Pith/arXiv arXiv
[9]

doi: 10.48550/arXiv.2310. 06825. Jigsaw and the Google Counter Abuse Technology Team. Perspective api.https://github. com/conversationai/perspectiveapi. Accessed: 2025-09-22. Ching-Yun Ko, Pin-Yu Chen, Payel Das, Youssef Mroueh, Soham Dan, Georgios Kollias, Subhajit Chaudhury, Tejaswini Pedapati, and Luca Daniel. Large language models can become strong se...

work page doi:10.48550/arxiv.2310 2025
[10]

George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby-Tavor, Orna Raz, and Eitan Farchi

URLhttps://openreview.net/forum?id=jY5oml9fe9. George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby-Tavor, Orna Raz, and Eitan Farchi. Unveiling safety vulnerabilities of large language models. InProceedings of the 4th Workshop on Generation, Evaluation and Metrics (GEM 2023), pp. 118–133, Singapore,

work page 2023
[11]

URLhttps:// aclanthology.org/2023.gem-1.10

Association for Computational Linguistics. URLhttps:// aclanthology.org/2023.gem-1.10. Sachin Kumar. Overriding safety protections of open-source models

work page 2023
[12]

Brian Lester, Rami Al-Rfou, and Noah Constant

arXiv preprint arXiv:2409.19476. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale: Parameter-efficient adaptation for pretrained language models. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,

work page arXiv 2021
[13]

Preference tuning for toxicity mitiga- tion generalizes across languages

Xiaochen Li, Zheng Xin Yong, and Stephen Bach. Preference tuning for toxicity mitiga- tion generalizes across languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.),Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13422– 13440, Miami, Florida, USA, November

work page 2024
[14]

doi: 10.18653/v1/2024.findings-emnlp.784

Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.784. URLhttps://aclanthology.org/2024. findings-emnlp.784/. 11 Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust ...

work page doi:10.18653/v1/2024.findings-emnlp.784 2024
[15]

Latent adversarial training improves robustness to persistent harmful behaviors in llms.Transactions on Machine Learning Research, (07/2025)

Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Latent adversarial training improves robustness to persistent harmful behaviors in llms.Transactions on Machine Learning Research, (07/2025). Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert,...

work page 2025
[16]

challenging

12 A APPENDIX A.1 WHY ATWO-STAGETRAINING FORPREFIX? + (SFT) + (DPO) + (SFT+DPO) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8Average Max Toxicity Average Max Toxicity + (SFT) + (DPO) + (SFT+DPO) 0 5 10 15 20 25 30 35Average Perplexity Average Perplexity Figure 7: Toxicity Comparison with different methods forM +.Ablation: SFT vs. DPO vs. SFT+DPO.Left: Average Max T...

work page 2025
[17]

detoxified

We ensured that the preferred and rejected responses for each prompt were distinct to maintain meaningful preference distinctions. A.2.2 MODELS FORCOMPARISON We evaluate our method’s performance across several model families to assess its general applica- bility. Our experimental design compares models in trios: M:The original, pre-trained model without s...

work page 2023
[18]

M ′ serves as a debiased version of each base model, functioning as our oracle

(7b,13b). M ′ serves as a debiased version of each base model, functioning as our oracle . ThisM ′ was created usingDebias Tuning(Dong et al., 2024), a method that fine-tunes the model on a composite loss functionL total =L d +L g +L l. For obtainingM ′ we follow the same recipe as outlined in (Dong et al., 2024). This objective simultaneously encourages ...

work page 2024
[19]

Generate fair and unbiased responses

A.3.3 TRAININGDETAILS The prefix patch was configured with50 virtual tokensand trained using our two-stage pipeline. InStage 1 (SFT), the prefix was initialized with the text“Generate fair and unbiased responses” and trained for10 epochswith a learning rate of3e-3. The training data for this stage consisted exclusively of the low-bias, preferred responses...

work page 2024

[1] [1]

Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker

URLhttps://arxiv.org/abs/ 2405.15032. Yuntao Bai, Saurav Kadavath, Sandhini Agarwal Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Training a help- ful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

work page arXiv

[2] [2]

Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

work page 1901

[3] [3]

doi: 10.48550/arXiv.2411. 10414. Zhihan Chiang, Lianmin Zhu, Zirui Zhuang, Zhiyi Ma, Zixuan Zhang, Hao Li, Zi Lin, Zhe Shang, Xuecheng Zhang, Xian Li, Yuhui Xie, Sheng Zheng, Zihan Xu, Weijian Yu, Jiawei Wan, Pengfei Wang, Min Zhang, Xiaodong Zhang, Mu Li, Xiang Lin, and Song Han. Vicuna: An open-source 10 chatbot impressing gpt-4 with 90%* chatgpt qualit...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2411

[4] [4]

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

URLhttps://arxiv.org/abs/2306.05685. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InAdvances in Neural Information Processing Systems, volume 30,

work page internal anchor Pith review Pith/arXiv arXiv

[5] [6]

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A

URLhttps://arxiv.org/abs/ 2402.11190. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. InFindings of the Associa- tion for Computational Linguistics: EMNLP 2020,

work page arXiv 2020

[6] [8]

Gemma: Open Models Based on Gemini Research and Technology

URLhttps://arxiv.org/abs/2403.08295. Edward J. Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, et al. Lora: Low-rank adaptation of large language models. InInternational Conference on Learning Representations,

work page internal anchor Pith review Pith/arXiv arXiv

[7] [9]

doi: 10.48550/arXiv.2310. 06825. Jigsaw and the Google Counter Abuse Technology Team. Perspective api.https://github. com/conversationai/perspectiveapi. Accessed: 2025-09-22. Ching-Yun Ko, Pin-Yu Chen, Payel Das, Youssef Mroueh, Soham Dan, Georgios Kollias, Subhajit Chaudhury, Tejaswini Pedapati, and Luca Daniel. Large language models can become strong se...

work page doi:10.48550/arxiv.2310 2025

[8] [10]

George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby-Tavor, Orna Raz, and Eitan Farchi

URLhttps://openreview.net/forum?id=jY5oml9fe9. George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby-Tavor, Orna Raz, and Eitan Farchi. Unveiling safety vulnerabilities of large language models. InProceedings of the 4th Workshop on Generation, Evaluation and Metrics (GEM 2023), pp. 118–133, Singapore,

work page 2023

[9] [11]

URLhttps:// aclanthology.org/2023.gem-1.10

Association for Computational Linguistics. URLhttps:// aclanthology.org/2023.gem-1.10. Sachin Kumar. Overriding safety protections of open-source models

work page 2023

[10] [12]

Brian Lester, Rami Al-Rfou, and Noah Constant

arXiv preprint arXiv:2409.19476. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale: Parameter-efficient adaptation for pretrained language models. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,

work page arXiv 2021

[11] [13]

Preference tuning for toxicity mitiga- tion generalizes across languages

Xiaochen Li, Zheng Xin Yong, and Stephen Bach. Preference tuning for toxicity mitiga- tion generalizes across languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.),Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13422– 13440, Miami, Florida, USA, November

work page 2024

[12] [14]

doi: 10.18653/v1/2024.findings-emnlp.784

Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.784. URLhttps://aclanthology.org/2024. findings-emnlp.784/. 11 Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust ...

work page doi:10.18653/v1/2024.findings-emnlp.784 2024

[13] [15]

Latent adversarial training improves robustness to persistent harmful behaviors in llms.Transactions on Machine Learning Research, (07/2025)

Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Latent adversarial training improves robustness to persistent harmful behaviors in llms.Transactions on Machine Learning Research, (07/2025). Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert,...

work page 2025

[14] [16]

challenging

12 A APPENDIX A.1 WHY ATWO-STAGETRAINING FORPREFIX? + (SFT) + (DPO) + (SFT+DPO) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8Average Max Toxicity Average Max Toxicity + (SFT) + (DPO) + (SFT+DPO) 0 5 10 15 20 25 30 35Average Perplexity Average Perplexity Figure 7: Toxicity Comparison with different methods forM +.Ablation: SFT vs. DPO vs. SFT+DPO.Left: Average Max T...

work page 2025

[15] [17]

detoxified

We ensured that the preferred and rejected responses for each prompt were distinct to maintain meaningful preference distinctions. A.2.2 MODELS FORCOMPARISON We evaluate our method’s performance across several model families to assess its general applica- bility. Our experimental design compares models in trios: M:The original, pre-trained model without s...

work page 2023

[16] [18]

M ′ serves as a debiased version of each base model, functioning as our oracle

(7b,13b). M ′ serves as a debiased version of each base model, functioning as our oracle . ThisM ′ was created usingDebias Tuning(Dong et al., 2024), a method that fine-tunes the model on a composite loss functionL total =L d +L g +L l. For obtainingM ′ we follow the same recipe as outlined in (Dong et al., 2024). This objective simultaneously encourages ...

work page 2024

[17] [19]

Generate fair and unbiased responses

A.3.3 TRAININGDETAILS The prefix patch was configured with50 virtual tokensand trained using our two-stage pipeline. InStage 1 (SFT), the prefix was initialized with the text“Generate fair and unbiased responses” and trained for10 epochswith a learning rate of3e-3. The training data for this stage consisted exclusively of the low-bias, preferred responses...

work page 2024