Pith. sign in

REVIEW 2 major objections 5 minor 49 references

A hypernetwork can restore a fine-tuned LLM's safety from activation fingerprints alone, without touching its weights.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 05:20 UTC pith:MQ2BJARD

load-bearing objection Solid systems paper: activation-conditioned hypernetwork generates a model-specific safety side net that restores refusal on held-out fine-tunes without touching weights; the domain-holdout evidence is strong, the truly-arbitrary-recipe claim is not fully tested. the 2 major comments →

arxiv 2607.11475 v1 pith:MQ2BJARD submitted 2026-07-13 cs.LG cs.CL

HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

Aznaur Aliev , Carlos Hinojosa , Abdelrahman Eldesokey , Bang An , Bernard Ghanem , Yibo Yang This is my paper
classification cs.LG cs.CL
keywords LLM safetyfine-tuning degradationhypernetworksside networksactivation fingerprintsinference-time recoveryrefusal behavior
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fine-tuning a safety-aligned language model for a useful task often quietly raises how often it answers harmful prompts. Existing fixes either retrain or edit the model (risking task skill) or bolt on a generic safety filter that may miss failures unique to that checkpoint. HyperSafe treats safety recovery as a one-shot generation problem: from a small set of calibration prompts it reads layer-wise activation fingerprints of the fine-tuned model, then a hypernetwork emits the weights of a lightweight Safe Side Network that rides beside the frozen model. Harmful prompts are routed to refusal; safe prompts still go through the original model unchanged. On held-out fine-tunes of Qwen2-7B and LLaMA-3-8B the method drops harmful rates from the 19-31% range to under 1% while holding task accuracy within about one point of the fine-tuned baseline, with no gradients, no safety data at deploy time, and no weight edits.

Core claim

For any fine-tuned checkpoint of a given backbone, a hypernetwork trained on other domains can map that checkpoint's layer-wise activation fingerprint into a model-specific Safe Side Network that restores near-baseline refusal behavior at inference time while leaving the deployed weights and task performance essentially intact.

What carries the argument

Activation-conditioned hypernetwork: layer-wise mean activations from ~50 calibration prompts are split into direction and magnitude, fused into a global embedding, and fed to factorized generators that emit the weights of a ladder-side Safe Side Network that scores prompts and routes only the harmful ones to refusal.

Load-bearing premise

A small set of domain-specific calibration prompts produces an activation fingerprint rich enough that a hypernetwork trained on other domains can generate a safety classifier that works for unseen fine-tuning recipes and for harmful prompts never seen in training.

What would settle it

On a held-out fine-tuned checkpoint whose domain was never seen by the hypernetwork, measure whether the generated Safe Side Network still drives harmful rates on AdvBench or HEx-PHI above a few percent while task accuracy stays near the fine-tuned baseline; failure of either metric would falsify the claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. HyperSafe proposes a post-hoc, model-specific, non-invasive recovery of safety for fine-tuned LLMs. Given a fine-tuned checkpoint Md, it extracts layer-wise mean last-token activation fingerprints A(Md) from a small set of domain calibration prompts (Eqs. 3–4), then uses a hypernetwork G to generate the weights θ of a lightweight Safe Side Network (SSN) in one forward pass (Eqs. 1, 8). The SSN is a ladder side-transformer that scores prompts and routes harmful ones to refusal while leaving safe prompts to the frozen fine-tuned model. The hypernetwork is supervised by ground-truth SSNs trained per domain on domain data plus BeaverTails, using reconstruction, classification, and functional losses. On Qwen2-7B and LLaMA-3-8B, across three domain-holdout splits (18 train / 4 held-out domains each), HyperSafe reduces harmful rates from 19–31% (fine-tuned averages) to below 1% on BeaverTails, AdvBench, and HEx-PHI for every held-out checkpoint, while keeping task accuracy within ~1% of the fine-tuned baseline. Ablations cover loss components, training-domain diversity, Ncal, architecture capacity, and transfer of a LoRA-generated SSN to full fine-tuning counterparts.

Significance. If the central claim holds under the stated protocol, the paper offers a practically useful third path between weight-editing defenses and model-agnostic guards: checkpoint-specific safety recovery without gradients, safety data at deploy time, or weight modification. Strengths include a clear problem formulation, a reproducible multi-split holdout design on two 7–8B families, fully zero-shot evaluation on AdvBench and HEx-PHI, systematic ablations (Figs. 3–4, Tables 10–12), transfer evidence from LoRA to full FT (Table 4), and public code. The combination of activation-conditioned hypernetworks with ladder side networks for safety is a concrete systems contribution. The result is significant for practitioners who receive already-fine-tuned checkpoints and cannot re-run alignment, provided the fingerprint premise generalizes beyond the paper’s domain-holdout and fixed-recipe setting.

major comments (2)
  1. [§4, Tables 2 and 4; Eqs. 3–4, 1, 8] The strongest claim (harmful rates 19–31% → <1% on every held-out checkpoint, task accuracy within ~1%, no weight edits) is demonstrated only under domain holdouts with a fixed LoRA recipe (rank 16, α=32, 3 epochs) on the same backbone family (§4, Table 7, Table 2). Table 4 reuses an SSN generated from a LoRA fingerprint on a full-FT counterpart of the same domain; it does not test a new PEFT recipe, optimizer, data mixture, or safety-degradation pattern outside the 22-domain support. The load-bearing premise is that A(Md) from Ncal=50 domain calibration prompts (Eqs. 3–4) encodes recipe-specific safety failure modes rather than mainly domain identity (§3.1–3.3). Without at least one experiment on a genuinely novel fine-tuning recipe or out-of-support degradation pattern, the post-hoc generality advertised in the introduction and Table 1 remains incompletely supported. A limited addition
  2. [§3.4, §4.1, Table 2] Ground-truth SSN supervision mixes per-domain safe data with BeaverTails harmful data (§3.4); BeaverTails is also one of the three evaluation benchmarks (disjoint split). AdvBench and HEx-PHI are zero-shot and show near-zero harmful rates, which is strong, but the paper should more explicitly quantify how much of the BeaverTails recovery is attributable to the shared harmful-data distribution versus the activation-conditioned generation. A short analysis (e.g., SSN trained without BeaverTails, or hypernetwork trained with a different harmful source) would clarify that the method is not primarily transferring a BeaverTails-specific classifier.
minor comments (5)
  1. [Table 1, Table 3, §4] Table 1 lists Llama Guard 4 as model-agnostic and standalone; the quantitative comparison in Table 3 is useful but should note that Guard is a separate 12B model with its own full forward pass, while HyperSafe adds 3–4% parameters and bypasses on safe prompts—so cost comparison is not only accuracy.
  2. [§3.2] The selective routing description (§3.2) states that safe prompts bypass the SSN during decoding; it would help to state explicitly whether the SSN still requires a full backbone forward pass to obtain ladder activations for scoring, or whether early-exit / partial-layer scoring is used.
  3. [Eq. (2), Table 5] Notation for the gate μk = σ(gk/τ) and the mapping ϕ(k) is clear in Eq. (2), but the temperature τ is only mentioned as a fixed hyperparameter; give its value in Table 5 or the text.
  4. [Figure 1, §3.4] Figure 1 stages are helpful; a one-sentence note on whether the hypernetwork is backbone-family-specific (separate G for Qwen2 vs LLaMA-3) would avoid ambiguity for multi-family deployment.
  5. [Abstract, §4] Minor presentation: abstract and intro use both “post hoc” and “post-hoc”; pick one. Also ensure AdvBench/HEx-PHI “never seen in any training phase” is stated once in the main evaluation protocol without relying only on the appendix.

Circularity Check

0 steps flagged

No significant circularity: HyperSafe is an empirical hypernetwork method with supervised SSN targets, domain holdouts, and external zero-shot safety benchmarks.

full rationale

The paper's central claims are empirical performance numbers (harmful rates 19–31% → <1% on held-out checkpoints; task accuracy within ~1%), not first-principles derivations. Training constructs ground-truth SSNs per domain via selective LM+BCE loss on BeaverTails train + in-domain safe data, then trains a hypernetwork with reconstruction + classification + functional losses on 18 domains; evaluation uses three disjoint domain-holdout splits (Table 7), a disjoint BeaverTails eval split, and fully zero-shot AdvBench/HEx-PHI never seen in any training phase (Tables 2–3). Activation fingerprints (Eqs. 3–4) and factorized generation (Eqs. 5–8) are architectural inputs, not quantities that definitionally equal the reported safety rates. No uniqueness theorem, self-citation chain, or fitted parameter is renamed as a prediction of the same quantity. Mild data-source overlap on BeaverTails is mitigated by the disjoint split and external benchmarks, so the evaluation is independent of the training supervision. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central claim rests on engineering choices and domain assumptions rather than free physical constants. Load-bearing free parameters are the calibration size, routing threshold, loss weights, and capacity knobs that control whether the generated SSN over-blocks or under-refuses. Domain axioms include that benign fine-tuning systematically weakens refusal and that mean last-token activations encode enough of that shift for a hypernetwork to invert. Invented entities are the SSN role, the fingerprint object A(Md), and the activation-conditioned generator G as used for safety recovery.

free parameters (5)
  • Ncal (calibration prompt count) = 50
    Default 50; sweep shows accuracy saturates there while harmful rate stays near zero (Fig. 4, Table 12). Fingerprint quality and thus generated SSN behavior depend on this choice.
  • Safety threshold λ = 0.5
    Binary routing cutoff at inference; set to 0.5 in Table 5. Directly controls false-block vs miss-harmful trade-off for the reported rates.
  • Loss weights β_recon, β_cls, β_func = 1, 10, 5
    Set to 1, 10, 5 (Table 5). Ablation shows removing functional/safe-probe terms changes accuracy and safety (Fig. 3a, Table 10), so reported operating point depends on these weights.
  • SSN capacity (K layers, hs=h/4, reduction factor) = K=12, hs=h/4
    Architecture knobs (~3–4% backbone params). Sensitivity table shows lower rank/dim hurts accuracy (Table 8).
  • Hypernetwork architecture dims and classifier rank = ~230M params; rank 16
    Hidden 512, direction proj 256, magnitude 128, classifier rank 16 (~230M params). Chosen design parameters that enable weight generation.
axioms (5)
  • domain assumption Even benign fine-tuning can substantially increase harmful compliance of safety-aligned LLMs.
    Stated in §1 citing Qi et al.; underpins the problem formulation and the need for recovery.
  • ad hoc to paper Layer-wise mean last-token activations on a small calibration set form a sufficient behavioral fingerprint of fine-tuning-induced representation shift for safety recovery.
    Eqs. 3–4 and §3.3; core modeling assumption enabling single-pass SSN generation without safety labels at deploy time.
  • domain assumption A lightweight ladder side network reading gated backbone states can perform prompt-level safety classification without modifying backbone weights.
    §3.2 extending Ladder Side-Tuning; architecture premise for non-invasive routing.
  • ad hoc to paper Ground-truth SSNs trained with domain data plus BeaverTails harmful data are valid supervision targets for hypernetwork weight prediction across domains.
    §3.4 training procedure; if GT SSNs are misaligned or non-transferable, generated SSNs inherit that failure.
  • domain assumption Holdout domains and AdvBench/HEx-PHI evaluate generalization of the generated classifier beyond calibration prompts and training harmful data.
    §4 evaluation protocol; standard ML generalization assumption for the zero-shot claim.
invented entities (3)
  • Safe Side Network (SSN) no independent evidence
    purpose: Model-specific lightweight classifier attached via ladder connections that outputs safety score λ and routes harmful prompts to refusal.
    Central constructed module; not a physical entity but a new system component defined for this method.
  • Activation fingerprint A(Md) no independent evidence
    purpose: Compact layer-wise representation of how fine-tuning shifted the model, used as hypernetwork input.
    Defined in Eqs. 3–4; purpose-built conditioning object for G.
  • Activation-conditioned hypernetwork G for safety SSN weights no independent evidence
    purpose: Maps fingerprints to full SSN parameters in one forward pass for unseen fine-tunes.
    §3.3; combines known hypernetwork idea with safety side networks as the paper’s generator.

pith-pipeline@v1.1.0-grok45 · 21020 in / 3985 out tokens · 43101 ms · 2026-07-14T05:20:37.438973+00:00 · methodology

0 comments
read the original abstract

Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance. Existing defenses mainly follow two directions: they either intervene during or after fine-tuning through retraining or weight modification, which can be costly and may hurt task performance, or they use model-agnostic safety classifiers, which may miss failures specific to a given fine-tuned checkpoint. These limitations motivate a post hoc, model-specific, and non-invasive approach to safety restoration. To meet these requirements, we propose HyperSafe, a framework that restores safety behavior by generating a model-specific Safe Side Network (SSN) for each fine-tuned checkpoint. HyperSafe uses layer-wise activation fingerprints to capture how fine-tuning changes the model's inner representations. With a small set of given calibration prompts, the hypernetwork maps these fingerprints to the parameters of the \ssn{} in a single forward pass. The generated \ssn{} runs alongside the frozen fine-tuned model and performs prompt-level safety classification: harmful prompts are routed to refusal, while safe prompts are answered by the original fine-tuned model. Thus, HyperSafe requires no gradient updates, no safety data at deployment time, and no modification to the deployed model weights. We evaluate HyperSafe on two model families, Qwen2-7B and LLaMA-3-8B, across multiple safety benchmarks. HyperSafe reduces harmful response rates from 19-31% to below 1% on every held-out checkpoint, while keeping downstream task accuracy within 1% of the fine-tuned baseline on average. Code is available at https://github.com/nokronim/project-safety-remedy.

Figures

Figures reproduced from arXiv: 2607.11475 by Abdelrahman Eldesokey, Aznaur Aliev, Bang An, Bernard Ghanem, Carlos Hinojosa, Yibo Yang.

Figure 1
Figure 1. Figure 1: HyperSafe pipeline. Stages 1–3 (training, one-time): fine-tune the base model on each training domain (1), train a domain-specific SSN per fine-tuned checkpoint (2), and train the hypernetwork to map activation fingerprints to SSN weights (3). Stage 4 (deployment): for any new fine-tuned model, extract calibration activations and generate a model-specific SSN in a single forward pass. original fine-tuned m… view at source ↗
Figure 2
Figure 2. Figure 2: SSN architecture. The frozen backbone’s hidden states are downsampled via gated ladder [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation summary on LLaMA-3, Split A holdout, shown as deviation from the full system. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of calibration set size Ncal (LLaMA-3, Split A holdout). Avg. task accuracy saturates at Ncal=50 (our default) while the harmful rate stays near zero; larger Ncal gives no further gains [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 13 linked inside Pith

  1. [1]

    Fine-tuning aligned language models compromises safety, even when users do not intend to

    Xiangyu Qi, Yi Zeng, Tinghao Xie, et al. Fine-tuning aligned language models compromises safety, even when users do not intend to. InInternational Conference on Learning Representa- tions (ICLR), 2024. URLhttps://openreview.net/forum?id=hTEGyKf0dZ

    2024

  2. [2]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, et al. Safety alignment should be made more than just a few tokens deep. InInternational Conference on Learning Representations (ICLR), 2025. URLhttps://openreview.net/forum?id=6Mxhg9PtDE

    2025

  3. [3]

    Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack

    Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. URL https://proceedings.neurips.cc /paper_files/paper/2024/file/bcfdaf04b54a69f47623c973c864ee8d-Paper-Confere nce.pdf

    2024

  4. [4]

    Representation noising: A defence mechanism against harmful finetuning

    Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, David Atanasov, Robie Gonza- les, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  5. [5]

    Safe LoRA: The silver lining of reducing safety risks when fine-tuning large language models

    Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Safe LoRA: The silver lining of reducing safety risks when fine-tuning large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  6. [6]

    Safety at one shot: Patching fine-tuned LLMs with a single instance

    Jiawen Zhang, Tony He, Kejia Chen, Jian Lou, Jian Liu, Xiaohu Yang, and Ruoxi Jia. Safety at one shot: Patching fine-tuned LLMs with a single instance. InThe Fourteenth International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/for um?id=EyH8Fu3vtZ

    2026

  7. [7]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  8. [8]

    Llama Guard: LLM-based input-output safeguard for human-AI conversations.arXiv preprint arXiv:2312.06674, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, et al. Llama Guard: LLM-based input-output safeguard for human-AI conversations.arXiv preprint arXiv:2312.06674, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs.arXiv preprint arXiv:2406.18495, 2024

    Seungju Han, Kavel Rao, Allyson Ettinger, et al. WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs.arXiv preprint arXiv:2406.18495, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Granite Guardian.arXiv preprint arXiv:2412.07724, 2024

    Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, et al. Granite Guardian.arXiv preprint arXiv:2412.07724, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack

    Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  12. [12]

    Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. InInternational Conference on Learning Representations (ICLR), 2024. 10

    2024

  13. [13]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation

    Meta AI. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. Meta AI Blog, 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intellige nce/

    2025

  14. [14]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  15. [15]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  16. [16]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  17. [17]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  18. [18]

    Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurabh Kadavath, Sandipan Kundu, et al. Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  20. [20]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. InInternational Conference on Machine Learning (ICML), 2024

    2024

  23. [23]

    Fraser, Hillary Dawkins, Isar Nejadgholi, and Svetlana Kiritchenko

    Kathleen C. Fraser, Hillary Dawkins, Isar Nejadgholi, and Svetlana Kiritchenko. Fine-tuning lowers safety and disrupts evaluation consistency. In Leon Derczynski, Jekaterina Novikova, and Muhao Chen, editors,Proceedings of the First Workshop on LLM Security (LLMSEC), pages 129–141, Vienna, Austria, August 2025. Association for Computational Linguistics. I...

    2025

  24. [24]

    LoRA fine-tuning efficiently undoes safety training in Llama 2-Chat 70B

    Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. LoRA fine-tuning efficiently undoes safety training in Llama 2-Chat 70B. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  25. [25]

    Removing RLHF protections in GPT-4 via fine-tuning

    Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing RLHF protections in GPT-4 via fine-tuning. InConference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024

    2024

  26. [26]

    Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation

    Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Joshua Kimball, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409.01586, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    TIES-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  28. [28]

    Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic

    Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2024. 11

    2024

  29. [29]

    Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment

    Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick McAuley, Somesh Jha, and Chaowei Zhao. Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  30. [30]

    ShieldGemma: Generative AI content moderation based on Gemma

    Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, et al. ShieldGemma: Generative AI content moderation based on Gemma. arXiv preprint arXiv:2407.21772, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    AEGIS: On- line adaptive AI content safety moderation with ensemble of LLM experts.arXiv preprint arXiv:2404.05993, 2024

    Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. AEGIS: On- line adaptive AI content safety moderation with ensemble of LLM experts.arXiv preprint arXiv:2404.05993, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    A holistic approach to undesired content detection in the real world

    Todor Markov, Chong Zhang, Sandhini Agarwal, et al. A holistic approach to undesired content detection in the real world. InAAAI Conference on Artificial Intelligence, 2023

    2023

  33. [33]

    A new generation of Perspective API: Efficient multilingual character-level transformers

    Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasser- man. A new generation of Perspective API: Efficient multilingual character-level transformers. InSIGKDD Conference on Knowledge Discovery and Data Mining, 2022

    2022

  34. [34]

    HiddenDetect: Detecting jailbreak attacks against multimodal large language models via monitoring hidden states

    Yilei Jiang, Xinyan Gao, Tianshuo Peng, Yingshui Tan, Xiaoyong Zhu, Bo Zheng, and Xiangyu Yue. HiddenDetect: Detecting jailbreak attacks against multimodal large language models via monitoring hidden states. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for...

    2025

  35. [35]

    The internal state of an LLM knows when it’s lying

    Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP, 2023

    2023

  36. [36]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Obeso, Janosch Sygnowski, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  37. [37]

    Representation engineering: A top-down approach to AI transparency.arXiv preprint arXiv:2310.01405, 2023

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to AI transparency.arXiv preprint arXiv:2310.01405, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    David Ha, Andrew Dai, and Quoc V . Le. Hypernetworks. InInternational Conference on Learning Representations (ICLR), 2017. URL https://openreview.net/forum?id=rkpACe 1lx

    2017

  39. [39]

    A brief review of hypernetworks in deep learning.Artificial Intelligence Review, 57(9):250, 2024

    Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review of hypernetworks in deep learning.Artificial Intelligence Review, 57(9):250, 2024

    2024

  40. [40]

    Continual learning with hypernetworks

    Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  41. [41]

    Equivariant architectures for learning in deep weight spaces

    Aviv Navon, Aviv Shamsian, Idan Achituve, et al. Equivariant architectures for learning in deep weight spaces. InInternational Conference on Machine Learning (ICML), 2023

    2023

  42. [42]

    HyperTuning: Toward adapting large language models without back-propagation

    Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. HyperTuning: Toward adapting large language models without back-propagation. InInternational Conference on Machine Learning (ICML), 2023

    2023

  43. [43]

    Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks

    Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2021

    2021

  44. [44]

    Parameter prediction for unseen deep architectures

    Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero-Soriano. Parameter prediction for unseen deep architectures. InAdvances in Neural Information Processing Systems (NeurIPS), 2021. 12

    2021

  45. [45]

    Bronstein, Yang You, Zhangyang Wang, and Kai Wang

    Zhiyuan Liang, Dongwen Tang, Yuhao Zhou, Xuanlei Zhao, Mingjia Shi, Wangbo Zhao, Zekai Li, Peihao Wang, Konstantin Schürholt, Damian Borth, Michael M. Bronstein, Yang You, Zhangyang Wang, and Kai Wang. Drag-and-drop LLMs: Zero-shot prompt-to-weights.arXiv preprint arXiv:2506.16406, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    LST: Ladder side-tuning for parameter and memory efficient transfer learning

    Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. LST: Ladder side-tuning for parameter and memory efficient transfer learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/ha sh/54801e196796134a2b0ae5e8adef1fef-Abstract-Conference.html

    2022

  47. [47]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL https://openre view.net/forum?id=nZeVKeeFYf9

    2022

  48. [48]

    BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  49. [49]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. InInternational Conference on Machine Learning (ICML), 2023. URLhttps://arxiv.org/abs/2307.15043. 13 Appendix Overview.This appendix provides additional implementation details, dataset statistics, ablation studies, and exte...

    Pith/arXiv arXiv 2023