Pith. sign in

REVIEW 3 major objections 4 minor 75 references

Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fixed null-space gate zeros gradients of harmful samples during downstream fine-tuning, keeping post-release attack success at the pre-release level.

desk verdict Clever and correct gradient-blocking mechanism whose headline resistance result is calibrated to the attack set, plus an in-threat-model LM-head bypass that is never evaluated. read the letter →

arxiv 2608.05045 v1 pith:TAQAKFWS submitted 2026-08-05 cs.CR cs.AIcs.CL

Yuxuan Huang , Xingyu Zeng , Tianhang Zheng , Chaochao Lu This is my paper
classification cs.CRcs.AIcs.CL
keywords nullspacegradientblockingsafefine-tuningopen-weightLLMUnidirectionalSafetyGateInverseAdapterPPOWreleaseattacksuccessrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a model provider can release an open-weight LLM that stays trainable for benign users while remaining resistant to malicious fine-tuning, without asking the downstream trainer to follow any safety procedure. The proposed mechanism, the Unidirectional Safety Gate (USG), inserts a Null Space Cubic Layer and an Inverse Adapter after the final Transformer layer: the cubic layer zeroes the backpropagated gradient for any sample whose last-layer hidden state falls inside a region built from defender-held harmful examples, and the Inverse Adapter analytically restores the base model's forward outputs. The paper's central empirical claim is that under a fixed release threshold calibrated to block 100% of the defender's harmful reference set, post-fine-tuning attack success rate stays at the pre-release level in all six evaluated model-dataset settings, while safe pass rates remain 100% on JailbreakBench and HarmfulBench and drop to 79% and 69% on BeaverTails-H. If correct, this would show that representation-space blocking at release time can raise the cost of malicious adaptation without relying on downstream cooperation.

What carries the argument

The load-bearing object is the Null Space Cubic Layer composed with its analytic Inverse Adapter. The cubic layer has output $O(h)=((hW)h^T)h$ for hidden state $h$ and a fixed symmetric $W$; each term of its gradient, term1 $=(G_B h^T)(h W^T)$, term2 $=(h G_B^T)(h W)$, term3 $=(h W^T h^T)G_B$, contains $hW$ or $hW^T$, so when $W$ is constructed as $VV^T$ from the common right null space of harmful hidden states, harmful gradients are forced to zero. Because each local hidden-state vector is only rescaled by the scalar $s=hWh^T$, the Inverse Adapter $h=y(yWy^T)^{-1/3}$ recovers the original vector in the forward pass, preserving base-model behavior. The practical version turns the hard zero condition into a backward gate $g(h)=\mathbf{1}[r(h)\leq \tau]$ with $r(h)=\|hW\|/\|h\|$, and when exact null-space construction becomes rank-limited the paper replaces exact zero directions with smallest-singular-value directions (extended null-space selection).

What would settle it

Fine-tune a released safeguarded model on a harmful question-answer set whose prompts have been paraphrased or otherwise shifted so that their hidden states are unlikely to lie in the defender-built protected region, then measure post-fine-tuning ASR under the same frozen threshold; if ASR rises substantially above the pre-release level, the claim that the gate keeps post-fine-tuning ASR at the pre-release level under a fixed release threshold is falsified for that coverage regime.

Watch

Extended reading notes

Core claim

At the paper's core is a gradient-blocking identity. For a hidden state $h$ reaching the cubic layer, the layer computes $O(h)=((hW)h^T)h$, and the backward gradient splits into three terms, each containing $hW$ or $hW^T$ as a factor. The parameter $W=VV^T$ is built from a basis $V$ of the common right null space of the defender-held harmful hidden states, so for participating harmful samples $hW=hW^T=0$ and the gradient vanishes exactly (Equation 4). Because exact null spaces collapse once the harmful reference set grows past tens of samples, the paper's practical instantiation replaces the exact condition with a threshold gate on the norm ratio $r(h)=\|hW\|/\|h\|$: gradients are suppressed when this ratio falls at or below the calibrated threshold $\tau$. The paper claims that non-participating but in-distribution harmful samples also tend to have smaller norm ratios than safe samples, so the protected region extends beyond the build set; that with a strict 100% harmful-block threshold the post-fine-tuning ASR is unchanged from pre-release across six settings; and that the analytic Inverse Adapter $h=y(yWy^T)^{-1/3}$ restores base-model forward outputs, with perfect top-1 token agreement and cosine similarity numerically equal to 1 in the evaluated runs.

Load-bearing premise

The load-bearing premise is that the attacker's harmful fine-tuning examples fall inside the protected region defined by the defender-held harmful reference set that builds W and calibrates the threshold, and the paper's strict persistence experiments use the same 100 examples as both the build set and the downstream harmful training set, leaving no end-to-end evidence for harmful data outside that corpus.

Editorial extensions

If this is right

  • Under the strict threshold that blocks 100% of defender-held harmful reference data, post-fine-tuning ASR equals pre-release ASR in all six model-dataset settings (Qwen and Llama on JailbreakBench, HarmfulBench, and BeaverTails-H).
  • Safe pass rate under that threshold is 100% on JailbreakBench and HarmfulBench for both model families, and 79% for Qwen-14B and 69% for Llama-8B on BeaverTails-H, so blocking coverage trades against safe utility.
  • The Inverse Adapter preserves the base model on Alpaca (ACC 66.0% for Qwen and 60.0% for Llama) and reconstructs hidden states with top-1 token agreement of 1.0 and near-zero logit KL divergence on a 500-sample benign mixed set.
  • Non-participating, in-distribution harmful samples show lower norm ratios than safe samples, allowing threshold calibration to extend blocking beyond the exact null-space build set.
  • Exact null-space construction supports only tens of harmful samples (about 13 to 91 per dataset), while extended null-space selection retains 100% safe pass rate at about 200 participating harmful samples and over 60% at 500 in the BeaverTails/Qwen case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the formal guarantee is zero gradient on the protected subset, the defense's effective strength is set by coverage of the attacker's training corpus: an adversary who can shift harmful prompts out of the calibrated region through rephrasing, format changes, or rare topics would bypass the gate without touching the protected component. This follows from the paper's own sensitivity discussio
  • The same norm-ratio calibration recipe could transfer to other safety dimensions, such as private-data leakage or biosecurity prompts, whenever the defender can supply a representative reference set; the paper does not test those domains.
  • A concrete testable extension is to fine-tune the released model on paraphrased or hidden-state-space adversarial harmful examples and compare post-fine-tuning ASR with the paper's strict 100% coverage result; the threshold gate should degrade before the protected layer itself is removed.
  • The PPOW assumption that the protected component cannot be cheaply deleted is only preliminarily explored through a parameter-fusion feasibility check, so the defense's practical durability depends on that hardening step maturing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a Unidirectional Safety Gate (USG) for a provider-controlled partially protected open-weight (PPOW) release setting, instantiated as a Null Space Cubic Layer plus an Inverse Adapter inserted after the final Transformer layer. During downstream fine-tuning, the cubic layer blocks gradients for samples whose hidden-state norm ratio r(h)=||hW||/||h|| falls below a calibrated threshold, while the Inverse Adapter analytically restores the base forward behavior. Experiments on Qwen3-14B and Llama-3.1-8B across JailbreakBench, HarmfulBench, and BeaverTails-H report that, under a strict fixed-threshold 100% harmful-block release policy, post-fine-tuning ASR remains at the pre-release level; safe pass rates are 100% on the easier settings and lower on BeaverTails-H; and the Inverse Adapter achieves near-exact forward restoration. The paper also studies blocking generalization to non-participating harmful samples and the stability of the threshold under continued benign and mildly shifted training.

Significance. If the central claim held as stated, the paper would constitute a useful new release-time defense mechanism: gradient blocking in representation space that does not require downstream cooperation. The paper has real strengths: the cubic-layer gradient derivation in Section 4.2.2 and Appendix C is mathematically sound, the forward-restoration measurements in Table 4 are strong and directly support the Inverse Adapter claim, the code is promised, and the limitations section is unusually candid. However, the headline resistance result is currently close to a construction-level tautology: in the strict persistence experiments the threshold is calibrated on the exact corpus later used as the downstream harmful fine-tuning set, so every harmful training sample is blocked by design. In addition, the gate is placed before the output LM head, leaving an in-threat-model attack vector (head updates) completely unexamined. These issues do not invalidate the mechanism, but they substantially narrow the empirical claim that the paper can currently support.

major comments (3)
  1. [Section 5.1.1 and Appendix A] The central resistance result is produced by construction rather than by an empirical test. Section 5.1.1 states that, for JailbreakBench and HarmfulBench, the same 100 examples serve both as the participating harmful build set and as the downstream harmful fine-tuning set; Appendix A defines the strict 100% release policy as setting the threshold to the maximum observed norm ratio over all defender-held harmful reference samples. Under this policy every harmful training sample satisfies r(h) <= tau, so Eq. (17) in Appendix A suppresses its gradient entirely, no harmful update reaches the trained layers, and post-fine-tuning ASR must equal pre-fine-tuning ASR. The reported constancy of ASR in Table 1 is therefore a direct consequence of the calibration rule, not evidence that the safeguard resists harmful fine-tuning. The authors should either test on non-participating harmful samples that actually fall above the threshold and measure ASR persistence, or explicitly relabel the current experiment as a coverage-based blocking test rather than a resistance result.
  2. [Section 4.6 Step 1 and Section 5.1.3] The gate location leaves the output LM head completely unprotected, and head-update attacks are inside the stated threat model but never evaluated. Section 4.6 Step 1 inserts the cubic layer and Inverse Adapter after the last Transformer layer, and the backward gate in Eq. (17) suppresses only the gradient at the cubic-layer input h. Gradients for parameters downstream of that point, most importantly the output LM head, do not pass through h, so they are not blocked by the gate. Section 3.1 grants the attacker full control over all exposed weights, but Section 5.1.3 restricts the resistance experiments to fine-tuning only the last five Transformer layers. A harmful fine-tune that updates the LM head on the same 100 harmful QA pairs could remap the unchanged base-model hidden states to harmful tokens without ever being gated. The authors should evaluate head-only and full-model harmful fine-tuning; without such results the headline claim that post-fine-tuning ASR stays at the pre-release level is unsupported for an in-threat-model attack vector.
  3. [Section 3.1 and Appendix E] The protected-release assumption, which is load-bearing for the entire defense, is not established by any experiment in the paper. Section 3.1 excludes attacks that delete, bypass, replace, or independently retrain the protected component because such attacks are asserted to be substantially more expensive than ordinary fine-tuning, and Section 4.5 points to parameter fusion as one plausible realization. Appendix E, however, explicitly describes the fusion study as preliminary, fitted with limited calibration data and limited budget, and states that the current fusion quality should not be read as the best achievable fidelity-hardening trade-off. Since the defense is vacuous against an attacker who can simply drop the gate at negligible cost, the paper should either provide concrete evidence that structural removal is nontrivial for the released artifact, or state this as an axiom and remove it from the claimed evaluation of resistance.
minor comments (4)
  1. [Section 4.2.3] The dimension notation is inconsistent: the main text defines h in R^{B x L x D}, but Eq. (10) and the surrounding text use R^N and a matrix H in R^{N x N} without relating N to D; this makes the construction of V and W harder to follow.
  2. [Appendix C, Eq. (A1)] The gradient derivation reuses the symbol N for a different dimension: h is written as R^{B x M x N} and W as R^{N x N}, whereas the main paper uses D for the hidden size; the notation should be unified.
  3. [Abstract] The abstract contains a formatting typo, 'aUnidirectional Safety Gate', which should read 'a Unidirectional Safety Gate'.
  4. [Table 1 and Table 2] The safe pass rate is only reported for the strict 100% harmful-block policy; reporting safe pass rates for the same threshold in the BeaverTails-H non-participating setting would help readers disentangle blocking generalization from the by-construction blocking in the strict setting.

Circularity Check

1 steps flagged · score 8.0 of 10

The headline resistance result is forced by construction: the strict 100% threshold is calibrated on the same harmful examples later used as the attack set, so the gate blocks every harmful training gradient by definition.

  1. fitted input called prediction [Section 5.1.1 (Models and Datasets); Appendix A (Threshold Selection); Eq. (17)]
    "For JailbreakBench and HarmfulBench, we use 100 harmful training examples per dataset; these same 100 examples serve both as the participating harmful build set used to construct the Null Space Cubic Layer and as the downstream harmful fine-tuning set in the strict persistence experiments. ... The strict 100% release policy used in the main text is the limiting case b=1, i.e., the threshold is set to the maximum observed ratio over all defender-held harmful reference samples, including harmful samples that did not participate in constructing the null space. ..."

    For those examples, W is built from the common right null space of their own hidden states, so r(h)=||hW||/||h||≈0 on the attack set. The threshold τ is then the maximum r over exactly this reference set, forcing every harmful fine-tuning sample to satisfy r(h)≤τ. The gate g(h)=1[r(h)≤τ] therefore zeroes ∂L/∂h for every harmful training sample, so no harmful gradient reaches the trainable layers. The Table 1 result that post-fine-tuning ASR stays at the pre-release level is the algebraic consequence of this calibration protocol, not an empirically discovered property of the defense.

full rationale

The individual mechanism derivations are not circular: Eq. (4) is a definitional condition of the USG, and the cubic-layer gradient formula (Eqs. 6–7) correctly implies zero backward gradient when hW=0; the Inverse Adapter's forward restoration is an analytic inverse and is checked numerically rather than fitted. The norm-ratio separation for non-participating harmful samples (Fig. 3) is an empirical observation. The circularity is concentrated in the strict resistance evaluation: because τ is calibrated on the same harmful reference set that is later used as the downstream fine-tuning data, the headline 'post-ASR unchanged' claim is guaranteed by construction rather than tested. The paper's own limitation sections are transparent about the coverage assumption, but transparency does not remove the fact that the central empirical result reduces to the calibration choice. I do not count the LM-head fine-tuning omission as circularity; that is a threat-model coverage gap, not a derivation that reduces to its own inputs. The score is 8 because the paper's central resistance claim is forced by the definition of the threshold, even though independent content remains in the safe-pass measurements, the inverse-adapter fidelity checks, and the OOD stability boundary experiments.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

The central mechanism depends on a calibrated threshold, a chosen null-space rank, a low learning rate, a limited harmful-sample count, whether the defender's corpus covers the attacker's data, and whether the protected component is tamper-resistant. The paper is honest about most of these dependencies, but the headline resistance experiment is structured so that the calibrated threshold fully blocks every attack sample, making the main result a consequence of the calibration setup.

free parameters (4)
  • Release threshold tau = Qwen JBB 0.00049, Qwen HB 0.0021, Qwen BeaverTails 0.0036, Llama JBB 0.00044, Llama HB 0.0031, Llama BeaverTails…
    The threshold on norm ratio r(h) determines which samples have gradients zeroed; it is calibrated from defender-held harmful reference data and is the central operating point of the defense.
  • Null-space rank r (columns of V) = Not reported; selected by singular-value cutoff in extended null-space selection
    The matrix W = V V^T defines the protected region; the rank r controls the size and shape of that region and is chosen algorithmically rather than derived from first principles.
  • Fine-tuning learning rate = 4e-5 in the main resistance experiments; 3e-4 collapses blocking in Appendix G
    The central resistance result is demonstrated only at a low learning rate; Appendix G shows that a larger update step erases the harmful-safe separation.
  • Number of participating harmful samples = 100 per model-dataset setting
    The sample count determines whether an exact or extended null space exists; Appendix F Table 6 shows exact null-space selection becomes infeasible beyond tens of samples for some datasets.
assumptions (5)
  • domain assumption The protected component (cubic layer and inverse adapter) cannot be removed or retrained at negligible cost after release.
    Section 3.1 defines the PPOW setting and excludes deletion, bypass, replacement, or independent retraining of W_prot. Appendix E only sketches a preliminary fusion feasibility check, not a demonstrated tamper-proofing guarantee.
  • ad hoc to paper Defender-held harmful reference data covers the attacker's downstream harmful fine-tuning data in the strict persistence experiments.
    Section 5.1.1 states that the same 100 examples serve as the build set and the attack set for JailbreakBench and HarmfulBench, and BeaverTails harmful samples used for threshold calibration are described as covered. This assumption is what makes the post/pre ASR flat.
  • domain assumption Norm-ratio separation between harmful and safe hidden states persists under fine-tuning with a fixed threshold.
    Section 5.4 and Appendices B and G report the separation visually; Appendix G shows it collapses at learning rate 3e-4, so the persistence is update-dependent.
  • domain assumption The analytic inverse of the cubic layer is defined for essentially all samples, meaning the per-token scale factor s is positive.
    Section 4.3 and Appendix D rely on s > 0 for the inverse map h = y(yWy^T)^(-1/3). If exact zeros occur, the inverse collapses and forward restoration is not guaranteed.
  • domain assumption The stacked harmful hidden-state matrix has a usable exact or extended null space for the build set.
    Appendix F Table 6 shows exact null-space selection is limited to tens of harmful samples, and extended selection is an approximation that also loses separation at larger scales.
invented entities (3)
  • Null Space Cubic Layer
    purpose: A fixed, non-learnable cubic layer after the final transformer layer whose gradient is zero when hidden states lie in a protected null-space region.
    It is well specified and tested in the paper, but no external benchmark independently confirms its behavior beyond the authors' code and internal measurements.
  • Inverse Adapter
    purpose: Analytic inverse of the cubic layer, restoring forward outputs to base-model behavior without adding trainable parameters.
    The paper verifies top-1 token agreement and near-zero reconstruction error, but these are internal measurements rather than independent external evidence.
  • Unidirectional Safety Gate
    purpose: Abstract mechanism class that blocks backward gradients for forbidden inputs while preserving forward behavior, realized by the cubic layer plus inverse adapter.
    Defined in Section 3.3 as a definitional class; it is a framework concept rather than a separately testable artifact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning." pith.science (2026). https://pith.science/paper/TAQAKFWS

@misc{pith2026260805045,
  author       = {Pith},
  title        = {Pith review of: Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TAQAKFWS}},
  note         = {Machine review of arXiv:2608.05045}
}
read the original abstract

Released aligned large language models remain vulnerable to malicious downstream finetuning. Existing defenses are largely designed for the fine-tuning-as-a-service (FTaaS) paradigm or rely on downstream users to follow additional safety procedures, and therefore do not directly address the setting we study: a provider controlled partially protected open-weight (PPOW) release setting in which most weights remain trainable while a small safety-critical component is preserved at release. We propose a Unidirectional Safety Gate (USG), instantiated as a Null Space Cubic Layer together with an Inverse Adapter inserted after the final Transformer layer. During downstream fine-tuning, the cubic layer suppresses or blocks gradients from harmful samples whose hidden states fall in a calibrated protected region, while the Inverse Adapter restores the base model's forward behavior. In practice, we calibrate a threshold using defender-held harmful data, allowing protection to generalize to nearby in-distribution harmful samples. Across six evaluated model-dataset settings, USG keeps post-finetuning attack success rate close to the pre-release level under a fixed release threshold, while maintaining high safe-pass rates on easier settings and exhibiting a clearer safety-utility trade-off on unsafe samples from BeaverTails. These results suggest that release-time representation-space blocking can raise the cost of malicious downstream adaptation without requiring downstream cooperation. The code is available at https://github.com/OpenCausaLab/Gradient-Immunity.

Figures

Figures reproduced from arXiv: 2608.05045 by the authors.

Figure 1
Figure 1. The figure illustrates the idealized behavior of [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The roles of the Cubic Layer and Inverse [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Norm-ratio visualization in the Qwen￾14B BeaverTails-H setting of participating harmful samples Dforbidden, non-participating harmful samples Dtoxic \ Dforbidden, and safe samples. Although the non￾participating harmful samples are not used to construct the null space, they still tend to lie closer to the protected region than safe samples, showing a representation-level tendency that supports defender-calibrated th… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Visual Threshold Selection 15 [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Threshold trajectories during continued safe training. For each model–dataset pair, we tune the threshold [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Safe pass rate during continued safe training under thresholds tuned to maintain a 95% harmful block rate. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Ratio-scatter comparisons for Qwen3-14B before and after continued safe training. Within each row, the [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Ratio-scatter comparisons for Llama-3.1-8B before and after continued safe training. Within each row, the [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Hidden states matrix rank increases with sample size [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Safe sample pass rate (with 95% harmful block rate) as a function of the number of participating [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Safe-pass and harmful-block trajectories under a fixed strict-100 release threshold. The strict reference [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Ratio-scatter comparisons at initialization and the final monitored step under a fixed strict-100 release [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 33 canonical work pages

  1. [3]

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2025. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering

    work page 2025

  2. [6]

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, and 1 others. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Advances in Neural Information Processing Systems, 37:55005--55029

    work page 2024

  3. [7]

    Junqi Chen, Sirui Chen, and Chaochao Lu. 2026. Can post-training transform llms into causal reasoners? In Findings of the Association for Computational Linguistics: ACL 2026, pages 17020--17038

    work page 2026

  4. [12]

    Guoqing Hu, An Zhang, Shuo Liu, Zhibo Cai, Xun Yang, and Xiang Wang. 2025 a . Alphafuse: Learn id embeddings for sequential recommendation in null space of language embeddings. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1614--1623

    work page 2025

  5. [16]

    Tiansheng Huang, Sihao Hu, and Ling Liu. 2024 c . Vaccine: Perturbation-aware alignment for large language model. The Thirty-Eighth Annual Conference on Neural Information Processing Systems …

    work page 2024

  6. [17]

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678--24704

    2023

  7. [20]

    Lizhi Lin, Honglin Mu, Zenan Zhai, Minghan Wang, Yuxia Wang, Renxi Wang, Junjie Gao, Yixuan Zhang, Wanxiang Che, Timothy Baldwin, and 1 others. 2025. Against the achilles' heel: A survey on red teaming for generative models. Journal of Artificial Intelligence Research, 82:687--775

    2025

  8. [22]

    Zihan Liu, Yizhen Wang, Rui Wang, and Sai Wu. 2025. Dualguard: A parameter space transformation approach for bidirectional defense in split-based llm fine-tuning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17065--17080

    work page 2025

Show all 75 references
  1. [25]

    Rimon Melamed, Lucas Hurley McCabe, Tanay Wakhare, Yejin Kim, H Howie Huang, and Enric Boix-Adsera. 2024. Prompts have evil twins. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 46--74

    2024

  2. [27]

    Quoc Minh Nguyen, Trung Le, Jing Wu, Anh Tuan Bui, and Mehrtash Harandi. 2026. Antibody: Strengthening defense against harmful fine-tuning for large language models via attenuating harmful gradient influence. In International Conference on Learning Representations (ICLR). ArXi...

    2026

  3. [28]

    Alberto Purpura, Sahil Wadhwa, Jesse Zymet, Akshay Gupta, Andy Luo, Melissa Kazemi Rad, Swapnil Shinde, and Mohammad Shahed Sorower. 2025. Building safe genai applications: An end-to-end overview of red teaming for large language models. In Proceedings of the 5th Workshop on T...

    2025

  4. [35]

    Wei Sun, Tingyu Qu, Mingxiao Li, Jesse Davis, and Marie Francine Moens. 2025. Mitigating negative interference in multilingual knowledge editing through null-space constraints. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8796--8810

    2025

  5. [36]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  6. [38]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  7. [43]

    Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. 2025 b . Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14635--14645

    2025

  8. [44]

    Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. 2024. A safety realignment framework via subspace-oriented model fusion for large language models. Knowledge-Based Systems, 306:112701

    2024

  9. [51]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  10. [52]

    Publications Manual , year = "1983", publisher =

    1983

  11. [53]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981

  12. [54]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  13. [55]

    Dan Gusfield , title =. 1997

    1997

  14. [56]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  15. [57]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  16. [58]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  17. [59]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  18. [60]

    Hashimoto , title =

    Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , title =. GitHub repository , howpublished =. 2023 , publisher =

    2023

  19. [61]

    IEEE Transactions on Knowledge and Data Engineering , year=

    A survey on mixture of experts in large language models , author=. IEEE Transactions on Knowledge and Data Engineering , year=

  20. [62]

    arXiv preprint arXiv:2503.23829 , year=

    Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains , author=. arXiv preprint arXiv:2503.23829 , year=

    arXiv

  21. [63]

    arXiv preprint arXiv:2503.06639 , year=

    Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification , author=. arXiv preprint arXiv:2503.06639 , year=

  22. [64]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  23. [65]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    arXiv

  24. [66]

    Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025) , pages=

    Building safe GenAI applications: An end-to-end overview of red teaming for large language models , author=. Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025) , pages=

    2025

  25. [67]

    arXiv preprint arXiv:2406.05946 , year=

    Safety alignment should be made more than just a few tokens deep , author=. arXiv preprint arXiv:2406.05946 , year=

    arXiv

  26. [68]

    Journal of Artificial Intelligence Research , volume=

    Against the achilles' heel: A survey on red teaming for generative models , author=. Journal of Artificial Intelligence Research , volume=

  27. [69]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Prompts have evil twins , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  28. [70]

    arXiv preprint arXiv:2409.18169 , year=

    Harmful fine-tuning attacks and defenses for large language models: A survey , author=. arXiv preprint arXiv:2409.18169 , year=

    arXiv

  29. [71]

    arXiv preprint arXiv:2402.05162 , year=

    Assessing the brittleness of safety alignment via pruning and low-rank modifications , author=. arXiv preprint arXiv:2402.05162 , year=

    arXiv

  30. [72]

    arXiv preprint arXiv:2412.07097 , year=

    On evaluating the durability of safeguards for open-weight llms , author=. arXiv preprint arXiv:2412.07097 , year=

    arXiv

  31. [73]

    2024 , organization=

    Vaccine: Perturbation-aware alignment for large language model , author=. 2024 , organization=

    2024

  32. [74]

    arXiv preprint arXiv:2309.07875 , year=

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions , author=. arXiv preprint arXiv:2309.07875 , year=

    arXiv

  33. [75]

    arXiv preprint arXiv:2405.14577 , volume=

    Representation noising effectively prevents harmful fine-tuning on llms , author=. arXiv preprint arXiv:2405.14577 , volume=

    arXiv

  34. [76]

    arXiv preprint arXiv:2402.02207 , year=

    Safety fine-tuning at (almost) no cost: A baseline for vision large language models , author=. arXiv preprint arXiv:2402.02207 , year=

    arXiv

  35. [77]

    arXiv preprint arXiv:2403.05030 , year=

    Defending against unforeseen failure modes with latent adversarial training , author=. arXiv preprint arXiv:2403.05030 , year=

    arXiv

  36. [78]

    Knowledge-Based Systems , volume=

    A safety realignment framework via subspace-oriented model fusion for large language models , author=. Knowledge-Based Systems , volume=. 2024 , publisher=

    2024

  37. [79]

    arXiv preprint arXiv:2410.10343 , year=

    Locking down the finetuned llms safety , author=. arXiv preprint arXiv:2410.10343 , year=

    arXiv

  38. [80]

    arXiv preprint arXiv:2410.02355 , year=

    Alphaedit: Null-space constrained knowledge editing for language models , author=. arXiv preprint arXiv:2410.02355 , year=

    arXiv

  39. [81]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Mitigating negative interference in multilingual knowledge editing through null-space constraints , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  40. [82]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  41. [83]

    Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    Alphafuse: Learn id embeddings for sequential recommendation in null space of language embeddings , author=. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  42. [84]

    arXiv preprint arXiv:2510.27172 , year=

    Adaptive defense against harmful fine-tuning for large language models via bayesian data scheduler , author=. arXiv preprint arXiv:2510.27172 , year=

  43. [85]

    arXiv preprint arXiv:2510.07835 , year=

    MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generation , author=. arXiv preprint arXiv:2510.07835 , year=

  44. [86]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    DualGuard: A Parameter Space Transformation Approach for Bidirectional Defense in Split-Based LLM Fine-Tuning , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  45. [87]

    arXiv preprint arXiv:2601.22752 , year=

    OSNIP: Breaking the Privacy-Utility-Efficiency Trilemma in LLM Inference via Obfuscated Semantic Null Space , author=. arXiv preprint arXiv:2601.22752 , year=

  46. [88]

    arXiv preprint arXiv:2409.01586 , year=

    Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation , author=. arXiv preprint arXiv:2409.01586 , year=

    arXiv

  47. [89]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  48. [90]

    2023 , publisher=

    Stanford alpaca: An instruction-following llama model , author=. 2023 , publisher=

    2023

  49. [91]

    Advances in Neural Information Processing Systems , volume=

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset , author=. Advances in Neural Information Processing Systems , volume=

  50. [92]

    Advances in Neural Information Processing Systems , volume=

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models , author=. Advances in Neural Information Processing Systems , volume=

  51. [93]

    arXiv preprint arXiv:2402.04249 , year=

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal , author=. arXiv preprint arXiv:2402.04249 , year=

    arXiv

  52. [94]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations) , pages=

    Llamafactory: Unified efficient fine-tuning of 100+ language models , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations) , pages=

  53. [95]

    International Conference on Learning Representations (ICLR) , year=

    Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence , author=. International Conference on Learning Representations (ICLR) , year=

  54. [96]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    arXiv

  55. [97]

    arXiv preprint arXiv:2607.19829 , year=

    DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protection , author=. arXiv preprint arXiv:2607.19829 , year=

    arXiv

  56. [98]

    arXiv preprint arXiv:2605.08930 , year=

    Internalizing Safety Understanding in Large Reasoning Models via Verification , author=. arXiv preprint arXiv:2605.08930 , year=

    arXiv

  57. [99]

    arXiv preprint arXiv:2607.15081 , year=

    DataShield: Uncovering Risky Fine-Tuning Data Across LLMs Through Consensus Subspace Alignment , author=. arXiv preprint arXiv:2607.15081 , year=

    arXiv

  58. [100]

    arXiv preprint arXiv:2605.31140 , year=

    EvoDefense: Co-Evolving Black-Box Defense with Large Language Models , author=. arXiv preprint arXiv:2605.31140 , year=

    arXiv

  59. [101]

    arXiv preprint arXiv:2605.30883 , year=

    TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking , author=. arXiv preprint arXiv:2605.30883 , year=

    arXiv

  60. [102]

    arXiv preprint arXiv:2605.08778 , year=

    Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking , author=. arXiv preprint arXiv:2605.08778 , year=

    arXiv

  61. [103]

    arXiv preprint arXiv:2605.08936 , year=

    Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories , author=. arXiv preprint arXiv:2605.08936 , year=

    arXiv

  62. [104]

    arXiv preprint arXiv:2507.18576 , year=

    SafeWork-R1: Coevolving Safety and Intelligence under the AI-45 Law , author=. arXiv preprint arXiv:2507.18576 , year=

    arXiv

  63. [105]

    arXiv preprint arXiv:2512.07141 , year=

    Think-reflect-revise: A policy-guided reflective framework for safety alignment in large vision language models , author=. arXiv preprint arXiv:2512.07141 , year=

  64. [106]

    arXiv preprint arXiv:2602.01539 , year=

    MAGIC: A co-evolving attacker-defender adversarial game for robust LLM safety , author=. arXiv preprint arXiv:2602.01539 , year=

  65. [107]

    arXiv preprint arXiv:2607.18056 , year=

    An Early Warning of Emerging Biosecurity Risks in Frontier LLMs , author=. arXiv preprint arXiv:2607.18056 , year=

    arXiv

  66. [108]

    arXiv preprint arXiv:2512.01827 , year=

    CauSight: Learning to Supersense for Visual Causal Discovery , author=. arXiv preprint arXiv:2512.01827 , year=

  67. [109]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    Can Post-Training Transform LLMs into Causal Reasoners? , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    2026

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.