Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned LLM safety can be restored post-hoc: keep only the parameter changes that buy utility cheaply, then add a Hessian-derived compensation vector, preserving nearly all task gains across datasets and models.

desk verdict Genuinely useful post-hoc fine-tuning safety defense with strong experiments, but the multi-delta compensation in Eq. (10) is an unproven approximation, so the central guarantee is weaker than claimed. read the letter →

arxiv 2505.12038 v1 pith:IGC4CW72 submitted 2025-05-17 cs.LG cs.AIcs.CR

Ning Lu , Shengcai Liu , Jiahao Wu , Weiyu Chen , Zhirui Zhang , Yew-Soon Ong , Qi Wang , Ke Tang This is my paper · ORCID
classification cs.LGcs.AIcs.CR MSC 68T0768T50
keywords fine-tuningsafetyalignmentdeltaparametersoptimalbrainsurgeonHessiancompensationharmfuldefenseutility-safetytrade-offgreedyparameterselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the safety damage caused by fine-tuning an aligned LLM can be repaired after the fact, without giving up the task improvements the fine-tuning was meant to deliver. It proposes a defence, Safe $\Delta$, that works on the parameter change itself: each individual weight change is scored for how much utility it buys per unit of safety loss, the high-scoring changes are kept up to a safety budget, and a compensation vector derived from the safety Hessian is added to cancel the safety damage the kept changes would otherwise cause. On datasets spanning explicitly harmful, implicitly harmful, dirty benign, and clean benign fine-tuning, with varying data sizes and several base models, the method reports safety close to the original aligned model while retaining nearly all of the fine-tuning utility gain. If this holds, fine-tuning service providers gain a single per-request post-processing step that adapts to any user dataset instead of per-dataset tuning of defense strength.

What carries the argument

The load-bearing object is the compensation identity of Theorem 4.1, an Optimal Brain Surgeon-style result reapplied to a safety objective: for a single delta $\delta w_m$, the parameter perturbation that realizes that change at minimal safety cost is $C_m = \delta w_m [H^{-1}]_{mm}^{-1} H^{-1}_{:,m}$, with residual cost $\delta L^m_{\text{safe}} = (\delta w_m)^2 / (2[H^{-1}]_{mm})$. Because the utility proxy is simply the squared delta, the selection score simplifies to $r_m = 2[H^{-1}]_{mm}$: the ranking of parameters depends only on the precomputed inverse Hessian diagonal, not on the fine-tuning data. The machinery then does three jobs: it prices each parameter change in safety units, it ranks changes for the greedy knapsack-style selection up to the threshold $\epsilon = s \cdot \tfrac{1}{N_m} \sum_m \tfrac{1}{2}[H^{-1}]_{mm}$, and it constructs the final model $W_{\text{sd}} = W_{\text{orig}} + M \odot \Delta W_{\text{sft}} + C$ with the mask $(I - M)$ protecting the selected utility gains.

What would settle it

After running Safe $\Delta$, directly measure the final model's safety loss on a hold-out safety set and compare it with the summed per-delta estimates the method uses to declare the constraint satisfied; if the measured loss (or the attack success rate on the safety benchmark) exceeds the predicted value when the selected deltas are chosen to lie in the same Hessian subspace, the joint-compensation assumption fails. A simpler version: fine-tune on a dataset engineered so that the kept deltas have overlapping compensation columns and check that the model actually stays within the claimed safety budget.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is stated in Theorem 4.1 and the two-step procedure built on it. Adding a single fine-tuning delta $\delta w_m$ to the original aligned weights raises the safety loss $L_{\text{safe}} = \|W X_{\text{safe}} - W_{\text{orig}} X_{\text{safe}}\|_2^2$, and the minimal such increase is $\delta L^m_{\text{safe}} = (\delta w_m)^2 / (2[H^{-1}]_{mm})$, achieved by spreading the change along the vector $C_m = (\delta w_m / [H^{-1}]_{mm}) H^{-1}_{:,m}$, where $H$ is the safety Hessian at the original weights. Safe $\Delta$ uses this quantity as a per-parameter safety price, sorts the deltas by the utility-per-safety-loss ratio $r_m = 2[H^{-1}]_{mm}$, keeps the most favourable ones until the cumulative price reaches a threshold $\epsilon$, and then adds the sum of the individual compensation vectors on the unselected positions. The paper argues this consistently preserves safety while leaving the utility gain from benign fine-tuning nearly intact, and reports experiments supporting it across harmful, dirty, and clean datasets, dataset sizes from 50 to 10,000 examples, three LLMs, and LoRA.

Load-bearing premise

The safety guarantee assumes that the compensation vectors computed for each selected weight change in isolation still work when all of them are added together, so overlapping or correlated effects cannot push the final model's actual safety loss above the budget the estimates claim.

Editorial extensions

If this is right

  • Fine-tuning providers can post-process any user-trained model with one cached object — the inverse safety Hessian computed once from the aligned model and a safety dataset — adding roughly 62 seconds per request instead of re-tuning defense hyperparameters per dataset.
  • The defence scales with harmful data: attack success rate stays near the original model's level as the harmful fine-tuning set grows from 50 to 200 examples (and to 1K–10K in the extended experiments), where data-augmentation baselines degrade sharply.
  • Benign-task utility is largely preserved: on Dirty Summary, Safe Delta reaches 0.489 Rouge-1 F1 versus 0.491 for undefended fine-tuning while cutting ASR from 63.94% to 5.15%, and on Math it keeps 0.334 accuracy versus 0.337 while cutting ASR from 11.52% to 3.33%.
  • The same procedure transfers across model families and parameter-efficient fine-tuning: it works on Llama-2-7B-Chat, Llama-2-13B-Chat, Llama-3-8B-Instruct, and on LoRA deltas formed as $\Delta W = AB$.
  • A user-controlled threshold $s$ provides a practical safety-utility dial, and the measured trade-off shows a wide range of $s$ where utility is near fine-tuning level while safety remains high.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the selection ratio $r_m = 2[H^{-1}]_{mm}$ is independent of the delta magnitudes, the parameter ranking is fixed once the Hessian is known; only the cutoff moves per request. A testable consequence is that an adversary who knows the cached Hessian can predict which changes will be kept and which compensated, then shape fine-tuning data to hide harmful directions among low-ranked parameter
  • The selection step is a 0/1 knapsack solved greedily; since item ratios are static, an exact or better combinatorial selection is a natural extension that could push the utility-safety frontier further, an improvement the paper itself signals.
  • The utility estimate is a proxy — squared parameter distance, not task loss; on a small model where a per-request utility Hessian is affordable, one could test whether the proxy-chosen mask differs from an output-space-utility mask and whether that changes reported utility at equal safety.
  • The method is linear in the deltas, so an adversary could spread harmful content across many small parameter changes that individually fall under the per-parameter screen; stress-testing Safe Delta against such obfuscated harmful fine-tuning sets would probe a gap the paper's benchmarks do not cover.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Safe Delta proposes a post-hoc defense for fine-tuned LLMs. After standard fine-tuning, the method selects a subset of delta parameters (W_sft - W_orig) that are judged to give high utility per unit safety cost, and then adds a compensation vector inspired by Optimal Brain Surgeon to mitigate safety degradation. The safety cost is computed via the Hessian of a layer-wise quadratic output-distance loss on a fixed safety dataset. The paper proves a single-delta optimal compensation theorem (Thm 4.1) and uses a greedy selection with ratio r_m = 2[H^{-1}]_{mm}, followed by summing per-delta compensation vectors. Experiments on PureBad, Identity Shift, Dirty Summary, and Math datasets with Llama-2-7B-Chat, Llama-3-8B-instruct, and Llama-2-13B-chat show that Safe Delta preserves low ASR/HS while retaining most utility, compared with data-based (SafeInstr, BEA) and weight-based (Safe LoRA, Resta) baselines.

Significance. If the safety guarantee held, Safe Delta would be a valuable practical defense: it requires a one-time Hessian computation, adds ~62s per request, is model-agnostic, supports LoRA, and empirically outperforms baselines across dataset types and sizes. The paper's strengths are its extensive empirical evaluation and the clean OBS derivation for a single delta, together with open-source code and detailed implementation notes. However, the central theoretical claim — that the method 'consistently preserves safety' for arbitrary multi-delta fine-tunes — is not established, because the multi-delta compensation is asserted rather than derived, and the safety guarantee is for a layer-output quadratic proxy rather than the evaluated ASR/HS. The utility claim similarly rests on a parameter-distance proxy rather than measured task utility. These gaps are significant but appear addressable in revision.

major comments (4)
  1. [§4.3, Eq. (10)] The multi-delta compensation in Eq. (10) is not proven. Theorem 4.1 solves the single-constraint problem min_ΔW 1/2 ΔW^T H ΔW s.t. e_m^T ΔW = δw_m, giving C_m = δw_m/[H^{-1}]_{mm} H^{-1}_{:,m}. For a selected set S, the compensation that actually minimizes the quadratic safety loss while fixing all selected deltas is the joint OBS solution, whose unselected components are H^{-1}_{U,S}(H^{-1}_{S,S})^{-1} δw_S. Equation (10), which sums per-delta vectors and masks selected positions, coincides with the joint solution only when H^{-1}_{S,S} is diagonal. For the actual Hessian H = 2 X_safe^T X_safe, the selected delta coordinates can be strongly correlated, so cross-terms in 1/2 ΔW^T H ΔW are generally nonzero and the post-compensation safety loss can exceed the cumulative estimate Σ_{m∈S} δL_safe^m used in the greedy selection. Thus the abstract's claim that Safe Delta 'consistently preserves safety' is not established for multiple deltas. The ASR/HS experiments do not measure the quadratic layer-output loss against ε, so they cannot close this gap; I suggest either deriving the joint compensation (with a practical approximation) or stating and verifying a diagonality/uncorrelatedness condition.
  2. [§3, Eq. (2) vs §4.2, Eqs. (4)-(5)] The theoretical guarantee is for a proxy, not the declared safety objective. Equation (2) constrains L_safe(W_sd) - L_safe(W_orig) ≤ ε, where L_safe is a loss on the safety dataset, but the method subsequently replaces this with layer-wise quadratic output distances in Eqs. (4)-(5) and per-layer thresholds of the form ε = s·mean(...). The paper never shows that the layer-output distance bound implies a bound on the actual safety loss L_safe, nor that it implies the ASR/HS metrics on which safety is evaluated. Consequently, the statement 'consistently preserves safety' in the abstract is not a theorem about the evaluated metrics; it is an empirical observation. The authors should either prove a transfer bound from the quadratic proxy to the safety loss/metric or clearly limit the theoretical claim to the proxy.
  3. [§4.3, Eqs. (7)-(8)] The utility objective in Eq. (2) is task loss on D_sft, but the method optimizes parameter distance L_util = ||W_sd - W_orig||^2. The claim that small parameter distance 'naturally constrains' output perturbation (Appendix E.2) is a heuristic; it is not proven and does not directly control task utility. The empirical utility scores in Tables 3-4 are close to the fine-tuned model, but the selection procedure does not explicitly maximize task performance. If the authors wish to claim 'utility gain remains unaffected', they should measure task utility directly in the selection or provide a Lipschitz-style argument connecting parameter distance to the task loss.
  4. [§5.1 / Appendix E.3] The Safe LoRA threshold is tuned on the Dirty Summary dataset, which is also the evaluation dataset for that scenario ('we optimize this parameter on the Dirty Summary dataset, selecting a threshold of 0.52'). This is test-set tuning. The comparison in Table 3 and Figure 5(b) is therefore not apples-to-apples: Safe Delta's s=0.1 is fixed, while Safe LoRA is given access to the evaluation split. Please re-tune the baseline on a separate validation set or report results without tuning.
minor comments (5)
  1. [§5.3] There is a typo in 'senario' in the first sentence; please correct it.
  2. [Tables 2-4] The main results report single runs without error bars or standard deviations, even though Section 5.9 averages time cost over five trials. Given the randomness of fine-tuning and the use of GPT-4 as a judge, please report means and variances over multiple seeds for the main safety and utility numbers.
  3. [Figure 2] The figure caption and text refer to a 'Selective metric' that is not defined in the main text; please clarify what the y-axis represents and how the numbers are computed.
  4. [Appendix E.2] The Hessian is computed on 512 safe examples generated from AdvBench, while the evaluation benchmark uses 330 questions across 11 policy categories. Please discuss whether the safety examples cover the same distribution as the benchmark, since a mismatch could affect the validity of the layer-output proxy.
  5. [§1, Contributions] The claim that Safe Delta is 'the first method to jointly estimate safety degradation and dynamically optimize delta parameter' is strong; consider softening it or substantiating the novelty in light of related safety re-alignment methods (e.g., NLSR, Booster) cited in the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the safety/utility claims are tested on external benchmarks, and the proxy-based selection/compensation chain is internally derived rather than fitted to the evaluation.

full rationale

Safe Delta's derivation is self-contained, and its headline claims are checked against external benchmarks, so no circular step is present. The safety proxy L_safe = ||W_sd X_safe - W_orig X_safe||_2^2 (Eq. 5) is the objective being optimized; Theorem 4.1 (Eq. 6) solves the single-delta constrained quadratic minimization for that proxy, and Eq. (10) is the paper's chosen multi-delta extrapolation. This is an optimization construction, not a prediction that reduces to its inputs: the safety evaluation in Section 5.1 uses the Policy-Oriented Safety Evaluation Benchmarks of Qi et al. (2024) with ASR and harmfulness score, and utility is assessed with MMLU, MT-Bench, Rouge-1 F1, and GSM8k accuracy, none of which is defined in terms of the selected mask M, the compensation C, or the layer-output loss L_safe. The scaling factor s = 0.1 is a fixed hyperparameter, not fitted to the evaluation metrics. Self-citations in the paper (Appendix F background references to the authors' prior work on adversarial examples and multi-objective optimization) are not load-bearing for the Safe Delta construction. The paper's own Limitation section acknowledges greedy selection as a limitation, and Appendix C proves only the single-delta theorem; these are completeness/rigor gaps rather than circular reasoning. In particular, the joint optimality of Eq. (10) for multiple deltas is not established, but that is a correctness concern, not a case of the derivation being equivalent to its inputs by definition.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The central claim rests on the safety proxy, the utility proxy, the multi-delta compensation approximation, and the representativeness of the safety dataset.

free parameters (2)
  • Threshold scaling factor s = 0.1
    Controls the layer-wise safety degradation budget in the threshold ε = s * mean(1/(2[H^{-1}]_mm)). Set by hand, not tuned per dataset in the main experiments, but Figure 7 shows the trade-off as s varies.
  • Safe LoRA threshold = 0.52
    Baseline hyperparameter tuned on the Dirty Summary test set in Appendix E.3. Included because the reported comparison depends on this choice, and it is tuned on the same distribution used for evaluation.
assumptions (4)
  • domain assumption L_safe = ||W X_safe - W_orig X_safe||^2 is a valid proxy for model safety
    The method optimizes output preservation on a safety dataset and evaluates ASR/HS on a benchmark. The transfer from output distance to actual safety behavior is assumed, not proven.
  • domain assumption Parameter distance ||W_sd - W_orig||^2 approximates utility loss on the fine-tuning task
    Section 4.3 and E.2 replace the layer-output utility objective with an L2 parameter distance. No validation is provided that this proxy correlates with task metrics like F1 or accuracy.
  • ad hoc to paper The sum of per-delta OBS compensations approximates the joint optimal compensation
    Theorem 4.1 is proven for a single delta. Eq. (10) sums independent compensations for all selected deltas, which is not generally optimal under multiple constraints and is not analyzed.
  • domain assumption The safety dataset used for Hessian computation is representative of the evaluation benchmark
    512 safety examples from AdvBench instructions are used for H, while the 330-question policy benchmark is used for evaluation. Overlap or distribution shift is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets." pith.science (2026). https://pith.science/paper/IGC4CW72

@misc{pith2026250512038,
  author       = {Pith},
  title        = {Pith review of: Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IGC4CW72}},
  note         = {Machine review of arXiv:2505.12038}
}
read the original abstract

Large language models (LLMs) have shown great potential as general-purpose AI assistants across various domains. To fully leverage this potential in specific applications, many companies provide fine-tuning API services, enabling users to upload their own data for LLM customization. However, fine-tuning services introduce a new safety threat: user-uploaded data, whether harmful or benign, can break the model's alignment, leading to unsafe outputs. Moreover, existing defense methods struggle to address the diversity of fine-tuning datasets (e.g., varying sizes, tasks), often sacrificing utility for safety or vice versa. To address this issue, we propose Safe Delta, a safety-aware post-training defense method that adjusts the delta parameters (i.e., the parameter change before and after fine-tuning). Specifically, Safe Delta estimates the safety degradation, selects delta parameters to maximize utility while limiting overall safety loss, and applies a safety compensation vector to mitigate residual safety loss. Through extensive experiments on four diverse datasets with varying settings, our approach consistently preserves safety while ensuring that the utility gain from benign datasets remains unaffected.

Figures

Figures reproduced from arXiv: 2505.12038 by the authors.

Figure 1
Figure 1. Existing defense methods struggle when fine-tuning on diverse datasets, causing insufficient protection or utility loss. Left: Data-based methods (BEA, SafeInstr) with 10% aug￾mented safe examples fail to maintain safety as the size of harmful fine-tuning datasets grows. Right: Weight modification methods (Safe LoRA, Resta) fail to balance the utility gained from benign fine-tuning with the need to preserve safety a… view at source ↗
Figure 2
Figure 2. Preliminary experiments. Guided selection achieves a higher utility score with comparable harmfulness level, compared to random selection. Experiments fine-tune llama-2-7B-Chat on the Dirty Summary dataset. Selective metric is introduced later. Formally, the optimization objective is to minimize the loss on the fine-tuning task compared with the fine-tuned model, while ensuring safety performance is degraded by an a… view at source ↗
Figure 3
Figure 3. Overview of Safe Delta. Safe Delta consists of a preparation step performed before fine-tuning and two main steps executed for each fine-tuning request. In the preparation step, the Hessian inverse is computed and saved once, leveraging the original aligned model and safety dataset. For each fine-tuning request, Step 1 selects a subset of delta parameters that maximize total utility improvement while ensuring the sa… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: (a), Safe Delta successfully achieves this balance, whereas baseline methods fail to do so. Furthermore, for a specific fine-tuning dataset, per-request hyperparameter optimization can help mitigate these issues. We adopt such optimization process on the Dirty Summary …
Figure 6
Figure 6. Figure 6: Performance of SafeDelta with LoRA. “#n” denotes the harmfulness score for the n-th safety category (lower is better; see Appendix B.1 for category details). “MT-B” reflects basic utility, and “F1” measures summarization ability (higher is better). contrast, weight mod…
Figure 7
Figure 7. Figure 7: Safety-utility trade-off of Safe Delta at different thresh￾olds controlled by s, evaluated on Dirty Summary fine-tuning. 5.8. Ablation Study Effectiveness of Safety Compensation Vector. To evaluate the role of the safety compensation vector in the Safe Delta process, w…
Figure 8
Figure 8. Figure 8: GPT-4 evaluation prompt used to compute the Harmfulness Score. 0.3 0.35 0.4 0.5 0.52 0.54 0.55 0.56 0.6 0.7 Safe LoRA Threshold 0 10 20 30 40 50 ASR (%) 0.25 0.30 0.35 0.40 0.45 F1 [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Safety-utility trade-off of Safe LoRA at different threshold values, evaluated on the Dialog Summary dataset. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TRACE: Trajectory-Based Safety Patch Learning for LLM Post-Training Realignment

    cs.LG 2026-06 conditional novelty 6.0 of 10

    One low-rank adapter, trained across simulated harmful fine-tuning trajectories, restores ≥94% safety on fine-tuned LLMs while keeping task accuracy within ±1.7% of the undefended model.

  2. Open Your Eyes: Vision Enhances Message Passing Neural Networks in Link Prediction

    cs.CV 2025-05 accept novelty 6.0 of 10

    Visual features extracted from rendered subgraph images consistently improve message-passing link predictors on seven benchmarks, and an efficient node-centered variant scales to large graphs.

Reference graph

Works this paper leans on

74 extracted references · 65 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint, 2023

    work page 2023

  3. [3]

    Claude, 2023

    Anthropic. Claude, 2023. URL https://claude.ai/

    work page 2023

  4. [4]

    A general language assistant as a laboratory for alignment

    Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint, 2021

    work page 2021

  5. [5]

    Constitutional AI: harmlessness from AI feedback

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., et al. Constitutional AI: harmlessness from AI feedback. arXiv preprint, 2022

    work page 2022

  6. [6]

    D., and Poria, S

    Bhardwaj, R., Anh, D. D., and Poria, S. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp.\ 14138--14149, 2024

    work page 2024

  7. [7]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions

    Bianchi, F., Suzgun, M., Attanasio, G., R \" o ttger, P., Jurafsky, D., Hashimoto, T., and Zou, J. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations,, 2024

    work page 2024

  8. [8]

    J., and Wong, E

    Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language models in twenty queries. arXiv preprint, 2023

    work page 2023

Show all 74 references
  1. [9]

    and Kwok, J

    Chen, W. and Kwok, J. Multi-objective deep learning with adaptive reference vectors. In Advances in Neural Information Processing Systems, 2022

    2022

  2. [10]

    and Kwok, J

    Chen, W. and Kwok, J. You only merge once: Learning the pareto set of preference-aware model merging. arXiv preprint, 2024 a

    2024

  3. [11]

    and Kwok, J

    Chen, W. and Kwok, J. T. Efficient pareto manifold learning with low-rank structure. In ICML, 2024 b

    2024

  4. [12]

    Chen, W., Zhang, X., Lin, B., Lin, X., Zhao, H., Zhang, Q., and Kwok, J. T. Gradient-based multi-objective deep learning: Algorithms, theories, applications, and beyond. arXiv preprint, 2025

    2025

  5. [13]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint, 2021

    2021

  6. [14]

    Or-bench: An over-refusal benchmark for large language models

    Cui, J., Chiang, W.-L., Stoica, I., and Hsieh, C.-J. Or-bench: An over-refusal benchmark for large language models. arXiv preprint, 2024

    2024

  7. [15]

    Safe RLHF: safe reinforcement learning from human feedback

    Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe RLHF: safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024

    2024

  8. [16]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, volume 202, pp.\ 10323--10337, 2023

    2023

  9. [17]

    Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP , pp.\ 3356--3369, 2020

    2020

  10. [18]

    SAMS um corpus: A human-annotated dialogue dataset for abstractive summarization

    Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. SAMS um corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp.\ 70--79, 2019

    2019

  11. [19]

    T., Zhang, Y., and Wang, M

    Gou, Y., Ko, T., Yang, H., Kwok, J. T., Zhang, Y., and Wang, M. Leveraging per image-token consistency for vision-language pre-training. In CVPR, pp.\ 19155--19164, 2023 a

    2023

  12. [20]

    T., and Zhang, Y

    Gou, Y., Liu, Z., Chen, K., Hong, L., Xu, H., Li, A., Yeung, D., Kwok, J. T., and Zhang, Y. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arXiv preprint, 2023 b

    2023

  13. [21]

    T., and Zhang, Y

    Gou, Y., Chen, K., Liu, Z., Hong, L., Xu, H., Li, Z., Yeung, D., Kwok, J. T., and Zhang, Y. Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation. In ECCV, volume 15075 of Lecture Notes in Computer Science, pp.\ 388--404, 2024

    2024

  14. [22]

    The llama 3 herd of models

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint, 2024

    2024

  15. [23]

    and Stork, D

    Hassibi, B. and Stork, D. G. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, pp.\ 164--171, 1992

    1992

  16. [24]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020

    2020

  17. [25]

    Safe lo RA : The silver lining of reducing safety risks when finetuning large language models

    Hsu, C.-Y., Tsai, Y.-L., Lin, C.-H., Chen, P.-Y., Yu, C.-M., and Huang, C.-Y. Safe lo RA : The silver lining of reducing safety risks when finetuning large language models. In Advances in Neural Information Processing Systems, 2024

    2024

  18. [26]

    J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022

    2022

  19. [27]

    Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning

    Huang, T., Bhattacharya, G., Joshi, P., Kimball, J., and Liu, L. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning. arXiv preprint, 2024 a

    2024

  20. [28]

    F., and Liu, L

    Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint, 2024 b

    2024

  21. [29]

    F., and Liu, L

    Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 c

    2024

  22. [30]

    Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack

    Huang, T., Hu, S., and Liu, L. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 d

    2024

  23. [31]

    F., and Liu, L

    Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In ICLR, 2025

    2025

  24. [32]

    Catastrophic jailbreak of open-source llms via exploiting generation

    Huang, Y., Gupta, S., Xia, M., Li, K., and Chen, D. Catastrophic jailbreak of open-source llms via exploiting generation. In The Twelfth International Conference on Learning Representations, 2024 e

    2024

  25. [33]

    Accurate post training quantization with small calibration sets

    Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Accurate post training quantization with small calibration sets. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 4466--4475, 2021

    2021

  26. [34]

    Beavertails: Towards improved safety alignment of LLM via a human-preference dataset

    Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Chen, B., Sun, R., Wang, Y., and Yang, Y. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), NeurIPS, 2023

    2023

  27. [35]

    S., and Solla, S

    LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain damage. In Advances in Neural Information Processing Systems, pp.\ 598--605, 1989

    1989

  28. [36]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

    2004

  29. [37]

    Efficient combinatorial optimization for word-level adversarial textual attack

    Liu, S., Lu, N., Chen, C., and Tang, K. Efficient combinatorial optimization for word-level adversarial textual attack. IEEE ACM Trans. Audio Speech Lang. Process. , 30: 0 98--111, 2022

    2022

  30. [38]

    Effective and imperceptible adversarial textual attack via multi-objectivization

    Liu, S., Lu, N., Hong, W., Qian, C., and Tang, K. Effective and imperceptible adversarial textual attack via multi-objectivization. ACM Trans. Evol. Learn. Optim. , 4 0 (3): 0 16:1--16:23, 2024 a

    2024

  31. [39]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Liu, X., Xu, N., Chen, M., and Xiao, C. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations, 2024 b

    2024

  32. [40]

    Jailbreaking chatgpt via prompt engineering: An empirical study

    Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., and Liu, Y. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint, 2023

    2023

  33. [41]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  34. [42]

    Large language models can be guided to evade ai-generated text detection

    Lu, N., Liu, S., He, R., Ong, Y., Wang, Q., and Tang, K. Large language models can be guided to evade ai-generated text detection. Trans. Mach. Learn. Res., 2024, 2024 a

    2024

  35. [43]

    Less is more: Understanding word-level textual adversarial attack via n-gram frequency descend

    Lu, N., Liu, S., Zhang, Z., Wang, Q., Liu, H., and Tang, K. Less is more: Understanding word-level textual adversarial attack via n-gram frequency descend. In 2024 IEEE Conference on Artificial Intelligence (CAI), pp.\ 823--830, 2024 b

    2024

  36. [44]

    Training overhead ratio: A practical reliability metric for large language model training systems

    Lu, N., Xie, Q., Zhang, H., Fang, W., Zheng, Y., Hu, Z., and Ma, J. Training overhead ratio: A practical reliability metric for large language model training systems. In ISSRE, pp.\ 391--393, 2024 c

    2024

  37. [45]

    Lundberg, S. M. and Lee, S. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp.\ 4765--4774, 2017

    2017

  38. [46]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022

    2022

  39. [47]

    Fine-tuning now available for gpt-4o, 2024

    Peng, A., Allard, J., and Heidel, S. Fine-tuning now available for gpt-4o, 2024. URL https://openai.com/index/gpt-4o-fine-tuning/. Accessed: 16-Jan-2025

    2024

  40. [48]

    Instruction tuning with GPT-4

    Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with GPT-4 . arXiv preprint, 2023

    2023

  41. [49]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024

    Qi, X., Zeng, Y., Xie, T., Chen, P., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024

    2024

  42. [50]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023

    2023

  43. [51]

    Modelgrow: Continual text-to-video pre-training with model expansion and language understanding enhancement

    Rao, Z., Ji, L., Xing, Y., Liu, R., Liu, Z., Xie, J., Peng, Z., He, Y., and Chen, Q. Modelgrow: Continual text-to-video pre-training with model expansion and language understanding enhancement. arXiv preprint, 2024

    2024

  44. [52]

    Representation noising: A defence mechanism against harmful finetuning

    Rosati, D., Wehner, J., Williams, K., Bartoszcze, L., Gonzales, R., Maple, C., Majumdar, S., Sajjad, H., and Rudzicz, F. Representation noising: A defence mechanism against harmful finetuning. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., an...

    2024

  45. [53]

    Shen, L., Chen, W., and Kwok, J. T. Multi-resolution diffusion models for time series forecasting. In ICLR, 2024 a

    2024

  46. [54]

    do anything now

    Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pp.\ 1671--1685, 2024 b

    2024

  47. [55]

    Axiomatic attribution for deep networks

    Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp.\ 3319--3328, 2017

    2017

  48. [56]

    J., and Fergus, R

    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing properties of neural networks. In Bengio, Y. and LeCun, Y. (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2...

    2014 arXiv

  49. [57]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  50. [58]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023

    2023

  51. [59]

    Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment

    Wang, J., Li, J., Li, Y., Qi, X., Hu, J., Li, Y., McDaniel, P., Chen, M., Li, B., and Xiao, C. Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  52. [60]

    W., Lester, B., Du, N., Dai, A

    Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021

    2021

  53. [61]

    Wei, Y., Huang, Q., Zhang, Y., and Kwok, J. T. KICGPT: large language model with knowledge in context for knowledge graph completion. In Findings of the Association for Computational Linguistics: EMNLP , pp.\ 8667--8683, 2023

    2023

  54. [62]

    T., and Zhang, Y

    Wei, Y., Fu, S., Jiang, W., Zhang, Z., Zeng, Z., Wu, Q., Kwok, J. T., and Zhang, Y. GITA: graph to visual and textual integration for vision-language graph reasoning. In Advances in Neural Information Processing Systems, 2024

    2024

  55. [63]

    Open the eyes of mpnn: Vision enhances mpnn in link prediction, 2025

    Wei, Y., Wang, X., Zhuang, Z., Chen, Y., Chen, S., Zhang, Y., Zhang, Y., and Kwok, J. Open the eyes of mpnn: Vision enhances mpnn in link prediction, 2025

    2025

  56. [64]

    Backdoor graph condensation

    Wu, J., Lu, N., Dai, Z., Fan, W., Liu, S., Li, Q., and Tang, K. Backdoor graph condensation. arXiv Preprint, 2024

    2024

  57. [65]

    Tf-dcon: Leveraging large language models (llms) to empower training-free dataset condensation for content-based recommendation, 2025

    Wu, J., Liu, Q., Hu, H., Fan, W., Liu, S., Li, Q., Wu, X.-M., and Tang, K. Tf-dcon: Leveraging large language models (llms) to empower training-free dataset condensation for content-based recommendation, 2025

    2025

  58. [66]

    RLCD: reinforcement learning from contrastive distillation for LM alignment

    Yang, K., Klein, D., Celikyilmaz, A., Peng, N., and Tian, Y. RLCD: reinforcement learning from contrastive distillation for LM alignment. In The Twelfth International Conference on Learning Representations, 2024

    2024

  59. [67]

    Y., Zhao, X., and Lin, D

    Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint, 2023

    2023

  60. [68]

    NLSR: neuron-level safety realignment of large language models against harmful fine-tuning

    Yi, X., Zheng, S., Wang, L., de Melo, G., Wang, X., and He, L. NLSR: neuron-level safety realignment of large language models against harmful fine-tuning. In AAAI, pp.\ 25706--25714, 2025

    2025

  61. [69]

    Yu, R., Chen, W., Wang, X., and Kwok, J. T. Enhancing meta learning via multi-objective soft improvement functions. In ICLR, 2023

    2023

  62. [70]

    GPT-4 is too smart to be safe: Stealthy chat with llms via cipher

    Yuan, Y., Jiao, W., Wang, W., Huang, J., He, P., Shi, S., and Tu, Z. GPT-4 is too smart to be safe: Stealthy chat with llms via cipher. In The Twelfth International Conference on Learning Representations, 2024

    2024

  63. [71]

    Removing RLHF protections in GPT-4 via fine-tuning

    Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T., and Kang, D. Removing RLHF protections in GPT-4 via fine-tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Short Pa...

    2024

  64. [72]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, 2023

    2023

  65. [73]

    Model tailor: Mitigating catastrophic forgetting in multi-modal large language models

    Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Wu, C., and Kuang, K. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  66. [74]

    Z., and Fredrikson, M

    Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint, 2023

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.