Pith. sign in

REVIEW 4 major objections 8 minor 32 references

Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Restoring a tiny fraction of pruned neurons recovers 27% of the safety that pruning removes.

desk verdict First to tackle safety restoration in pruned LVLMs, with a sensible hierarchical selection, but missing a random-restoration control and with several effect sizes close to noise. read the letter →

arxiv 2505.16104 v2 pith:O7RFFMVI submitted 2025-05-22 cs.CL cs.CVcs.LG

Yue Li , Xin Yi , Dongsheng Shi , Gerard de Melo , Xiaoling Wang , Linlin Wang This is my paper
classification cs.CLcs.CVcs.LG
keywords safetyrealignmentmodelpruninglargevision-languagemodelssafety-criticalneuronsattentionheadslightweightrestorationWandaShipsscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pruning large vision-language models for deployment reliably degrades their safety: across six models, attack success rates rise by 2.8 to 15.4 percentage points after 50% sparsity. This paper claims that most of that loss is recoverable by restoring only a small, identifiable set of pruned weights. Its method, Hierarchical Safety Realignment (HSR), first locates attention heads whose removal most distorts the model's outputs on harmful inputs, then, inside those heads, finds pruned neurons that rank high on safety importance while retaining some utility importance. Restoring just 0.02 to 1.8 ten-thousandths of the pruned set recovers over 14% of the lost safety in every tested case and over 27% in most, with negligible utility change. If correct, this means safe deployment of compressed multimodal models does not require retraining or full fine-tuning.

What carries the argument

The load-bearing object is the safety-critical neuron set S(p, q, pmax) = (Ss(q) ∩ Su(pmax)) − Su(p), the pruned weights that rank in the top q% of safety importance, in the top pmax% of utility importance, and outside the utility-important set that pruning kept. Identification is hierarchical: the head level uses the Ships metric, which computes the KL divergence between the model's output distribution on harmful inputs with and without a head scaled to near zero, then takes the sum of principal angles between the original and ablated activation feature spaces via SVD; the neuron level computes importance scores (Wanda, SparseGPT, or SNIP) on a safety dataset of unsafe instructions paired with refusal responses and on a utility dataset of safe instructions, then applies the set formula. Only the weights in this set are restored, which is what keeps the realignment lightweight, at 0.02 to 1.8 ten-thousandths of the pruned weights.

What would settle it

Randomly restore the same number of pruned weights within the same heads, matched for scale and layer, and compare safety recovery: if random restoration within the selected heads achieves safety recovery comparable to HSR's targeted restoration, the claim that the safety-critical-neuron selection is what carries the effect would fail.

Watch

Extended reading notes

Core claim

The central discovery is expressed as a hierarchical attribution: safety in LVLMs is carried by a sparse set of attention heads, and within those heads by a sparse set of neurons, and utility-based pruning preferentially removes exactly those neurons because they contribute little to utility. HSR operationalizes this by scoring each attention head with the Safety head importance score (Ships), a measure of how much scaling a head's query/key/value matrices to near zero changes the output distribution on harmful instructions, aggregated over a safety dataset via singular-value comparison. The top-h such heads are selected, and for each, the paper computes safety importance and utility importance scores for weights using Wanda, SparseGPT, or SNIP on paired safe-utility and unsafe-safety datasets. The restored set is S(p, q, pmax) = (Ss(q) ∩ Su(pmax)) − Su(p): pruned neurons with top-q safety importance, top-pmax utility importance, but outside the original utility-important set. The paper reports that this restores on average 2.19, 3.45, and 0.42 ASR points for Qwen2.5-VL under SNIP, Wanda, and SparseGPT pruning, respectively, with recovery ratios above 14% in all models and above 100% for LLaVA-NeXT-Mistral, and that total Ships across the six models correlates with pruning-induced safety drop at Spearman ρ = 0.8857.

Load-bearing premise

The load-bearing premise is that pruning-induced safety loss is localized to a small set of attention heads and neurons that utility-based importance tends to remove, so restoring exactly those pruned components can recover the lost safety.

Editorial extensions

If this is right

  • Across six LVLMs at 50% sparsity and three pruning methods, HSR lowers attack success rates by 0.42 to 5.46 points and recovers between 14.81% and 104.12% of the safety lost to pruning, restoring only 0.016 to 1.803 ten-thousandths of the pruned weights.
  • Utility is roughly preserved: MMbench and DocVQA scores change by at most about two points, with small gains in Qwen-based models and slight declines in Llama3-based ones.
  • The strong rank correlation (ρ = 0.8857) between total Ships and ASR increase after pruning means the metric can flag in advance which models are most likely to lose safety under utility-based pruning.
  • The ablation shows that restoring entire attention heads is both heavier (18 to 650 times more parameters) and less effective, and can even worsen safety, because some neurons inside safety-critical heads themselves reduce safety.
  • HSR also transfers to text-only LLMs: it recovers 33.33% and 21.05% of lost safety for Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct pruned with Wanda at 50% sparsity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If safety-critical neurons are this sparse and this identifiable, the cleanest fix may be to exclude them from the pruning mask in the first place, preempting the safety drop instead of restoring it afterwards.
  • The existence of neurons that actively hurt safety when restored suggests a two-sided mask: pruning could keep the safety-positive neurons and deliberately remove the safety-negative ones, potentially improving safety beyond the full model.
  • The strong Ships-to-degradation correlation offers a cheap pre-deployment diagnostic: measure total Ships on the full model to predict which pruned configuration will need realignment before committing to deployment.
  • The same hierarchical attribution logic could be tested on other alignment properties, such as factuality or instruction following, in pruned multimodal models, to see whether those are also carried by sparsely localizable components.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes Hierarchical Safety Realignment (HSR), a two-stage method to recover safety lost by pruning LVLMs: it first identifies safety-critical attention heads using the Ships score from Zhou et al. (2025), then restores a small set of pruned neurons within those heads that have high safety importance and sufficient utility importance. Experiments on six LVLMs under Wanda, SNIP, and SparseGPT pruning at 50% sparsity, plus 2:4 structured pruning and sparsity variations, report average ASR reductions and RSR values of 14–104%, with restored parameters of 0.016‰ to 1.803‰ of the pruned weights. The paper also reports an ablation restoring whole heads (HSR-a), hyperparameter sensitivity, a Spearman correlation between total Ships and ASR drop, and an extension to text-only LLMs.

Significance. If the central mechanism claim were established, HSR would be a useful and lightweight post-pruning safety realignment method for LVLMs, and the paper would be the first to address this specific problem; the breadth of the evaluation (six LVLMs, three pruning methods, structured pruning, sparsity levels, and an LLM extension) is a genuine strength, as is the use of safety benchmarks separate from the calibration data. The paper also candidly acknowledges its main limitations, including some utility loss and weaker results on Llama3-based models. However, the missing equal-budget control and the low statistical resolution of the safety metric mean that the current evidence is directionally suggestive rather than conclusive for the paper's central mechanistic claim.

major comments (4)
  1. [§4, Tables 1–2 and Table 8] The central claim that safety recovery is due to hierarchical safety-critical selection is not yet isolated from a generic capacity effect. The only ablation, HSR-a, restores 18–650 times more parameters (Table 8), so it does not control for restoration count. No experiment restores the same number of randomly chosen pruned neurons, or the same number chosen by utility importance alone. Without such an equal-budget baseline, the observed ASR gains could in principle be obtained by restoring any small set of pruned weights; the paper's premise that safety-critical neurons are underrepresented in utility importance is therefore untested. Please add a random-restoration control matched in parameter count and report per-prompt significance.
  2. [§4, Table 1; Appendix A, Table 9] Several headline quantitative claims sit within a few examples of evaluation noise. The safety benchmarks contain 500 and 487 prompts (Table 9), so Qwen2.5-VL SparseGPT ASR 3.10 to 2.68 is roughly 4 changed examples and the reported 34.43% RSR is driven by a 1.22-point denominator. LLaVA-NeXT-Mistral's 104.12% RSR in Table 2 corresponds to about 10 examples. All experiments are single-run with greedy decoding, so no sampling variance is reported. Please report confidence intervals, multiple seeds, or a paired significance test (e.g., McNemar on per-prompt safety labels) for at least the main tables, and interpret RSR values above 100% as not a literal restoration ratio.
  3. [§4, Hyperparameter Effects, Tables 5–7] The manuscript does not state how h, q, and pmax are chosen for each model and experiment. The hyperparameter study is conducted on one model (LLaVA-NeXT-Mistral) and reports trends, but the main results in Tables 1–3 do not disclose the values used, whether they are fixed defaults, or whether they were tuned on SafeBench/Ch3Ef. If the latter, the reported RSRs are in-sample and the generalization claim is weakened. Please specify the protocol (e.g., fixed values for all models, or a held-out validation split) and provide sensitivity results for all models.
  4. [§5, Eq. (13) and Appendix C, Table 12] The Spearman correlation of 0.8857 between total Ships and pruning-induced ASR increase is computed on only six models. No p-value, permutation test, or confidence interval is reported; with n=6 the correlation is borderline at conventional significance levels even if the point estimate is high. This evidence should be described as suggestive, not as confirmation of the causal mechanism.
minor comments (8)
  1. [Eq. (9)] In the definition of Ss(q), the condition appears to use Iu_{i,j} where Is_{i,j} is intended; as written, the top-q safety set is defined by utility scores.
  2. [Table 7 caption] The caption reads "where pmax = 4, q = 0.5", but pmax is elsewhere a quantile in [0,1]; 4 is presumably a typo and the actual value should be stated.
  3. [Table 8] The ablation table labels the method "RSAC" while the rest of the paper uses "HSR"; the naming should be unified.
  4. [References] The SNIP pruning method appears twice, as Lee et al. (2018) and Lee et al. (2019); these are the same paper and should be merged.
  5. [Introduction and Appendix B] Minor language issues: "apporach" in the Introduction and "as showen" in Appendix B should be corrected.
  6. [Appendix E/F] The caption for Figure 6 is duplicated from Figure 5 ("Example image for the harmful instruction") and does not match the line chart it accompanies; Appendix E's figure needs its own descriptive caption.
  7. [§3 and §2.2.2] Section 3 describes the safety dataset as "Unsafe-Unsafe pairs (unsafe images with matching unsafe instructions)", while Section 2.2.2 and Figure 2 state that the safety dataset contains malicious instructions paired with appropriate refusal responses; please clarify which responses are used when computing safety importance.
  8. [§4, Table 3] For 2:4 structured pruning, the paper does not explain how restoring individual pruned neurons interacts with the 2:4 mask; if the restored weights break the structured pattern, the resulting model is no longer 2:4-sparse and the practical claim needs qualification.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HSR's safety-critical selection uses VLGuard calibration data, while safety gains are measured on held-out SafeBench and Ch3Ef benchmarks, so the central claim does not reduce to its inputs by construction.

full rationale

The paper's central claim is that restoring a very small set of safety-critical attention heads and neurons, identified from VLGuard-derived importance scores, recovers a substantial fraction of pruning-induced safety loss. This claim is not circular: the safety importance scores are computed on the VLGuard safety dataset, whereas the reported safety improvements are evaluated on SafeBench-mini and the harmful subset of Ch3Ef under zero-shot greedy decoding. These evaluation benchmarks are not the same data used to select the neurons, so the improvement is an empirical finding rather than a consequence of the selection definition. The Ships metric and the neuron-level importance scoring are imported from prior work by other authors (Zhou et al. 2025; Wei et al. 2024), not from the present authors' own prior results, so there is no self-citation chain carrying the argument. The 'small subset of neurons is disproportionately important' claim is an interpretive summary of experiments, and the absence of an equal-budget random-restoration baseline is a legitimate experimental-rigor concern, but it is not a circularity: a random-restoration baseline would test whether the selection mechanism matters, not whether the evaluation is independent. The RSR metric and restoration ratios are reporting quantities, not fitted parameters that are later relabeled as predictions. Minor typographical issues, such as Eq. 9 writing Iu in the definition of Ss(q) and the Table 7 caption listing 'pmax = 4', do not create a self-definitional or fitted-input reduction. Overall, the derivation chain is self-contained against external benchmarks, and no load-bearing step reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method depends on assumptions about localization, transfer, and safety/utility separability that are not independently verified. The only tuned quantities are the q, pmax, and h hyperparameters. No new physical or architectural entities are introduced.

free parameters (3)
  • h (number of top safety-critical heads) = 4 for LLaVA-NeXT-Mistral; per-model values not fully specified
    Controls how many attention heads receive neuron-level analysis; selected based on benchmark ablations showing best safety at h=4.
  • q (safety importance quantile threshold) = 0.5 default; ablated from 0.35 to 0.50
    Defines the top fraction of safety-important weights eligible for restoration; tuned on safety evaluation data.
  • pmax (utility importance upper quantile threshold) = 0.7 default; ablated from 0.51 to 1.00
    Defines which restored neurons retain enough utility; tuned on safety and utility evaluation data.
assumptions (4)
  • domain assumption Safety behavior in LVLMs is localized in a small set of attention heads and neurons that can be identified by ablation and importance scoring.
    Introduction and Section 2; if safety were distributed across the network, selective restoration could not recover the lost safety.
  • domain assumption The VLGuard calibration subsets are representative of the safety and utility distributions in the evaluation benchmarks.
    Section 3 dataset description; no cross-domain validation is provided to show that importance scores transfer.
  • domain assumption Safety importance and utility importance are sufficiently disentangled so that the set difference Ss(q) ∩ Su(pmax) - Su(p) isolates safety-specific neurons without major utility loss.
    Section 2.2.2 and Appendix E; the paper itself acknowledges entanglement between safety and utility, making this assumption load-bearing.
  • domain assumption Head masking with a small epsilon approximates causal removal of a head's contribution, and SVD-based singular angles capture safety-relevant representation change.
    Section 2.1, Eqs. 1-4; the Ships metric is inherited from Zhou et al. (2025) without re-validation on LVLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models." pith.science (2026). https://pith.science/paper/O7RFFMVI

@misc{pith2026250516104,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O7RFFMVI}},
  note         = {Machine review of arXiv:2505.16104}
}
read the original abstract

With the increasing size of Large Vision-Language Models (LVLMs), network pruning techniques aimed at compressing models for deployment in resource-constrained environments have garnered significant attention. However, we observe that pruning often leads to a degradation in safety performance. To address this issue, we present a novel and lightweight approach, termed Hierarchical Safety Realignment (HSR). HSR operates by first quantifying the contribution of each attention head to safety, identifying the most critical ones, and then selectively restoring neurons directly within these attention heads that play a pivotal role in maintaining safety. This process hierarchically realigns the safety of pruned LVLMs, progressing from the attention head level to the neuron level. We validate HSR across various models and pruning strategies, consistently achieving notable improvements in safety performance. To our knowledge, this is the first work explicitly focused on restoring safety in LVLMs post-pruning.

Figures

Figures reproduced from arXiv: 2505.16104 by the authors.

Figure 1
Figure 1. The Average ASR (lower values indicate higher safety) of full baselines versus pruned models (50% sparsity) across safety evaluation benchmarks. The abbreviations on the x-axis, listed from left to right, correspond to the following models: Qwen2-VL, Qwen2.5-VL, LLaVA-NeXT-Mistral, LLaVA-NeXT￾Vicuna, LLaVA-NeXT-Llama3, and Llama3.2-Vision. et al., 2024; Frantar and Alistarh, 2023; Lee et al., 2018) compute importanc… view at source ↗
Figure 2
Figure 2. HSR hierarchically achieves safety realignment of the pruned model in two steps: The first step identifies [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Results after pruning with 50% sparsity using Wanda and HSR realignment, with the classification here [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average ASR increase and ranking of total [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Example image for the harmful instruction. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Example image for the harmful instruction. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 16 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895--4901

    work page 2023

  4. [4]

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865--10873

    2024

  5. [5]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/f545448535dfde4f9786555403ab7c49-Paper-Conference.pdf Refusal in language models is mediated by a single direction . In Advances in Neural Information Processing Systems, volume 37, pages 1360...

    work page 2024

  6. [6]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. https://openreview.net/forum?id=vXxardq6db Slice GPT : Compress large language models by deleting rows and columns . In The Twelfth International Conference on Learning Representations

    work page 2024

  7. [7]

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2024. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations

    work page 2024

  8. [8]

    Jianjian Cao, Peng Ye, Shengze Li, Chong Yu, Yansong Tang, Jiwen Lu, and Tao Chen. 2024. Madtp: Multimodal alignment-guided dynamic token pruning for accelerating vision-language transformer. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15710--15719. IEEE Computer Society

    work page 2024

Show all 32 references
  1. [9]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com...

    2019

  2. [10]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323--10337

    2023

  3. [11]

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. 2025. Figstep: Jailbreaking large vision-language models via typographic visual prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, page...

    2025

  4. [12]

    Adib Hasan, Ileana Rugina, and Alex Wang. 2024. Pruning for protection: Increasing jailbreak resistance in aligned llms without fine-tuning. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 417--430

    2024

  5. [13]

    Tian Jin, Michael Carbin, Dan Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. 2022. Pruning’s effect on generalization through the lens of training and regularization. Advances in Neural Information Processing Systems, 35:37947--37961

    2022

  6. [14]

    Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79--86

    1951

  7. [15]

    N Lee, T Ajanthan, and P Torr. 2019. Snip: single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations

    2019

  8. [16]

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340

    2018 arXiv

  9. [17]

    Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. 2024 a . A survey of attacks on large vision-language models: Resources, advances, and future trends. arXiv preprint arXiv:2407.07403

    2024 arXiv

  10. [18]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2025. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216--233

    2025

  11. [19]

    Zhendong Liu, Yuanbi Nie, Yingshui Tan, Xiangyu Yue, Qiushi Cui, Chongjun Wang, Xiaoyong Zhu, and Bo Zheng. 2024 b . Safety alignment for vision language models. arXiv preprint arXiv:2405.13581

    2024 arXiv

  12. [20]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200--2209

    2021

  13. [21]

    Zhelun Shi, Zhipin Wang, Hongxing Fan, Zaibin Zhang, Lijun Li, Yongting Zhang, Zhenfei Yin, Lu Sheng, Yu Qiao, and Jing Shao. 2024. Assessment of multimodal large language models in alignment with human values. arXiv preprint arXiv:2403.17830

    2024 arXiv

  14. [22]

    C Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72--101

    1904

  15. [23]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. https://openreview.net/forum?id=PxoFut3dWW A simple and effective pruning approach for large language models . In The Twelfth International Conference on Learning Representations

    2024

  16. [24]

    Yi-Lin Sung, Jaehong Yoon, and Mohit Bansal. 2024. https://openreview.net/forum?id=iIT02bAKzv EC o FL ap: Efficient coarse-to-fine layer-wise pruning for vision-language models . In The Twelfth International Conference on Learning Representations

    2024

  17. [25]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 a . Qwen2-vl: Enhancing vision-language m...

    2024 arXiv

  18. [26]

    Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, and Chaowei Xiao. 2024 b . Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting. In European Conference on Computer Vision, pages 77--94

    2024

  19. [27]

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Proceedings of the 41st International Conference on ...

    2024

  20. [28]

    Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jinyang Guo, Wenbo Zhou, Xianglong Liu, and Dacheng Tao. 2024 a . Safebench: A safety evaluation framework for multimodal large language models. arXiv preprint arXiv:2410.18927

    2024 arXiv

  21. [29]

    Zonghao Ying, Aishan Liu, Tianyuan Zhang, Zhengmin Yu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. 2024 b . Jailbreak vision language models via bi-modal adversarial prompt. arXiv preprint arXiv:2406.04031

    2024 arXiv

  22. [30]

    Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li. 2025. https://openreview.net/forum?id=h0Ak8A5yqw On the role of attention heads in large language model safety . In The Thirteenth International Conference on Lear...

    2025

  23. [31]

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2024. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556--1577

    2024

  24. [32]

    Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. 2024. Safety fine-tuning at (almost) no cost: A baseline for vision large language models. In International Conference on Machine Learning, pages 62867--62891

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.