Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Security tensors route text safety into vision-language models.

desk verdict A solid, practical defense that likely works, but the mechanistic 'safety layer activation' story is overclaimed and the benign-case FRR on text-similar queries deserves a closer look before this is adopted as a general-purpose safeguard. read the letter →

arxiv 2507.20994 v1 pith:F3SP7WFN submitted 2025-07-28 cs.CV cs.AI

Shen Li , Liuyi Yao , Wujia Niu , Lan Zhang , Yaliang Li This is my paper
classification cs.CVcs.AI
keywords securitytensorsLVLMsafetycross-modalalignmentlayersvisualjailbreakdefenseparameter-freeinputperturbationmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a learned input vector, called a security tensor, can extend the text-only safety alignment of a large vision-language model (LVLM) to the visual modality without touching the model's weights. Applied at inference to either the image tokens or the text tokens, the tensor makes the model refuse harmful image-text queries while leaving benign answers nearly unchanged. The authors support this with experiments on three LVLMs, where the tensors raise the harmless rate on both seen and unseen harmful image categories and keep false rejections and benchmark scores close to the base model. They also present layer-wise evidence that the tensors re-engage the same intermediate layers in the language module that already distinguish malicious text from benign text. If correct, this gives a lightweight, parameter-free route to visual safety that leverages safety the model already has.

What carries the argument

The load-bearing mechanism is the security tensor, a trainable input-level perturbation applied at inference time in one of two modalities: a textual tensor $\delta_t$ inserted between the image and text token embeddings, or a visual tensor $\delta_v$ added to the standardized preprocessed image. The tensor is optimized on three curated sets: safety activation pairs (harmful images with benign text and refusal responses), general benign pairs (to preserve original outputs via KL distillation), and text-contrast benign pairs whose text mirrors the harmful queries' syntax but pairs with benign images, which discourages the tensor from latching onto surface-level textual patterns. The claim is that the optimized tensor shifts harmful visual inputs into the semantic region where the language module's safety layers become active, reactivating the model's pre-trained refusal behavior.

What would settle it

A control experiment would train security tensors with the same magnitude constraint but with rejection labels attached to random images rather than harmful ones, or use random Gaussian tensors in place of the learned ones; if such non-specific tensors produce the same harmless-rate gains and the same layer 9–20 gap in the hidden-layer analysis, then the measured effect is generic input perturbation, not activation of the text safety layers.

Watch

Extended reading notes

Core claim

The central discovery is that the language module of an LVLM already contains the machinery to reject harmful content, and that machinery can be switched on for visual inputs by adding a small learned perturbation to the input representation. The paper calls these perturbations security tensors and shows that a single tensor can be learned for either the textual embedding sequence (inserting virtual tokens between image and text embeddings) or the preprocessed image tensor (adding to the standardized image representation). On three tested vision-language models, both versions substantially increase the harmless rate on harmful visual categories, including categories never seen in training, while the false rejection rate on benign queries stays low and multimodal benchmark scores drop only slightly. Hidden-layer analysis shows that without the tensor, the model's text safety layers respond to malicious text but not to malicious images; with the tensor, the response gap for harmful versus benign inputs emerges in the same layers, indicating that the tensor aligns harmful visual representations with the semantic space the language module already treats as unsafe.

Load-bearing premise

The central premise is that the model's refusal behavior for unsafe text comes from specific internal circuits that stay dormant for unsafe images and can be turned on by a small input tweak; if that causal story is wrong, the observed rejection gains might just be a general tendency to refuse more whenever any extra vector is added.

Editorial extensions

If this is right

  • A frozen LVLM can be made to reject harmful visual inputs, including unseen harmful categories, using only a learned input vector, with no parameter updates and minimal degradation on benign tasks.
  • The method's gains track the strength of the language module's own textual safety: models with stronger text alignment show larger harmless-rate improvements from the tensors.
  • The text-contrast benign set is necessary: without it, the tensors overfit to textual patterns, hurting generalization and raising false rejections on benign queries that resemble malicious prompts.
  • Both textual and visual security tensors activate the same layer range as the language module's textual safety layers, suggesting the effect is cross-modal reactivation of existing circuits rather than input filtering or output post-processing.
  • The visual tensor's placement in the preprocessed image space lets it adapt to arbitrary input resolutions across different LVLM architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to apply the same tensor recipe to other modalities, such as audio or video, by locating the corresponding input representation where a learned perturbation can push harmful inputs into the language module's safe semantic space; the paper's layer-overlap analysis provides a check for whether the mechanism truly transfers.
  • The residual false rejection on text-similar benign queries suggests that the tensor reduces but does not eliminate textual over-reliance, so combining the tensor with a lightweight text-similarity detector could further cut these over-rejections.
  • If the safety-layer activation story is correct, the security tensor is effectively a prompt in representation space rather than in token space, which opens the possibility of composing multiple tensors for different safety dimensions without retraining the model.
  • The method's generality depends on the language module having meaningful safety layers to begin with; on models with weak text alignment, the tensor may mainly supply a generic refusal bias, so the practical procedure should first measure the text-layer gap before expecting cross-modal transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes 'security tensors'—trainable input perturbations injected into either the visual preprocessing space (δv) or the text embedding sequence (δt)—as a parameter-free-at-inference method to make frozen LVLMs reject harmful image-text queries. The tensors are optimized on a curated dataset with three components: Safety Activation pairs (harmful image + benign text + rejection output), General Benign pairs (distillation of original outputs), and Text Contrast Benign pairs (benign images paired with text syntactically similar to the harmful queries). Experiments on LLaMA-3.2-11B-Vision, Qwen-VL-Chat, and LLaVA-1.5 report substantial gains in Harmless Rate on both seen and unseen harmful categories compared with the base models and with baselines, at low False Rejection Rates on a general benign test set. A layer-wise cosine-similarity analysis shows that, at baseline, the language module differentiates malicious from benign text-only queries in layers 9–20 but not for image-text queries; adding the security tensors makes the representation shift for malicious image-text queries diverge from benign ones in the same layer range, which the authors interpret as reactivating the language module's textual 'safety layers'.

Significance. If the empirical results hold, this is a practical and lightweight safety intervention for LVLMs: it does not fine-tune the base model, requires only 1,000 training samples, and shows generalization to harmful categories not seen during training. The TCB ablation is a genuine strength: it demonstrates that the contrastive benign set materially reduces over-rejection on TCB-style queries (FRR drops from 90–99% to 4.5–38%) and improves unseen-category HR, supporting the claim that the tensors learn visual cues rather than shallow text patterns. The paper also releases code and data, which aids reproducibility. However, the central mechanistic claim—that the tensors specifically activate the language module's textual safety layers—is supported only by correlational representation-similarity evidence, and the residual 35–38% FRR on TCB-style benign queries (Appendix Table 4) substantially tempers the paper's benignness claims. The work is nevertheless a useful contribution to inference-time visual safety if the mechanistic interpretation is appropriately weakened.

major comments (4)
  1. [Section 5.2, Figures 2–3] The central claim that security tensors 'activate the language module's textual safety layers' is under-supported. The evidence is that the layer range where the N–M vs. N–N gap appears under δ (the STA layers) overlaps with the 9–20 range identified for text-only safety layers in Section 5.1. This is a correlational observation: a divergence in representation shift under a perturbation is equally consistent with a generic refusal bias that routes harmful-image inputs to any refusal-associated region, without the language module executing the same safety computation as for malicious text. To substantiate the causal interpretation, the authors should either (i) intervene on the identified layers (e.g., ablate or patch the safety-layer representations) and show that refusal behavior changes accordingly, or (ii) demonstrate that the representation shift under δ is in the same direction as the text-only N–M shift, not merely that a gap appears. Without such evidence, the phrase 'successfully activate the safety layers' should be softened to a hypothesis.
  2. [Appendix A.1.4, Table 4] The benignness claim is significantly weakened by the TCB-style test set results. ST-δv and ST-δt achieve FRR of 35.00% and 38.00% on LLaMA-3.2-11B-Vision for benign queries whose text resembles harmful requests, compared with 7.75% and 0.50% on the general benign test set. This means that over one-third of such benign inputs are rejected, which is not 'negligible degradation' or 'minimal over-restriction' as stated in Sections 1 and 4.2. While the TCB set reduces over-rejection relative to the no-TCB variants (93% and 91.25%), the residual rate remains far above the general-benign FRR and is likely to be practically problematic for real deployments. The authors acknowledge this in the appendix and leave it to future work, but the main paper's framing should be revised to accurately represent this limitation, and the TCB-style FRR should be reported in the main results rather than relegated to an appendix.
  3. [Section 4.1, 'Evaluation Metrics in Security'] The refusal-detection protocol is unspecified. The paper defines Harmless Rate as 'the proportion of queries that the LVLM successfully refuses to answer' and False Rejection Rate as 'the proportion of benign queries that are wrongly rejected', but it does not state how refusal is determined: keyword matching, a classifier, or manual review. With three different model families and five methods, the absence of a consistent protocol makes HR and FRR non-reproducible and creates a risk that baseline comparisons are affected by differing output styles. Additionally, no error bars, confidence intervals, or repeated-run statistics are reported anywhere, although δ is initialized randomly (Section 3.3.2) and training involves stochastic optimization; given the small training set (1,000 samples) and the variance in harmful-image content, the reported HR differences between methods (e.g., 84.23 vs. 81.89 for ST-δv vs. ST-δt on LLaMA-3.2) may not be significant.
  4. [Sections 1, 3.3, and 3.3.2] The terminology 'parameter-free' and 'black-box' is misleading. The security tensors δv and δt are trainable parameters with thousands of elements (e.g., δv ∈ R^{4×560×560×3} for LLaMA-3.2), and they are optimized by backpropagating cross-entropy and KL losses through the LVLM (Equations 5–7). This requires white-box access to the model's gradients and internal representations, so 'black-box' is accurate only in the narrow sense that the base model's weights are not updated. Calling the framework 'parameter-free' (Section 1) is incorrect: it introduces a large number of new parameters, even if they are not part of the base model. The authors should replace these terms with precise descriptions, e.g., 'frozen-model' or 'input-perturbation' methods, and clarify the access requirements in the experimental setup.
minor comments (4)
  1. [Throughout] There are numerous typos and minor writing issues, e.g., 'benigh responses' (Section 3.3.1), 'image sapce' (Section 3.3.1), 'an bridge' (Section 1), and 'δx' should be 'δt' in Section 3.3.2 ('Both δv and δx are initialized'). A careful proofreading pass is needed.
  2. [Appendix A.1.1] The sentence 'The dataset was disrupted during training' is unclear; it should specify whether the data are shuffled, batched, or sampled with a particular strategy, as this affects reproducibility.
  3. [Section 5.1 and Figure 2] The claim that the N–N and N–M curves 'diverge around layer 9' and 'reach its peak near layer 20' is based on visual inspection of averaged cosine similarities from 100 pairs. The authors should report quantitative measures (e.g., the maximum gap, the layer index of the maximum, and variability across pairs) and ideally a statistical test that the gap is non-zero; otherwise the 'safety layer' range is anecdotal.
  4. [Section 4.2, Table 1] The test-set composition is underspecified: the paper says it combines 'unsafe class inputs from VLGuard and MM-SafeBench' but does not report the number of samples per category, the prompt templates used, or how the seen/unseen split is exactly defined beyond partial category overlap. Adding a table with per-dataset and per-category sample counts would aid reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Empirical safety gains are held-out and non-circular, but the central 'safety-layer activation' mechanism borrows its key construct from the authors' prior ICLR paper, making the mechanistic claim partially self-citational.

  1. self citation load bearing [Section 5.1-5.2, 'Language Module "Safety Layers"' and 'Security Tensors can Help Activate the Internal "Safety Layers"'; Ref. [16]]
    "This pattern indicates that the language module's safety layers are active within approximately layers 9–20, playing a critical role in recognizing malicious textual semantics [16] ... The exact overlap between STA layers and textual safety layers provides strong evidence that security tensors successfully activate and extend the language module's inherent textual safety mechanisms into the visual modality."

    The term 'safety layers' is not independently defined or established in this paper; it is imported from Ref. [16], which shares four of the five authors (Shen Li, Liuyi Yao, Lan Zhang, Yaliang Li). Section 5.1 reproduces the prior similarity-curve analysis and labels the 9-20 divergence 'safety layers' only by citing [16]. Section 5.2 then finds that the delta-induced STA divergence also lies in layers 9-20 and concludes the tensors 'activate' those safety layers. The overlap is an empirical observation and is not optimized into Eq. (7), so the held-out Harmless Rate gains are not circular.

full rationale

The paper's primary empirical contribution is not circular: security tensors are trained with the objective in Eqs. (5)-(7) on the SA, GB, and TCB sets, and then evaluated on held-out VLGuard and MM-SafeBench subsets, including harmful categories unseen in training and text prompts distinct from training. The reported Harmless Rate improvements are therefore genuine out-of-sample predictions rather than fitted quantities. The ablation and the Appendix A.1.4 FRR measurements likewise report fresh test sets; the Appendix even honestly discloses a high 35-38% false rejection rate on TCB-style benign queries for LLaMA-3.2, a limitation but not a circularity. The only load-bearing dependence on the authors' own prior work is the interpretation of the layer-wise analysis: Section 5.1 labels layers 9-20 as 'safety layers' based on Ref. [16] (same research group), and Section 5.2 interprets the observed overlap of the STA layers with that range as 'activating' those safety layers. The similarity-curve evidence itself is correlational, but correlation is not circularity unless the construct is defined by the outcome. Because the safety-layer construct is borrowed from the authors' own unverified-by-independent-means prior paper, the mechanistic bridge claim rests on a self-citation chain; the central safety numbers do not. This warrants a moderate circularity score rather than a high one.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method relies on learned input perturbations (the security tensors), whose magnitude and token count are hand-chosen, and on the domain assumption that text safety layers exist and can be activated. The TCB dataset is introduced as a contrastive tool, but its effectiveness is partial: false rejections on text-similar benign queries remain high.

free parameters (3)
  • Visual perturbation threshold λ = 1 (all models)
    Chosen by hand to constrain δv magnitude; no sensitivity analysis is reported (Appendix A.1.1).
  • Number of virtual tokens n for δt = 300 (LLaMA-3.2), 100 (LLaVA-1.5, Qwen-VL-Chat)
    Selected per model based on Appendix A.1.3 loss curves; trade-off between capacity and overfitting.
  • Loss weighting between L_SA and L_GB/TCB = Equal weights (implicit)
    No ablation of the safety-benignness trade-off is reported; Eq. 7 uses a simple sum.
assumptions (3)
  • domain assumption The LVLM's language module contains pre-trained textual safety layers that are identifiable and can be activated by input perturbations.
    Central hypothesis from Section 5.1, resting on the authors' prior work [16]; not independently established here.
  • domain assumption A universal perturbation in the preprocessed image space generalizes across input resolutions and image categories.
    Section 3.2 assumes the preprocessing function φ maps arbitrary inputs to a fixed space where a single δv is effective.
  • domain assumption The TCB contrastive dataset is sufficient to suppress text-pattern overfitting.
    Section 3.3.1 asserts this, but Table 4 shows FRR on TCB-style benign queries remains high (35-38% for LLaMA), so the assumption only partially holds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM." pith.science (2026). https://pith.science/paper/F3SP7WFN

@misc{pith2026250720994,
  author       = {Pith},
  title        = {Pith review of: Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3SP7WFN}},
  note         = {Machine review of arXiv:2507.20994}
}
read the original abstract

Large visual-language models (LVLMs) integrate aligned large language models (LLMs) with visual modules to process multimodal inputs. However, the safety mechanisms developed for text-based LLMs do not naturally extend to visual modalities, leaving LVLMs vulnerable to harmful image inputs. To address this cross-modal safety gap, we introduce security tensors - trainable input vectors applied during inference through either the textual or visual modality. These tensors transfer textual safety alignment to visual processing without modifying the model's parameters. They are optimized using a curated dataset containing (i) malicious image-text pairs requiring rejection, (ii) contrastive benign pairs with text structurally similar to malicious queries, with the purpose of being contrastive examples to guide visual reliance, and (iii) general benign samples preserving model functionality. Experimental results demonstrate that both textual and visual security tensors significantly enhance LVLMs' ability to reject diverse harmful visual inputs while maintaining near-identical performance on benign tasks. Further internal analysis towards hidden-layer representations reveals that security tensors successfully activate the language module's textual "safety layers" in visual inputs, thereby effectively extending text-based safety to the visual modality.

Figures

Figures reproduced from arXiv: 2507.20994 by the authors.

Figure 1
Figure 1. Examples of image-text query for SA, TCB, and GB ets. Notably, the textual inputs in the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The N-N pairs and N-M pairs analysis in LLaMA-3.2-vision, showing safety layers’ function when processing cross-modal queries. The lower part representing the angular difference of the curves. The gap between these curves reveals the layer-wise ability of the language module to differentiate malicious inputs from benign ones. The corresponding results are in figure 2. Pure-text Modality Result Analysis: A clear dive… view at source ↗
Figure 3
Figure 3. The N-N+δ pairs and M-M+δ pairs analy￾sis in LLaMA-3.2-vision, showing how δt and δv influ￾ences the model’s internal representations across layers. The gap between the two curves quantifies the degree to which δ causes the model to differentiate between benign and malicious inputs at each layer. Our findings reveal that both δv and δt induce less perturbations to the hidden layer outputs for benign image-text input… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Training loss curves for δv and δt across LVLMs. Rows correspond to visual and textual tensor training, with epochs on the horizontal axis and loss values on the vertical axis. Each point on the curve represents the average loss across the SA, TCB, and GB sets within t…
Figure 5
Figure 5. Figure 5: Loss Curves of LLaVA-1.5 δt Training Under n = 10, 100, and 300. (a) n=10 (b) n=100 (c) n=300 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Loss Curves of LLaMA-3.2-11B-Vision δt Training Under n = 10, 100, and 300. We observe that when the number of virtual tokens is set to n = 10, the loss for each dataset split fails to drop below 0.1. For LLaVA-1.5, even after 400 training epochs, the losses on the SA …
Figure 7
Figure 7. Figure 7: Examples of adversarial image-text query examples for SA and new TCB test set. In these [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AttenTrack: Mobile User Attention Awareness Based on Context and External Distractions

    cs.HC 2025-09 conditional novelty 5.0 of 10

    AttenTrack predicts a smartphone user's attention state from context and notification-response features, reaching cold-start F1 up to 80% in leave-one-user-out tests.

Reference graph

Works this paper leans on

36 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [16]

    Safety layers in aligned large language models: The key to LLM security

    Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers in aligned large language models: The key to LLM security. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    Nsfw image classification - resnet50

    Krishna Alagiri. Nsfw image classification - resnet50. https://www.kaggle.com/code/ krishnaalagiri/nsfw-image-classification-resnet50/notebook , 2020. Accessed: [2025-05-09]

    work page 2020

  4. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    2023

  5. [4]

    Dress: Instructing large vision-language models to align and interact with humans via natural language feedback

    Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14239–14250, June 2024

    work page 2024

  6. [5]

    Dress: Instructing large vision-language models to align and interact with humans via natural language feedback

    Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14239–14250, 2024

    work page 2024

  7. [6]

    Figstep: Jailbreaking large vision-language models via typographic visual prompts, 2025

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts, 2025

    work page 2025

  8. [7]

    Kwok, and Yu Zhang

    Yunhao Gou, Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, Dit-Yan Yeung, James T. Kwok, and Yu Zhang. Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation, 2024

    work page 2024

Show all 36 references
  1. [8]

    The llama 3 herd of models, 2024

    Andrew Grattafiori, Abhishek Dubey, Anurag Jauhri, et al. The llama 3 herd of models, 2024

    2024

  2. [9]

    Minillm: Knowledge distillation of large language models, 2024

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models, 2024

    2024

  3. [10]

    Hod: New harmful object detection bench- marks for robust surveillance

    Eungyeom Ha, Heemook Kim, and Dongbin Na. Hod: New harmful object detection bench- marks for robust surveillance. In Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 183–192, 2024

    2024

  4. [11]

    Tit-for-tat: Safeguarding large vision-language models against jailbreak attacks via adversarial defense, 2025

    Shuyang Hao, Yiwei Wang, Bryan Hooi, Ming-Hsuan Yang, Jun Liu, Chengcheng Tang, Zi Huang, and Yujun Cai. Tit-for-tat: Safeguarding large vision-language models against jailbreak attacks via adversarial defense, 2025

    2025

  5. [12]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  6. [13]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  7. [14]

    How does vision-language adaptation impact the safety of vision language models? In The Thirteenth International Conference on Learning Representations , 2025

    Seongyun Lee, Geewook Kim, Jiyeon Kim, Hyunji Lee, Hoyeon Chang, Sue Hyun Park, and Minjoon Seo. How does vision-language adaptation impact the safety of vision language models? In The Thirteenth International Conference on Learning Representations , 2025

    2025

  8. [15]

    The power of scale for parameter-efficient prompt tuning, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021

    2021

  9. [17]

    Improved baselines with visual instruction tuning, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. 11

    2024

  10. [18]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  11. [19]

    Query-relevant images jailbreak large multi-modal models, 2023

    Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. Query-relevant images jailbreak large multi-modal models, 2023

    2023

  12. [20]

    LLaV A-1.5-7B-HF: A hugging face implementation of llava-1.5, 2023

    LLaV A-HF Team. LLaV A-1.5-7B-HF: A hugging face implementation of llava-1.5, 2023. Accessed: 2025-05-15

    2023

  13. [21]

    Teaching small language models to reason

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arXiv preprint arXiv:2212.08410, 2022

    2022 arXiv

  14. [22]

    Llama-3.1-8b-instruct

    Meta AI. Llama-3.1-8b-instruct. https://huggingface.co/meta-llama/Llama-3. 1-8B-Instruct, 2024. Accessed: 2025-05-10

    2024

  15. [23]

    Llama-3.2-11b-vision

    Meta AI. Llama-3.2-11b-vision. https://huggingface.co/meta-llama/Llama-3. 2-11B-Vision, 2024. Accessed: 2025-05-10

    2024

  16. [24]

    Mllm-protector: Ensuring mllm’s safety without hurting performance, 2024

    Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. Mllm-protector: Ensuring mllm’s safety without hurting performance, 2024

    2024

  17. [25]

    On the adversarial robustness of multi-modal founda- tion models, 2023

    Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal founda- tion models, 2023

    2023

  18. [26]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  19. [27]

    Enhancing visual- language modality alignment in large vision language models via self-improvement, 2025

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, and Cao Xiao. Enhancing visual- language modality alignment in large vision language models via self-improvement, 2025

    2025

  20. [28]

    Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting, 2024

    Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, and Chaowei Xiao. Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting, 2024

    2024

  21. [29]

    Break the visual perception: Adversarial attacks targeting encoded visual tokens of large vision-language models, 2024

    Yubo Wang, Chaohu Liu, Yanqiu Qu, Haoyu Cao, Deqiang Jiang, and Linli Xu. Break the visual perception: Adversarial attacks targeting encoded visual tokens of large vision-language models, 2024

    2024

  22. [30]

    Tracking the copyright of large vision- language models through parameter learning adversarial images, 2025

    Yubo Wang, Jianting Tang, Chaohu Liu, and Linli Xu. Tracking the copyright of large vision- language models through parameter learning adversarial images, 2025

    2025

  23. [31]

    Cross-modal safety mechanism transfer in large vision-language models

    Shicheng Xu, Liang Pang, Yunchang Zhu, Huawei Shen, and Xueqi Cheng. Cross-modal safety mechanism transfer in large vision-language models. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  24. [32]

    Jailbreak vision language models via bi-modal adversarial prompt, 2024

    Zonghao Ying, Aishan Liu, Tianyuan Zhang, Zhengmin Yu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. Jailbreak vision language models via bi-modal adversarial prompt, 2024

    2024

  25. [33]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning . PMLR, 2024

    2024

  26. [34]

    Spa-vl: A comprehensive safety preference alignment dataset for vision language model, 2025

    Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, Feng Zhao, Tao Gui, and Jing Shao. Spa-vl: A comprehensive safety preference alignment dataset for vision language model, 2025

    2025

  27. [35]

    Spot risks before speaking! unraveling safety attention heads in large vision-language models, 2025

    Ziwei Zheng, Junyao Zhao, Le Yang, Lijun He, and Fan Li. Spot risks before speaking! unraveling safety attention heads in large vision-language models, 2025

    2025

  28. [36]

    Safety fine-tuning at (almost) no cost: A baseline for vision large language models, 2024

    Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: A baseline for vision large language models, 2024. 12 A Appendix A.1 Experiment A.1.1 Hyperparameter Settings Table 3 presents the hyperparameter settings f...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.