Pith. sign in

REVIEW 4 major objections 6 minor 48 references

VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a small white-box perturbation of a vision-language model's early attention and value layers can hide a chosen image region from the model while leaving the rest of the image's meaning intact.

desk verdict A genuinely new ROI-selective attention+value attack with striking detection drops, but the central information-removal claim outruns the evidence until residual-stream leaks and the single-prompt metric are addressed. read the letter →

arxiv 2507.08982 v1 pith:MCQCOZBN submitted 2025-07-11 eess.IV cs.CVcs.LG

Hanene F. Z. Brachemi Meftah , Wassim Hamidouche , Sid Ahmed Fezza , Olivier Déforges This is my paper
classification eess.IVcs.CVcs.LG
keywords adversarialattackvision-languagemodelsvisualinformationprotectionregionofinterestattentionmanipulationprivacypreservationwhite-boxencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that privacy in vision-language models can be engineered as a selective adversarial attack: by adding a carefully optimized perturbation to an image, a model can be made blind to a designated region of interest while still describing the rest of the image normally. The attack works entirely inside the visual encoder, suppressing the attention weights directed at the region's image patches and shrinking the norm of those patches' value vectors over the first $L_{\max}$ attention blocks. On a 1000-image ImageNet subset, the paper reports ROI detection dropping to 1.7% on LLaVA, 6.1% on BLIP2-T5, and 10.5% on Instruct-BLIP, with perceptual similarity (SSIM) above 0.75 on the hardest case. If correct, this turns a known weakness---adversarial vulnerability of VLMs---into a privacy tool, and it suggests that early-layer token-to-patch correspondence is what carries visual information in these encoders.

What carries the argument

The central object is the attack objective of Eq. (4): the sum over the first $L_{\max}$ multi-head attention blocks and all heads of (i) the attention weights $A^{(l,h)}(i,j)$ directed at ROI tokens $j$ from every token $i$, and (ii) the $\ell^2$ norms of the ROI rows of the value matrix $V^{(l,h)}$. This sum is minimized with respect to the image perturbation $\delta$, with $L_{\max}$ chosen per model (1 for LLaVA, 24 for the other two). The mechanism relies on the empirical observation that early attention maps are diagonally dominant, meaning each token's representation is still tied to one patch; suppressing attention and value norms at those depths blocks ROI information before global mixing occurs.

What would settle it

Run the attack on a vision encoder whose early attention maps are not diagonally dominant, or push $L_{\max}$ past the depth where diagonal dominance disappears. If ROI detection still drops substantially, the token-to-patch correspondence premise is not the mechanism; if the attack fails exactly at that depth, the premise is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that minimizing two quantities in the visual encoder's early multi-head attention blocks is sufficient to conceal a chosen image region from a VLM: the total attention weight received by ROI tokens, and the $\ell^2$ norms of the ROI tokens' value vectors. The paper formulates this as Eq. (4) and optimizes the perturbation $\delta$ with a sign-gradient step. The result is that the encoder's internal representation no longer carries the ROI content into later layers, so the language model cannot answer questions about it, even though global image semantics remain similar as measured by caption and feature cosine similarity. The paper reads this as evidence that in early ViT layers each token still corresponds to a specific image patch, making token-level suppression a viable mechanism.

Load-bearing premise

The attack works only if, at the layers it targets, each token still corresponds to a single image patch; if the encoder has already mixed patch information there, suppressing token-level attention and value norms cannot remove the ROI content.

Editorial extensions

If this is right

  • ROI detection on the three tested VLMs falls below 11% (1.7% on LLaVA), a reduction of up to 98% relative to un-attacked images.
  • Because the attack only touches the visual encoder, any VLM sharing that encoder is a candidate target without needing to modify the fusion module or language model.
  • Adding the value-norm term to the attention-term objective improves concealment on every tested model, while using only the value-norm term blurs the ROI or substitutes nearby-semantic objects on LLaVA.
  • The unconstrained perturbation setting yields better perceptual quality than a fixed $L_\infty=0.2$ budget on BLIP2-T5 and Instruct-BLIP, since perturbation visibility depends on local texture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: because the attack only touches the visual encoder, a perturbation optimized on one VLM may transfer to other VLMs built on the same CLIP or EVA-CLIP encoder.
  • Not tested in the paper: the per-model $L_{\max}$ calibration means each new encoder needs a depth sweep before the attack is reliable; an automatic detector of the diagonal-dominance depth would be a practical next step.
  • Not tested in the paper: architectures that mix patch information earlier, such as encoders with early global attention or convolution stems, should be inherently more resistant to this attack.
  • Not tested in the paper: the authors' privacy-as-a-spectrum framing suggests the attack could be tuned as a continuous privacy knob, trading detection suppression against caption similarity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VIP, a white-box adversarial attack on vision-language models that conceals a specified region of interest (ROI) by optimizing an additive perturbation so that, in the first L_max multi-head attention blocks of the visual encoder, attention weights targeting ROI tokens and the L2 norms of ROI value vectors are minimized. The attack is evaluated on LLaVA, BLIP2-T5, and Instruct-BLIP using a 1000-image ImageNet subset with bounding boxes as ROIs. Headline results are detection rates of 1.7% (LLaVA), 6.1% (BLIP2-T5), and 10.5% (Instruct-BLIP) for the 'A+V' configuration, with SSIM/LPIPS indicating moderate perceptual distortion. The paper also reports ablations on L_max, lambda_v, the value-norm-only variant, perturbation budgets, run times, attention-map visualizations, and failure cases.

Significance. The idea of using early-layer attention and value manipulations for ROI-level information concealment is a useful contribution to the privacy-attack literature, and the attack is more targeted than whole-image disruption baselines such as PRM. The manuscript ships a public code repository and includes ablations, failure cases, and a comparison to an adapted baseline, which is commendable. If the findings are robust, the method could serve as a practical privacy tool and as a foundation for understanding where ROI information is encoded in VLM visual encoders. However, the current evaluation does not yet establish that the ROI information is actually inaccessible: it relies on a single yes/no prompt and on hyperparameters selected on the same evaluation set.

major comments (4)
  1. [V-B (Table III) and V-C (Table IV)] L_max and lambda_v are selected by ablation on the same 1000-image subset that is then used to report the headline detection rates in Table I. Because the operating point is chosen on the evaluation set, the reported values (1.7% on LLaVA, 6.1% on BLIP2-T5, 10.5% on Instruct-BLIP) are partly fitted and no statement can be made about performance on unseen images. Please report results on a separate validation/test split or use cross-validation, and report the variance over multiple runs or bootstrap intervals.
  2. [IV-D and Table II] The detection metric is a single yes/no visual question ('Is there any [label] in the image?'). A model that develops a generic tendency to answer 'no', or that stops naming objects generally, would register as success even if the ROI content is still recoverable from the final visual features or by other prompts. The paper should demonstrate information absence, e.g., by probing the VLM's visual encoder features for the ROI class, by evaluating multiple prompt formulations (including open-ended questions), and by reporting clean-image detection accuracy for the same queries; the text only says ROIs were 'correctly detected in the absence of attack' without giving numbers.
  3. [III-C, Eq. (4), Fig. 7, Table III] The optimization only suppresses A(i,j) and ||V(j)|| for ROI tokens in the first L_max blocks. ViT blocks also carry residual connections, so ROI patch embeddings are present in the residual stream even if attention is zeroed, and blocks beyond L_max are untouched. The manuscript's own Fig. 7 shows diagonal attention only for layers below 24, and Table III shows a sharp degradation at L_max=32 (65.4% for BLIP2-T5 and 79.1% for Instruct-BLIP), indicating that the effect is tied to a narrow range of attacking depths. The claim that ROI information is 'prevented from propagating' therefore needs direct support, e.g., attention rollout that includes residual contributions or a linear probe on the encoder output, rather than raw attention maps.
  4. [V-A, Table II] Similarity scores in Table II are computed only on the subset of images where the model failed to detect the ROI. This conditions the similarity measure on attack success and can hide cases where the attack succeeds by destroying the image or by inducing a generic refusal. Please report similarity statistics on the full set as well, and specify the number of images in each subset used for the conditional calculation.
minor comments (6)
  1. [III-B, Eq. (1)] The threat model formulates a min-max problem involving textual prompts and descriptors, but the actual optimization in Eqs. (3) and (4) and Algorithm 1 does not use any text. Please align the threat model with the implemented objective or clarify that Eq. (1) is a conceptual idealization.
  2. [IV-E and Table IV] The implementation sets lambda_v to 1, while Table IV shows that lambda_v = 0.1 gives slightly lower detection on both BLIP2-T5 and Instruct-BLIP; the main results are therefore not at the best observed operating point. Please state whether this was a deliberate choice for robustness across settings.
  3. [V-B, Fig. 7] The hypothesis about token-patch correspondence in early layers is validated only on BLIP2-T5; similar evidence for Instruct-BLIP and LLaVA would strengthen the generalizability of the L_max selection.
  4. [Fig. 2 and Algorithm 1] Figure 2 states that the attack 'uses only the visual encoder', but Algorithm 1 line 12 calls ExtractHiddenStates(Mv(xadv)) on the full model's visual encoder; please clarify whether the projector and LLM are completely unused, or whether hidden states are obtained from the encoder alone.
  5. [Abstract and throughout] There are typographical issues, including 'this widespread raises serious concerns' in the abstract and the inconsistent spacing in 'LLaV A'; a careful proofread is needed.
  6. [Tables III and IV] All numbers are reported as single point estimates without standard deviations, confidence intervals, or the number of seeds; given the early-stopping and Adam-based optimization, some measure of run-to-run variability would help the reader assess the stability of the reported detection rates.

Circularity Check

2 steps flagged · score 2.0 of 10

Central claim is not circular by construction because the yes/no detection metric is external to the optimization objective, but two minor issues (the Fig. 6 attention-map 'confirmation' is the optimized objective itself, and the per-model L_max is tuned on the same test set that yields the headline numbers) cap the score at 2.

  1. self definitional [Section V-A, Fig. 6 analysis (p. 8)]
    "The attention maps visualized in Fig. 6 are generated using the attention rollout method [48] and compared with the attention maps derived from the same layers when processing a clean image. This analysis reveals a clear shift in attention away from the ROI in the adversarial examples, further confirming the effectiveness of our attack in achieving the desired information concealment."

    The quantity presented as confirmation is exactly the quantity the attack minimizes. Equation (4) defines delta as the argmin over the sum of attention weights A(l,h)(i,j) for all j in S_ROI (plus ROI value norms) across the first L_max blocks, so after gradient descent the rolled-out attention from those layers will, by construction, show reduced weight on ROI tokens. Citing that shift as evidence that 'information concealment' was achieved is self-definitional: the evidence equals the input objective, making the confirmation tautological. This step is not load-bearing, because the paper's headline results (1.7%-10.5% detection) rest on the external yes/no VLM answer, which is not part of Equation (4); the tautology only affects the secondary mechanistic 'confirmation' in Fig. 6.

  2. fitted input called prediction [Section V-B / Table III and Section IV-E (Implementation details)]
    "We conducted an ablation study by progressively increasing the value of Lmax to identify the optimal number of attackable layers. We then selected the configuration that offered a satisfactory trade-off between detection accuracy and similarity scores, while preserving overall image quality. ... The number of attackable layers, Lmax is set to 1, 24, and 24 for the LLaVA, BLIP2-T5, and Instruct-BLIP models, respectively."

    The per-model operating point L_max (1 for LLaVA; 24 for BLIP2-T5 and Instruct-BLIP) is selected from detection and similarity values measured on the same 1000-image ImageNet test set on which the headline Table I numbers are then reported, so the reported 6.10% (BLIP2-T5) and 10.5% (Instruct-BLIP) are partly a function of test-set selection rather than an a-priori configuration. The circularity is only partial: detection is stable across L_max 16-24 on BLIP2-T5 (6.50% to 6.10%), and the chosen point is not even the minimum-detection setting (Instruct-BLIP reaches 6.00% at L_max=1), so the headline numbers are not statistically forced. Still, the favorable sweet spot (and the collapse at L_max=32 to 65.4%/79.1%) is identified on the evaluation set itself.

full rationale

Derivation-chain check: the attack objective in Equation (4) minimizes (i) attention weights from all tokens to ROI tokens and (ii) L2 norms of ROI value vectors, summed over the first L_max MHA blocks of the visual encoder. The reported successes (1.7% on LLaVA, 6.1% on BLIP2-T5, 10.5% on Instruct-BLIP) are measured by the full VLM's answer to the query 'Is there any [label] in the image?' - a quantity that is neither the optimization objective nor a function of Equation (4)'s arguments by construction. The ImageNet bounding-box labels are used only to define the ROI token set and the queried label, never as a loss signal, so the detection drop is genuine external evidence rather than a renamed fit. Direct evidence against self-definitional circularity is the Table V ablation: minimizing only value norms ('V') yields 58.3% detection on LLaVA while 'A+V' yields 1.7%, and 'V'-only visibly blurs the ROI - the objective and the metric do not move together, so the metric is not the objective renamed. Two minor circularity-adjacent issues remain. First, the Fig. 6 'confirmation' of information concealment measures the attention shift away from the ROI, which is precisely the sum being minimized in Equations (3)-(4), making that confirmation tautological; it is not load-bearing because the detection numbers carry the central claim. Second, the per-model L_max and lambda_v are selected using detection and similarity scores computed on the same 1000-image test set that produces the headline tables, so the reported operating points are partly test-fitted; this is mitigated by the stability of the results over L_max 16-24 and by the fact that the chosen L_max is not the minimum-detection configuration. There are no load-bearing self-citations (the reference list contains none by these authors), no imported uniqueness theorems, no ansatz smuggled via citation, and no renamed known result. The skeptic concerns about residual-stream information flow and yes/no response bias are correctness and evaluation-validity risks, not circularity, and are left out of the score.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The attack is an empirical optimization procedure, so no physical constants are derived from first principles and no new entities are postulated. The load-bearing assumptions are architectural: early-layer token-patch locality, known ROI boxes, and yes/no detection as a proxy for concealment. The main free parameters are hyperparameters L_max and lambda_v selected per model on the same evaluation set, which means the headline numbers depend on test-set tuning rather than on a fitted physical constant.

free parameters (2)
  • L_max (number of attacked MHA blocks) = 1 for LLaVA, 24 for BLIP2-T5 and Instruct-BLIP
    Chosen by per-model ablation in Section V-B on the same 1000-image evaluation set; Table III shows L_max=32 degrades detection to 65.4-79.1 percent.
  • lambda_v (value-norm weight) = 1.0 in main experiments; 0.1 is best in Table IV
    Hand-set Lagrangian weight between attention and value-norm losses; ablation shows minor sensitivity for lambda_v greater than 0, but the choice is made on the evaluation set.
assumptions (3)
  • domain assumption Early-layer ViT attention is locally diagonal, so token j in early MHA blocks still carries ROI patch information, and suppressing its attention and value vectors blocks that information.
    Invoked in Section III-C and validated only indirectly by Figure 7 for BLIP2-T5; Table III shows the assumption breaks for L_max=32.
  • domain assumption The ROI is known a priori and provided as bounding-box coordinates.
    The problem statement in Section III-A and Algorithm 1 take boxes as input; the paper does not address automatic discovery of sensitive regions.
  • domain assumption A yes/no answer to 'Is there [label] in the image?' is a faithful test of information concealment.
    Evaluation metric in Section IV-D; a negative answer could stem from linguistic or contextual reasons rather than genuine concealment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models." pith.science (2026). https://pith.science/paper/MCQCOZBN

@misc{pith2026250708982,
  author       = {Pith},
  title        = {Pith review of: VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MCQCOZBN}},
  note         = {Machine review of arXiv:2507.08982}
}
read the original abstract

Recent years have witnessed remarkable progress in developing Vision-Language Models (VLMs) capable of processing both textual and visual inputs. These models have demonstrated impressive performance, leading to their widespread adoption in various applications. However, this widespread raises serious concerns regarding user privacy, particularly when models inadvertently process or expose private visual information. In this work, we frame the preservation of privacy in VLMs as an adversarial attack problem. We propose a novel attack strategy that selectively conceals information within designated Region Of Interests (ROIs) in an image, effectively preventing VLMs from accessing sensitive content while preserving the semantic integrity of the remaining image. Unlike conventional adversarial attacks that often disrupt the entire image, our method maintains high coherence in unmasked areas. Experimental results across three state-of-the-art VLMs namely LLaVA, Instruct-BLIP, and BLIP2-T5 demonstrate up to 98% reduction in detecting targeted ROIs, while maintaining global image semantics intact, as confirmed by high similarity scores between clean and adversarial outputs. We believe that this work contributes to a more privacy conscious use of multimodal models and offers a practical tool for further research, with the source code publicly available at: https://github.com/hbrachemi/Vlm_defense-attack.

Figures

Figures reproduced from arXiv: 2507.08982 by the authors.

Figure 1
Figure 1. Privacy as a spectrum: images fall into different exposure categories. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our proposed attack. We separate the different VLM’s [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the scaled-dot product attention on a single head level. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: A perceptual illustration of the perturbation’s visibility on a few [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: LLaVA generated answers on different user queries. ‘PRM’ and ‘Ours’ refer respectively to the constrained PRM configuration (that corresponds to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Depiction of attention maps extracted from attacked layers for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Averaged attention maps A(L) across multiple heads and multiple images extracted from different MHA blocks L of the BLIP2-T5 visual encoder [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: LLaVA generated answers on a few adversarial examples generated under the ‘V’ setting. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Failure cases of our proposed adversarial attack under the ‘A+V’ setting for LLaVA. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 18 canonical work pages

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023. 12

    arXiv 2023

  2. [2]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    arXiv 2023

  3. [3]

    Pixtral 12b,

    P. Agrawal, S. Antoniak, E. B. Hanna, D. Chaplot, J. Chudnovsky, S. Garg, T. Gervet, S. Ghosh, A. H ´eliou, P. Jacob et al., “Pixtral 12b,” arXiv preprint arXiv:2410.07073 , 2024

    arXiv 2024

  4. [4]

    Aria: An open multimodal native mixture-of-experts model,

    D. Li, Y . Liu, H. Wu, Y . Wang, Z. Shen, B. Qu, X. Niu, G. Wang, B. Chen, and J. Li, “Aria: An open multimodal native mixture-of-experts model,” arXiv preprint arXiv:2410.05993 , 2024

    arXiv 2024

  5. [5]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge et al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

  6. [6]

    Reconstructing training data from diverse ml models by ensemble inversion,

    Q. Wang and D. Kurz, “Reconstructing training data from diverse ml models by ensemble inversion,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2022, pp. 2909–2917

    work page 2022

  7. [7]

    Effective prompt extraction from language models,

    Y . Zhang, N. Carlini, and D. Ippolito, “Effective prompt extraction from language models,” in First Conference on Language Modeling , 2024. [Online]. Available: https://openreview.net/forum?id=0o95CVdNuz

    work page 2024

  8. [8]

    Extracting training data from large language models,

    N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models,” in 30th USENIX security symposium (USENIX Security 21) , 2021, pp. 2633–2650

    2021

Show all 48 references
  1. [9]

    Are aligned neural networks adversarially aligned?

    N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, P. W. W. Koh, D. Ippolito, F. Tramer, and L. Schmidt, “Are aligned neural networks adversarially aligned?” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  2. [10]

    Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,

    E. Shayegani, Y . Dong, and N. Abu-Ghazaleh, “Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/ forum?id=plmBsXHxgR

    2024

  3. [11]

    On the robustness of large multimodal models against image adversarial attacks,

    X. Cui, A. Aparcedo, Y . K. Jang, and S.-N. Lim, “On the robustness of large multimodal models against image adversarial attacks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24 625–24 634

    2024

  4. [12]

    An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models,

    H. Luo, J. Gu, F. Liu, and P. Torr, “An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https: //openreview.net/forum?id=...

    2024

  5. [13]

    As firm as their foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?

    A. Hu, J. Gu, F. Pinto, K. Kamnitsas, and P. Torr, “As firm as their foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?” arXiv preprint arXiv:2403.12693, 2024

    2024 arXiv

  6. [14]

    Inducing high energy-latency of large vision-language models with verbose images,

    K. Gao, Y . Bai, J. Gu, S.-T. Xia, P. Torr, Z. Li, and W. Liu, “Inducing high energy-latency of large vision-language models with verbose images,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https: //openreview.net/forum?id=...

    2024

  7. [15]

    Tafim: Targeted adversarial attacks against facial image manipulations,

    S. Aneja, L. Markhasin, and M. Nießner, “Tafim: Targeted adversarial attacks against facial image manipulations,” in Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIV . Berlin, Heidelberg: Springer- Verlag, 2022...

    2022

  8. [16]

    Adversary for social good: Leveraging adversarial attacks to protect personal attribute privacy,

    X. Li, L. Chen, and D. Wu, “Adversary for social good: Leveraging adversarial attacks to protect personal attribute privacy,” ACM Trans. Knowl. Discov. Data , vol. 18, no. 2, Nov. 2023. [Online]. Available: https://doi.org/10.1145/3614098

    2023 doi

  9. [17]

    m3it: A large-scale dataset towards multi-modal multilingual instruction tuning,

    L. Li, Y . Yin, S. Li, L. Chen, P. Wang, S. Ren, M. Li, Y . Yang, J. Xu, X. Sun et al., “ m3it: A large-scale dataset towards multi-modal multilingual instruction tuning,” arXiv preprint arXiv:2306.04387, 2023

    2023 arXiv

  10. [18]

    Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023,

    W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023,” arXiv preprint arXiv:2305.06500 , vol. 2, 2023

    2023 arXiv

  11. [19]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

    2023

  12. [20]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592 , 2023

    2023 arXiv

  13. [21]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  14. [22]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023) , vol. 2, no. 3, p. 6, 2023

    2023

  15. [23]

    Eva: Exploring the limits of masked visual representation learning at scale,

    Y . Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y . Cao, “Eva: Exploring the limits of masked visual representation learning at scale,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 358–19 369

    2023

  16. [24]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma et al., “Scaling instruction-finetuned language models,” Journal of Machine Learning Research , vol. 25, no. 70, pp. 1–53, 2024

    2024

  17. [25]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  18. [26]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  19. [27]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models,

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y . Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini et al. , “Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models,” arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  20. [28]

    Nvlm: Open frontier-class multimodal llms,

    W. Dai, N. Lee, B. Wang, Z. Yang, Z. Liu, J. Barker, T. Rintamaki, M. Shoeybi, B. Catanzaro, and W. Ping, “Nvlm: Open frontier-class multimodal llms,” arXiv preprint arXiv:2409.11402 , 2024

    2024 arXiv

  21. [29]

    Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning,

    H. Zhang, M. Gao, Z. Gan, P. Dufter, N. Wenzel, F. Huang, D. Shah, X. Du, B. Zhang, Y . Li et al., “Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning,” arXiv preprint arXiv:2409.20566 , 2024

    2024 arXiv

  22. [30]

    Emu3: Next-token prediction is all you need,

    X. Wang, X. Zhang, Z. Luo, Q. Sun, Y . Cui, J. Wang, F. Zhang, Y . Wang, Z. Li, Q. Yu et al., “Emu3: Next-token prediction is all you need,” arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  23. [31]

    Intriguing properties of neural networks,

    C. Szegedy, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013

    2013 arXiv

  24. [32]

    Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,

    A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 427–436

    2015

  25. [33]

    Explaining and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” 2015. [Online]. Available: https: //arxiv.org/abs/1412.6572

    2015 arXiv

  26. [34]

    A limited memory algo- rithm for bound constrained optimization,

    R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu, “A limited memory algo- rithm for bound constrained optimization,” SIAM Journal on scientific computing, vol. 16, no. 5, pp. 1190–1208, 1995

    1995

  27. [35]

    Towards evaluating the robustness of neural networks,

    N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in 2017 IEEE Symposium on Security and Privacy (SP) , 2017, pp. 39–57

    2017

  28. [36]

    Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,

    P.-Y . Chen, H. Zhang, Y . Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in Proceedings of the 10th ACM workshop on artificial intelligence and security , 2017, pp. 15–26

    2017

  29. [37]

    Towards deep learning models resistant to adversarial attacks,

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in International Conference on Learning Representations , 2018. [Online]. Available: https://openreview.net/forum?id=rJzIBfZAb

    2018

  30. [38]

    Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,

    F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in Proceedings of the 37th International Conference on Machine Learning , ser. ICML’20. JMLR.org, 2020

    2020

  31. [39]

    On evaluating adversarial robustness of large vision-language models,

    Y . Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, “On evaluating adversarial robustness of large vision-language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  32. [40]

    Attacking attention of foundation models disrupts downstream tasks,

    H. P. Silva, F. Becattini, and L. Seidenari, “Attacking attention of foundation models disrupts downstream tasks,” in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025, pp. 3534– 3543

    2025

  33. [41]

    Qava: Query- agnostic visual attack to large vision-language models,

    Y . Zhang, R. Xie, J. Chen, X. Sun, Z. Kang, and Y . Wang, “Qava: Query- agnostic visual attack to large vision-language models,” arXiv preprint arXiv:2504.11038, 2025

    2025 arXiv

  34. [42]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  35. [43]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015

    2015

  36. [44]

    Sentence-bert: Sentence embeddings using siamese bert- networks,

    N. Reimers, “Sentence-bert: Sentence embeddings using siamese bert- networks,” arXiv preprint arXiv:1908.10084 , 2019. 13

    1908 arXiv

  37. [45]

    Universal sentence encoder,

    D. Cer, “Universal sentence encoder,” arXiv preprint arXiv:1803.11175, 2018

    2018 arXiv

  38. [46]

    Mpnet: Masked and permuted pre-training for language understanding,

    K. Song, X. Tan, T. Qin, J. Lu, and T.-Y . Liu, “Mpnet: Masked and permuted pre-training for language understanding,” Advances in neural information processing systems , vol. 33, pp. 16 857–16 867, 2020

    2020

  39. [47]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  40. [48]

    Quantifying attention flow in transformers,

    S. Abnar and W. Zuidema, “Quantifying attention flow in transformers,” arXiv preprint arXiv:2005.00928 , 2020

    2005 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.