Pith. sign in

REVIEW 4 major objections 7 minor 65 references

Bottom singular vectors expose hidden attack surface in VLMs

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 12:40 UTC pith:MKSAW43K

load-bearing objection New spectral perspective on VLM adversarial vulnerability is interesting, but the attack comparison has a fairness problem and very high variance that undermines the headline gains. the 4 major comments →

arxiv 2607.07375 v1 pith:MKSAW43K submitted 2026-07-08 cs.LG cs.AI

On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

Chethan Krishnamurthy Ramanaik , Tobias Callies , Michael Hecht , Eirini Ntoutsi This is my paper
classification cs.LG cs.AI
keywords adversarial attacksvision-language modelsspectral analysissingular value decompositionrobustnesstransformerswhite-box attacksdeep neural networks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper identifies the bottom singular-vector subspaces of intermediate linear transformations in transformer-based vision-language models (VLMs) as a previously overlooked spectral attack surface. The authors argue that representations aligned with these near-null singular directions are strongly attenuated, weakening semantic information propagation through the network. They propose SSGRA (Spectral Subspace Guided Representation Attack), a white-box attack that explicitly aligns adversarial intermediate representations with these bottom singular-vector subspaces. The paper shows that SSGRA outperforms representative feature-space and output-space baselines on Qwen2.5-VL, LLaVA-1.5, and Gemma 3, and presents evidence that untargeted adversarial optimization naturally increases alignment with these information-attenuating subspaces even without explicit enforcement.

Core claim

The central discovery is that the bottom singular-vector subspaces of intermediate weight matrices in transformer-based VLMs constitute a spectral attack surface that can be deliberately exploited. The authors demonstrate this through SSGRA, which adds a spectral alignment objective to existing feature-discrepancy attacks, and through analysis showing that standard adversarial optimization already drifts toward these near-null directions unprompted. The degree of vulnerability correlates with the prevalence of near-null singular directions: Qwen2.5-VL, which has roughly twice as many near-null singular directions as Gemma 3, exhibits the largest degradation under SSGRA.

What carries the argument

The Spectral Subspace Guided Representation Attack (SSGRA), a white-box attack combining a blockwise representation-discrepancy objective (inherited from BSA) with a spectral alignment term that pushes adversarial representations into the span of the bottom-s right singular vectors of selected intermediate linear transformations.

Load-bearing premise

The layer-selection procedure uses a small validation set to pick which intermediate layers receive spectral guidance, tuning the attack post hoc on the same data distribution used for evaluation. If the selected layers are overfit to this 100-image set, the reported gains over baselines (which do not use such tuning) may not be directly comparable.

What would settle it

If aligning adversarial representations with bottom singular-vector subspaces were not the operative mechanism, then replacing the bottom-s subspace with a random subspace of the same dimension should yield comparable attack effectiveness, and removing the spectral alignment term should not degrade performance. The ablation showing that top-singular-subspace alignment produces much weaker attacks than bottom-singular-subspace alignment, and that removing the spectral term weakens SSGRA, partially addresses this.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Models with more near-null singular directions in their intermediate weight matrices may be systematically more vulnerable to adversarial perturbations, suggesting spectral conditioning as a diagnostic for robustness.
  • Regularization that controls or eliminates bottom singular-vector subspaces, complementing existing spectral-norm regularization on large singular values, could become a defense strategy.
  • The finding that standard attacks naturally drift toward bottom singular subspaces suggests this is an intrinsic geometric property of adversarial optimization in high-dimensional linear transformations, not an artifact of the proposed method.
  • Spectral profiling of weight matrices could serve as a lightweight pre-deployment audit for VLM robustness without requiring adversarial attack generation.
  • The correlation between near-null singular direction prevalence and attack vulnerability across models (Qwen2.5-VL vs. Gemma 3) suggests that architectural and training choices influence the spectral attack surface in measurable ways.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If bottom singular-vector subspaces are the primary spectral attack surface, then training-time interventions that reduce the rank deficiency of intermediate weight matrices (e.g., low-rank projection away from near-null directions, or structured regularization enforcing minimum singular value thresholds) could yield models that are harder to attack without changing architecture.
  • The observation that adversarial optimization naturally aligns with bottom singular subspaces may connect to the broader theory of ill-posed inverse problems: if adversarial vulnerability is fundamentally about information attenuation along near-null directions, then robustness bounds might be expressible in terms of the spectral profile of intermediate layers rather than end-to-end Lipschitz cons
  • The layer-selection procedure used by SSGRA (retaining layers with strongest attack performance on a validation set) could be inverted: layers where spectral guidance helps most might also be the layers where targeted regularization would yield the largest robustness gains, suggesting a principled layer-priority ordering for spectral defenses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces a spectral perspective on adversarial vulnerability in transformer-based vision-language models (VLMs). The central idea is that bottom singular-vector subspaces of intermediate linear transformations constitute an overlooked attack surface: representations aligned with these subspaces are strongly attenuated, and explicitly guiding adversarial representations toward them (via the proposed SSGRA attack) improves attack effectiveness. The paper provides evidence across three VLMs (Qwen2.5-VL, LLaVA-1.5, Gemma 3) and six baselines, using BERTScore and ROUGE-L. It also shows that untargeted adversarial optimization (BSA) naturally increases bottom-subspace alignment, providing independent support for the mechanism. The theoretical grounding draws on the instability of ill-posed inverse problems. The core mechanism is novel and the empirical evidence is broadly supportive, but the comparison fairness between SSGRA and baselines is compromised by asymmetric hyperparameter tuning and extremely high variance in SSGRA's results.

Significance. The paper identifies a genuinely underexplored spectral attack surface (bottom singular-vector subspaces of intermediate linear transformations) and provides a falsifiable mechanism (information attenuation along near-null directions) grounded in the theory of ill-posed inverse problems. The observation that BSA naturally increases bottom-subspace alignment (Figure 5) is a non-circular supporting result. The ablation comparing bottom- vs. top-subspace alignment (Figure 7b) is a clean test of the hypothesis. The framework is tested on three current VLMs with six representative baselines. However, the significance of the empirical gains is tempered by the fairness and variance issues detailed below.

major comments (4)
  1. Section 4.1, layer selection procedure: SSGRA's layer set S is selected by evaluating each layer on a validation set and retaining the strongest performers, while baselines receive only learning-rate grid search (Section 5.1). This asymmetry means SSGRA has optimization degrees of freedom that baselines lack. The paper acknowledges this ('Developing adaptive layer-selection methods that avoid validation-based tuning is left for future work') but the reported gains over baselines are not directly comparable under this protocol. At minimum, baselines should also receive equivalent per-layer validation tuning, or the paper should report SSGRA results with a non-tuned layer set (e.g., all layers or a fixed heuristic selection) to isolate the contribution of spectral guidance from the contribution of layer tuning.
  2. Section 5.5 and Table 3, Qwen2.5-VL: λ=0 is selected for the model showing the largest gains, reducing Eq. (2) to pure spectral alignment (the BSA discrepancy term drops out). This means the combined-objective design is not validated on the strongest-gain model. The ablation in Figure 7a removes the spectral term from the combined objective, but since λ=0 on Qwen2.5-VL, the ablation effectively compares pure spectral alignment vs. pure BSA — not the contribution of spectral guidance as an additive term. The paper should clarify whether the ablation was conducted at λ=0 or at the selected λ, and should report results for intermediate λ values on Qwen2.5-VL to validate the trade-off design.
  3. Table 3, Qwen2.5-VL ROUGE-L F1: SSGRA exhibits extremely high variance — e.g., at c=0.005, ROUGE-L F1 is 0.010±0.020 (std exceeds mean), and at c=0.0035, BERTScore F1 is 0.704±0.217 vs. BSA at 0.864±0.037. This suggests a bimodal outcome distribution where the mean improvement may be driven by a subset of images where the tuned configuration works well, rather than a uniform spectral vulnerability. The paper should report the distribution of per-image results (e.g., histograms or success rates) to clarify whether the gains are systematic or driven by a subset of images. This is load-bearing because the central claim is that bottom subspaces constitute a general spectral attack surface, not a property-specific or image-specific one.
  4. Section 4.1, subspace dimension s: The paper states that s is 'fixed' but does not describe how its value was selected or what value was used. Since s controls the dimensionality of the bottom singular-vector subspace and directly affects the spectral alignment term in Eq. (2), its selection procedure should be disclosed. If s was also tuned on the validation set, this adds to the fairness concern above.
minor comments (7)
  1. Table 3, Gemma 3: SSGRA does not consistently outperform the strongest baseline (CE) across all budgets. At c=0.002 and c=0.0025, SSGRA is worse than CE on ROUGE-L F1. The paper should acknowledge this more explicitly rather than stating 'SSGRA consistently achieves the strongest attacks.'
  2. Section 5.1: The perturbation budget range (0.002–0.005 in L∞) is described as selected via grid search, but the selection criterion ('lower bound captures the regime where outputs remain semantically similar') is vague. The specific criterion should be disclosed.
  3. Figure 2: The y-axis ranges differ across subplots (e.g., Qwen2.5-VL ROUGE-L ranges 0–0.6 while Gemma 3 ranges 0–0.5), making visual comparison across models difficult. Consider using consistent ranges or clearly labeling the different scales.
  4. Section 3, Eq. (1): The alignment measure Ψ_k normalizes by ||h_i||_2, but the paper does not discuss potential numerical instability when token representations have very small norms.
  5. Table 2: The FLOPs comparison notes that values should be compared only within the same VLM, but the table presents all three models together. A visual separator or clearer labeling would help.
  6. Reference [1] cites 'LLaVA-onevision-1.5' but the model is referred to as 'LLaVA-1.5' throughout the text. Clarify whether this is the same model or a different variant.
  7. Algorithm 1 (Appendix): Line 17 defines L_SSGRA = -λL_BSA + (1-λ)L_SS, but the sign convention should be clarified — the text describes maximizing both objectives, but the negative sign on L_BSA suggests minimization of similarity (which is equivalent to maximizing discrepancy). A brief note would help readers.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major concerns: (1) asymmetric hyperparameter tuning between SSGRA and baselines, (2) the λ=0 selection for Qwen2.5-VL undermining the combined-objective ablation, (3) extremely high variance in SSGRA's results suggesting bimodal outcomes, and (4) undisclosed selection procedure for the subspace dimension s. We agree that all four points identify legitimate gaps in the manuscript's experimental transparency. We will address each through new experiments and revised text in the next version.

read point-by-point responses

  1. Referee: Section 4.1, layer selection procedure: SSGRA's layer set S is selected by evaluating each layer on a validation set and retaining the strongest performers, while baselines receive only learning-rate grid search (Section 5.1). This asymmetry means SSGRA has optimization degrees of freedom that baselines lack. The paper acknowledges this ('Developing adaptive layer-selection methods that avoid validation-based tuning is left for future work') but the reported gains over baselines are not directly comparable under this protocol. At minimum, baselines should also receive equivalent per-layer validation tuning, or the paper should report SSGRA results with a non-tuned layer set (e.g., all layers or a fixed heuristic selection) to isolate the contribution of spectral guidance from the contribution of layer tuning.

    Authors: The referee is correct that the current protocol introduces an asymmetry: SSGRA benefits from validation-based layer selection while baselines do not receive equivalent per-layer tuning. We agree this confounds the comparison and will address it in the revision through two additions. First, we will report SSGRA results using a non-tuned, fixed layer set (specifically, applying spectral guidance to all eligible intermediate linear transformations rather than a validation-selected subset). This isolates the contribution of spectral guidance from the contribution of layer tuning. If the gains persist with the fixed layer set, they are attributable to the spectral mechanism rather than the selection procedure. Second, we will apply equivalent per-layer validation tuning to the strongest baselines (at least BSA, which is the most directly comparable method since SSGRA extends it). This ensures both methods receive comparable optimization budgets. We will present both sets of results alongside the existing ones and discuss the implications transparently. We acknowledge that if the gains diminish substantially under the fixed layer set, the current claims about the magnitude of improvement would need to be qualified. We also note that the non-circular supporting evidence — the BSA natural-alignment finding (Figure 5) and the bottom-vs-top ablation (Figure 7b) — does not depend on layer selection and thus provides independent evidence for the spectral mechanism regardless of this concern. revision: yes

  2. Referee: Section 5.5 and Table 3, Qwen2.5-VL: λ=0 is selected for the model showing the largest gains, reducing Eq. (2) to pure spectral alignment (the BSA discrepancy term drops out). This means the combined-objective design is not validated on the strongest-gain model. The ablation in Figure 7a removes the spectral term from the combined objective, but since λ=0 on Qwen2.5-VL, the ablation effectively compares pure spectral alignment vs. pure BSA — not the contribution of spectral guidance as an additive term. The paper should clarify whether the ablation was conducted at λ=0 or at the selected λ, and should report results for intermediate λ values on Qwen2.5-VL to validate the trade-off design.

    Authors: The referee's observation is accurate and we appreciate its precision. On Qwen2.5-VL, λ=0 was selected, which reduces Eq. (2) to pure spectral alignment without the BSA discrepancy term. The ablation in Figure 7a was conducted at the selected λ=0, meaning it compares pure spectral alignment (SSGRA with λ=0) against pure BSA (the spectral term removed, which leaves only the BSA objective). The referee is correct that this does not test the contribution of spectral guidance as an additive term within the combined objective — it tests the two objectives in isolation. We will make this explicit in the revised text. To properly validate the trade-off design, we will run and report results for intermediate λ values (e.g., λ ∈ {0.1, 0.25, 0.5, 0.75, 0.9}) on Qwen2.5-VL. This will show whether combining the two objectives at intermediate λ yields improvements over either pure objective (λ=0 or λ=1), or whether pure spectral alignment is indeed optimal for this model. If the latter, we will discuss why Qwen2.5-VL's spectral characteristics (its high proportion of near-null singular directions, Table 1) may make pure spectral alignment sufficient, while models with fewer near-null directions (e.g., Gemma 3) may benefit from the combined objective. We will also report the selected λ values for LLaVA-1.5 and Gemma 3 to show whether the combined objective is validated on those models. revision: yes

  3. Referee: Table 3, Qwen2.5-VL ROUGE-L F1: SSGRA exhibits extremely high variance — e.g., at c=0.005, ROUGE-L F1 is 0.010±0.020 (std exceeds mean), and at c=0.0035, BERTScore F1 is 0.704±0.217 vs. BSA at 0.864±0.037. This suggests a bimodal outcome distribution where the mean improvement may be driven by a subset of images where the tuned configuration works well, rather than a uniform spectral vulnerability. The paper should report the distribution of per-image results (e.g., histograms or success rates) to clarify whether the gains are systematic or driven by a subset of images. This is load-bearing because the central claim is that bottom subspaces constitute a general spectral attack surface, not a property-specific or image-specific one.

    Authors: This is a fair and important concern. The high variance — particularly cases where the standard deviation exceeds the mean — is consistent with a bimodal or heavy-tailed distribution of per-image outcomes. We will address this by reporting per-image result distributions (histograms and/or success rates at various thresholds) for SSGRA and the strongest baselines on Qwen2.5-VL. This will make transparent whether the gains are systematic across images or driven by a subset where the attack succeeds dramatically while others remain largely unaffected. We acknowledge that if the distribution is strongly bimodal — with gains concentrated on a subset of images — the claim that bottom subspaces constitute a 'general spectral attack surface' would need to be qualified. However, we note that even a bimodal distribution would be consistent with the spectral mechanism if the bimodality correlates with spectral properties of the specific images or layers involved (e.g., images whose clean representations have higher initial energy in near-null directions). We will investigate this and report whether per-image attack success correlates with spectral alignment measures, which would provide additional mechanistic evidence. We will also report median performance in addition to means to reduce the influence of outlier images on the reported comparisons. revision: yes

  4. Referee: Section 4.1, subspace dimension s: The paper states that s is 'fixed' but does not describe how its value was selected or what value was used. Since s controls the dimensionality of the bottom singular-vector subspace and directly affects the spectral alignment term in Eq. (2), its selection procedure should be disclosed. If s was also tuned on the validation set, this adds to the fairness concern above.

    Authors: The referee is correct that the selection procedure for s is not disclosed in the manuscript. We will add this information in the revision. In our experiments, s was set to 10, matching the value used in the spectral alignment analysis (Section 5.4, where k=10 is used for the top-k and bottom-k subspace analyses in Figures 5 and 6). The value was chosen based on the spectral analysis showing that the bottom-10 singular vectors capture the most attenuated directions, and was fixed across all models and perturbation budgets rather than being tuned per-model on the validation set. We will state this explicitly, including the rationale for the choice and its relationship to the analysis in Section 5.4. We will also add a sensitivity analysis over a small set of s values (e.g., s ∈ {5, 10, 20, 50}) to show that the results are not sensitive to this choice, which would further address the fairness concern. If s was in fact tuned on the validation set, we will disclose this honestly and add it to the list of tuned hyperparameters; if it was fixed as described, the sensitivity analysis will confirm robustness. revision: yes

Circularity Check

0 steps flagged

No significant circularity; one minor self-citation provides context but is not load-bearing for the central claims

full rationale

The paper's core mechanism — that bottom singular-vector subspaces of intermediate linear transformations attenuate information and constitute a spectral attack surface — is grounded in the external theory of ill-posed inverse problems, citing Antun et al. [2,3] and Gottschling et al. [26], which are independent works by different author groups. The SSGRA attack objective (Eq. 2) is a novel combination of BSA's representation discrepancy term (citing Yin et al. [62], an external work) with a spectral alignment measure (Eq. 1) that is defined from standard SVD projections. The key empirical findings — that BSA naturally increases bottom-subspace alignment (Figure 5), that SSGRA-bottom outperforms SSGRA-top (Figure 7b), and that models with more near-null singular directions show larger degradation (Table 1, Section 5.2) — are all independently measurable and not forced by construction. The layer-selection procedure (Section 4.1) uses validation-based tuning, which raises fairness concerns (baselines lack equivalent tuning), but this is a methodological concern about experimental design, not circularity in the derivation chain. The one self-citation to Ramanaik et al. [52] (GRILL) appears in the Related Work section as context for gradient restoration in ill-conditioned layers; it is not invoked as a load-bearing premise for any equation or claim in this paper. The paper is largely self-contained against external benchmarks, and no prediction or first-principles result reduces to its inputs by definition or by self-citation chain. The minor self-citation does not raise the circularity score beyond 2.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions). It introduces a new attack objective (spectral alignment) and a new alignment measure, but these are methods, not postulated physical or mathematical objects. The free parameters are hyperparameters selected by validation tuning, which is standard for attack methods but affects the fairness of baseline comparisons.

free parameters (5)
  • lambda (trade-off parameter) = 0 for Qwen2.5-VL (reducing to pure spectral alignment); not explicitly stated for LLaVA-1.5 or Gemma 3
    Controls the balance between BSA representation discrepancy and spectral alignment in Eq. 2. Selected via grid search on validation data.
  • s (bottom subspace dimension) = Fixed but exact value not stated in the main text
    Determines the number of bottom right singular vectors used in the alignment measure Psi_s. Stated as 'fixed subspace dimensions' in Section 3 but the specific value is not reported.
  • S (selected layer set) = Selected per model via validation-based tuning
    The subset of intermediate linear transformations to which spectral guidance is applied. Chosen by evaluating each layer independently on a small validation set and retaining the strongest (Section 4.1).
  • c (perturbation budget) = 0.002 to 0.005
    L-infinity perturbation budget, selected via grid search to span regimes from semantically similar to pronounced degradation.
  • learning rate = 1e-3
    Selected via grid search over {1e-2, 5e-3, 1e-3, 5e-4, 1e-4} as consistently best across models.
axioms (3)
  • domain assumption Near-null singular directions of intermediate linear transformations cause information attenuation analogous to the instability in ill-posed inverse problems.
    Section 3 draws an analogy between reconstruction maps in inverse problems and forward propagation through VLM layers. This analogy motivates the entire framework but is not formally proven; it is an interpretive assumption.
  • domain assumption SVD of frozen weight matrices captures meaningful spectral structure relevant to adversarial vulnerability during optimization.
    The method computes SVD of weight matrices (Algorithm 1, line 6) and uses the resulting singular vectors as fixed targets. This assumes the singular structure of the weights is relevant to the dynamics of adversarial optimization, which is plausible but not formally established.
  • domain assumption Cosine similarity between clean and adversarial intermediate representations is a meaningful proxy for semantic disruption.
    Inherited from BSA (Ref 62). The BSA component of SSGRA (Eq. 2) relies on this assumption, which is standard in feature-space attacks but unproven in general.

pith-pipeline@v1.1.0-glm · 31540 in / 2993 out tokens · 376397 ms · 2026-07-09T12:40:38.819562+00:00 · methodology

0 comments
read the original abstract

Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems. Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability. Specifically, we investigate transformer-based vision-language models, whose linear layers admit interpretable spectral decompositions and whose widespread adoption makes understanding their robustness increasingly important. We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors. Our experiments show improved attack effectiveness over existing baselines. In addition, SSGRA offers a spectral interpretation of adversarial vulnerability in VLMs, providing insights for improving their robustness.

Figures

Figures reproduced from arXiv: 2607.07375 by Chethan Krishnamurthy Ramanaik, Eirini Ntoutsi, Michael Hecht, Tobias Callies.

Figure 1
Figure 1. Figure 1: Overview of the proposed spectral framework. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: SSGRA vs representative baselines across perturbation budgets. Lower scores indicate stronger attacks. 5.2 Quantitative comparison with State-of-the-Art Attacks [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative adversarial examples generated under a common perturbation bud￾get of c = 0.003 across three vision–language models. shows the smallest relative degradation, consistent with its lower proportion of near-null singular directions compared with Qwen 2.5-VL and LLaVA 1.5, lim￾iting the opportunity to exploit bottom singular-vector subspaces. A detailed discussion on spectral characterization follow… view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of the largest (\sigma _{\max } , left) and smallest (\sigma _{\min } , right) singular values across non-attention linear operators in Qwen2.5-VL, LLaVA-1.5, and Gemma 3 [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of the spectral alignment measure Ψk before and after BSA opti￾mization for the top-10 and bottom-10 right singular-vector subspaces. Across all three models, near-null singular directions occur substantially more frequently than strongly amplifying ones. For example, σmin < 10−3 is observed in 24.90%, 27.67%, and 12.22% of operators in Qwen2.5-VL, LLaVA-1.5, and Gemma 3, respectively, with si… view at source ↗
Figure 6
Figure 6. Figure 6: Layer-wise evolution of top and bottom most singular vectors alignment during adversarial optimization on Gemma 3 (two representative samples) [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation analysis validating the design choices of SSGRA. Effect of the Spectral Alignment Term. Figure 7a compares SSGRA with and without the spectral alignment term in Eq. (2). Removing this term in￾creases both BERTScore F1 and ROUGE-L F1 across all perturbation budgets, indicating weaker attacks and confirming that spectral alignment is the primary contributor to the improved attack effectiveness. The … view at source ↗
Figure 8
Figure 8. Figure 8: BERT-score comparison of different adversarial attack methods across pertur￾bation budgets for Qwen 2.5-VL, LLaVa 1.5, and Gemma 3. Each row corresponds to a model, while the columns show Precision, Recall, and F1 score, respectively [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: ROUGE-L score comparison of sample-specific attacks across Qwen 2.5-VL, LLaVa 1.5, and Gemma 3. The three columns report Precision, Recall, and F1 score, respectively, while each row corresponds to a different vision-language model [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Additional qualitative adversarial examples (set 1) generated with a pertur￾bation budget of c = 0.002 across models [PITH_FULL_IMAGE:figures/full_fig_p028_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Additional qualitative adversarial examples (set 2) generated with a pertur￾bation budget of c = 0.002 across models [PITH_FULL_IMAGE:figures/full_fig_p029_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Additional qualitative adversarial examples (set 3) generated with a pertur￾bation budget of c = 0.002 across models [PITH_FULL_IMAGE:figures/full_fig_p030_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Additional qualitative adversarial examples (set 4) generated with a pertur￾bation budget of c = 0.003 across models [PITH_FULL_IMAGE:figures/full_fig_p031_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 65 canonical work pages · 23 internal anchors

  1. [1]

    LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

    An, X., Xie, Y., Yang, K., Zhang, W., Zhao, X., Cheng, Z., Wang, Y., Xu, S., Chen, C., Zhu, D., et al.: Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661 (2025) 8

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    SIAM NEWS MARCH54 (2021) 1, 3

    Antun, V., Gottschling, N.M., Hansen, A.C., Adcock, B.: Deep learning in scien- tific computing: Understanding the instability mystery. SIAM NEWS MARCH54 (2021) 1, 3

    work page 2021

  3. [3]

    Proceedings of the National Academy of Sciences117(48), 30088–30095 (2020) 1, 3, 5

    Antun, V., Renna, F., Poon, C., Adcock, B., Hansen, A.C.: On instabilities of deep learning in image reconstruction and the potential costs of ai. Proceedings of the National Academy of Sciences117(48), 30088–30095 (2020) 1, 3, 5

    work page 2020

  4. [4]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025) 2, 8

    work page 2025

  5. [5]

    Image Hijacks: Adversarial Images can Control Generative Models at Runtime

    Bailey, L., Ong, E., Russell, S., Emmons, S.: Image hijacks: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236 (2023) 4

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    In: AISTATS

    Barrett, B., Camuto, A., Willetts, M., Rainforth, T.: Certifiably robust variational autoencoders. In: AISTATS. pp. 3663–3683. PMLR (2022) 3

    work page 2022

  7. [7]

    In: 2017 ieee symposium on security and privacy (sp)

    Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 ieee symposium on security and privacy (sp). pp. 39–57. Ieee (2017) 3, 8

    work page 2017

  8. [8]

    In: ICLR (2020) 3

    Cemgil, T., Ghaisas, S., Dvijotham, K.D., Kohli, P.: Adversarially robust repre- sentations with smooth encoders. In: ICLR (2020) 3

    work page 2020

  9. [9]

    Jacobian Adversarially Regularized Networks for Robustness

    Chan, A., Tay, Y., Ong, Y.S., Fu, J.: Jacobian adversarially regularized networks for robustness. arXiv preprint arXiv:1912.10185 (2019) 3

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  10. [10]

    In: International conference on ma- chine learning

    Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., Usunier, N.: Parseval networks: Improving robustness to adversarial examples. In: International conference on ma- chine learning. pp. 854–863. PMLR (2017) 3

    work page 2017

  11. [11]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cui, X., Aparcedo, A., Jang, Y.K., Lim, S.N.: On the robustness of large multi- modal models against image adversarial attacks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24625–24634 (2024) 4

    work page 2024

  12. [12]

    In: CVPR

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255. Ieee (2009) 8

    work page 2009

  13. [13]

    Dong, Y., Chen, H., Chen, J., Fang, Z., Yang, X., Zhang, Y., Tian, Y., Su, H., Zhu, J.: How robust is google’s bard to adversarial image attacks? arXiv preprint arXiv:2309.11751 (2023) 4

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [14]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020) 2

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  15. [15]

    On the Connection Between Adversarial Robustness and Saliency Map Interpretability

    Etmann, C., Lunz, S., Maass, P., Schönlieb, C.B.: On the connection be- tween adversarial robustness and saliency map interpretability. arXiv preprint arXiv:1905.04172 (2019) 3

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  16. [16]

    Machine learning107(3), 481–508 (2018) 1, 3

    Fawzi, A., Fawzi, O., Frossard, P.: Analysis of classifiers’ robustness to adversarial perturbations. Machine learning107(3), 481–508 (2018) 1, 3

    work page 2018

  17. [17]

    NeurIPS32(2019) 3

    Fazlyab, M., Robey, A., Hassani, H., Morari, M., Pappas, G.: Efficient and accurate estimation of lipschitz constants for deep neural networks. NeurIPS32(2019) 3

    work page 2019

  18. [18]

    In: Proceed- ings of the IEEE/CVF international conference on computer vision

    Ganeshan, A., BS, V., Babu, R.V.: Fda: Feature disruptive attack. In: Proceed- ings of the IEEE/CVF international conference on computer vision. pp. 8069–8079 (2019) 4 16 C.K. Ramanaik et al

    work page 2019

  19. [19]

    Adversarial Robustness for Visual Grounding of Multimodal Large Language Models

    Gao, K., Bai, Y., Bai, J., Yang, Y., Xia, S.T.: Adversarial robustness for visual grounding of multimodal large language models. arXiv preprint arXiv:2405.09981 (2024) 4

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [20]

    Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images

    Gao, K., Bai, Y., Gu, J., Xia, S.T., Torr, P., Li, Z., Liu, W.: Inducing high energy-latency of large vision-language models with verbose images. arXiv preprint arXiv:2401.11170 (2024) 4

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Gemma Team: Gemma 3 technical report (2025) 2, 8

    work page 2025

  22. [22]

    Adversarial Spheres

    Gilmer, J., Metz, L., Faghri, F., Schoenholz, S.S., Raghu, M., Wattenberg, M., Goodfellow, I.: Adversarial spheres (2018). arXiv preprint arXiv:1801.02774 (1801) 1, 3

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  23. [23]

    arXiv (2018) 3

    Gondim-Ribeiro, G., Tabacof, P., Valle, E.: Adversarial attacks on variational au- toencoders. arXiv (2018) 3

    work page 2018

  24. [24]

    MIT Press (2016),http: //www.deeplearningbook.org1, 3

    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016),http: //www.deeplearningbook.org1, 3

    work page 2016

  25. [25]

    Explaining and Harnessing Adversarial Examples

    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014) 1, 3

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  26. [26]

    SIAM Review67(1), 73–104 (2025) 1, 3, 5

    Gottschling, N.M., Antun, V., Hansen, A.C., Adcock, B.: The troublesome kernel: On hallucinations, no free lunches, and the accuracy-stability tradeoff in inverse problems. SIAM Review67(1), 73–104 (2025) 1, 3, 5

    work page 2025

  27. [27]

    NeurIPS30(2017) 3

    Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. NeurIPS30(2017) 3

    work page 2017

  28. [28]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Gupta, K., Ajanthan, T.: Improved gradient-based adversarial attacks for quan- tized networks. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 6810–6818 (2022) 3

    work page 2022

  29. [29]

    High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models

    He, M., Tian, X., Shen, X., Ni, J., Zou, S., Yang, Z., Zhang, J.: Few to- kens matter: Entropy guided attacks on vision-language models. arXiv preprint arXiv:2512.21815 (2025) 4

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  30. [30]

    Advances in neural information processing systems30(2017) 3

    Hein, M., Andriushchenko, M.: Formal guarantees on the robustness of a classi- fier against adversarial manipulation. Advances in neural information processing systems30(2017) 3

    work page 2017

  31. [31]

    Robust Learning with Jacobian Regularization

    Hoffman, J., Roberts, D.A., Yaida, S.: Robust learning with jacobian regulariza- tion. arXiv preprint arXiv:1908.02729 (2019) 1, 3

    work page internal anchor Pith review Pith/arXiv arXiv 1908

  32. [32]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Huang, Y., Würfl, T., Breininger, K., Liu, L., Lauritsch, G., Maier, A.: Some investigations on robustness of deep learning in limited angle tomography. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 145–153. Springer (2018) 5

    work page 2018

  33. [33]

    Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examplesarenotbugs,theyarefeatures.Advancesinneuralinformationprocessing systems32(2019) 1, 3

    work page 2019

  34. [34]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Inkawhich,N.,Wen,W.,Li,H.H.,Chen,Y.:Featurespaceperturbationsyieldmore transferable adversarial examples. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7066–7074 (2019) 3

    work page 2019

  35. [35]

    Machine Learning: Science and Tech- nology6(4), 045029 (2025) 5

    Jain, S.B., Shao, Z., Hecht, M.: Automated detection of potential artifacts in ma- chine learning based bio-image segmentation. Machine Learning: Science and Tech- nology6(4), 045029 (2025) 5

    work page 2025

  36. [36]

    In: Proceedings of the European conference on computer vision (ECCV)

    Jakubovitz, D., Giryes, R.: Improving dnn robustness to adversarial attacks using jacobian regularization. In: Proceedings of the European conference on computer vision (ECCV). pp. 514–529 (2018) 1, 3

    work page 2018

  37. [37]

    Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment

    Jia, X., Gao, S., Qin, S., Pang, T., Du, C., Huang, Y., Li, X., Li, Y., Li, B., Liu, Y.: Adversarial attacks against closed-source mllms via feature optimal alignment. arXiv preprint arXiv:2505.21494 (2025) 4 Adversarial Vulnerability in VLMs via Spectral Subspaces 17

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    On the Geometry of Adversarial Examples

    Khoury, M., Hadfield-Menell, D.: On the geometry of adversarial examples. arXiv preprint arXiv:1811.00525 (2018) 1, 3

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  39. [39]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Khrulkov, V., Oseledets, I.: Art of singular vectors and universal adversarial per- turbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8562–8570 (2018) 1, 3

    work page 2018

  40. [40]

    arXiv (2021) 3

    Kuzina, A., Welling, M., Tomczak, J.M.: Diagnosing vulnerability of variational auto-encoders to adversarial attacks. arXiv (2021) 3

    work page 2021

  41. [41]

    In: Text sum- marization branches out

    Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text sum- marization branches out. pp. 74–81 (2004) 8, 20

    work page 2004

  42. [42]

    In: Forty-third International Conference on Machine Learning (2026) 4

    Liu, D., Cai, X., Dong, J., Guo, Z., Qu, X., Guan, R., Fang, X., Ye, D.: Attacking gray-box large vision-language models with adaptive svd-structured adversarial alignment. In: Forty-third International Conference on Machine Learning (2026) 4

    work page 2026

  43. [43]

    In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition

    Lu, Y., Jia, Y., Wang, J., Li, B., Chai, W., Carin, L., Velipasalar, S.: Enhanc- ing cross-task black-box transferability of adversarial examples with dispersion reduction. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 940–949 (2020) 4

    work page 2020

  44. [44]

    An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models

    Luo, H., Gu, J., Liu, F., Torr, P.: An image is worth 1000 lies: Adversarial transfer- ability across prompts on vision-language models. arXiv preprint arXiv:2403.09766 (2024) 4

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [45]

    Foveation-based Mechanisms Alleviate Adversarial Examples

    Luo, Y., Boix, X., Roig, G., Poggio, T., Zhao, Q.: Foveation-based mechanisms alleviate adversarial examples. arXiv preprint arXiv:1511.06292 (2015) 3

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  46. [46]

    stat1050, 9 (2017) 3

    Mądry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. stat1050, 9 (2017) 3

    work page 2017

  47. [47]

    Martin, C.H., Mahoney, M.W.: Implicit self-regularization in deep neural networks: Evidencefromrandommatrixtheoryandimplicationsforlearning.JMLR22(165), 1–73 (2021) 3

    work page 2021

  48. [48]

    arXiv (2018) 3

    Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv (2018) 3

    work page 2018

  49. [49]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1765–1773 (2017) 1, 3

    work page 2017

  50. [50]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Naseer, M., Khan, S., Hayat, M., Khan, F.S., Porikli, F.: A self-supervised ap- proach for adversarial robustness. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 262–271 (2020) 4

    work page 2020

  51. [51]

    In: Forty-second International Conference on Machine Learning (2025) 1, 3

    Paniagua, T., Savadikar, C., Wu, T.: Adversarial perturbations are formed by iter- atively learning linear combinations of the right singular vectors of the adversarial jacobian. In: Forty-second International Conference on Machine Learning (2025) 1, 3

    work page 2025

  52. [52]

    Revealing Hidden Vulnerabilities in Autoencoders through Gradient Signal Restoration

    Ramanaik, C.K., Roy, A., Callies, T., Ntoutsi, E.: Grill: Gradient signal restoration in ill-conditioned layers to enhance adversarial attacks on autoencoders. arXiv preprint arXiv:2505.03646 (2025) 3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  53. [53]

    Axioms 13(8), 535 (2024) 3

    Ramanaik, C.K., Willmann, A., Suarez Cardona, J.E., Hanfeld, P., Hoffmann, N., Hecht, M.: Ensuring topological data-structure preservation under autoencoder compression due to latent space regularization in gauss–legendre nodes. Axioms 13(8), 535 (2024) 3

    work page 2024

  54. [54]

    Adversarial Manipulation of Deep Representations

    Sabour, S., Cao, Y., Faghri, F., Fleet, D.J.: Adversarial manipulation of deep rep- resentations. arXiv preprint arXiv:1511.05122 (2015) 3

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  55. [55]

    Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models

    Samangouei, P., Kabkab, M., Chellappa, R.: Defense-gan: Protecting clas- sifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.06605 (2018) 1, 3 18 C.K. Ramanaik et al

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  56. [56]

    PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples

    Song, Y., Kim, T., Nowozin, S., Ermon, S., Kushman, N.: Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv:1710.10766 (2017) 1, 3

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  57. [57]

    Intriguing properties of neural networks

    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fer- gus, R.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013) 1, 3, 5

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  58. [58]

    Advances in neural information pro- cessing systems30(2017) 2

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017) 2

    work page 2017

  59. [59]

    Advances in neural information processing systems31 (2018) 3

    Virmaux, A., Scaman, K.: Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in neural information processing systems31 (2018) 3

    work page 2018

  60. [60]

    Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image

    Wang, Z., Han, Z., Chen, S., Xue, F., Ding, Z., Xiao, X., Tresp, V., Torr, P., Gu, J.: Stop reasoning! when multimodal llm with chain-of-thought reasoning meets adversarial image. arXiv preprint arXiv:2402.14899 (2024) 4

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  61. [61]

    arXiv (2019) 3

    Willetts, M., Camuto, A., Rainforth, T., Roberts, S., Holmes, C.: Improving vaes’ robustness to adversarial attack. arXiv (2019) 3

    work page 2019

  62. [62]

    Advances in Neural Information Processing Systems36, 52936– 52956 (2023) 1, 4, 6, 12

    Yin, Z., Ye, M., Zhang, T., Du, T., Zhu, J., Liu, H., Chen, J., Wang, T., Ma, F.: Vlattack: Multimodal adversarial attacks on vision-language tasks via pre- trained models. Advances in Neural Information Processing Systems36, 52936– 52956 (2023) 1, 4, 6, 12

    work page 2023

  63. [63]

    arXiv (2017) 3

    Yoshida, Y., Miyato, T.: Spectral norm regularization for improving the generaliz- ability of deep learning. arXiv (2017) 3

    work page 2017

  64. [64]

    In: International conference on machine learning

    Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: International conference on machine learning. pp. 7472–7482. PMLR (2019) 1, 3

    work page 2019

  65. [65]

    Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019) 8, 20 Adversarial Vulnerability in VLMs via Spectral Subspaces 19 A Appendix A.1 Spectral Subspace Guided Representation Attack (SSGRA) Algorithm Algorithm 1Spectral Subspace Guided Representation Attack (SSG...

    work page internal anchor Pith review Pith/arXiv arXiv 1904