{"id":"e9ceb744-ccfe-488f-8e9b-db809b577acc","arxiv_id":"2506.05394","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A task-agnostic attack that perturbs attention and embeddings of CLIP/ViT backbones degrades classification, retrieval, captioning, segmentation, and depth estimation without using labels or text.","lead":"This paper presents an adversarial attack that disrupts the attention and embedding layers of vision foundation models such as CLIP and ViT, using no labels or text. It demonstrates that a single image perturbation degrades performance across classification, retrieval, captioning, segmentation, and depth estimation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 5 defines Lemb as an L2 distance to be minimized, which would preserve embeddings; reported attack success requires the opposite sign, so the stated objective cannot produce the results.","rationale":"I agree with the reader's weakest_assumption. The sign inconsistency is not cosmetic: every downstream result in Tables 1–6 includes an embedding-loss variant or a combined variant whose efficacy depends on moving embeddings away from the clean representation. If Eq. 5 were implemented literally, the Lemb component would be a regularizer preserving the representation, and the combined attack would reduce to Latn alone; Table 1's 98.4% for Lcomb versus 86.4% for Latn shows Lemb contributes positively to the attack, which is impossible under the stated minimization. The paper does provide a code link, which could resolve this, and I credit that as independent evidence if the code matches the intended attack. The proper verdict remains conditional pending correction or verification, so I do not move the reader's verdict. However, if code inspection shows the released implementation also uses a minimizing L2 distance, the results as reported would be unsupported and the central claim would need to be re-evaluated.","tokens_in":14815,"tokens_out":2990,"duration_ms":34558,"concrete_test":"Run the official implementation (github.com/HondamunigePrasannaSilva/attack-attention) on the ImageNet classification setting of Table 1 and instrument the gradient of Lemb with respect to z at the first iteration. If the gradient step decreases ||E_gt - E_adv||_2, the attack success rate should be near zero; if the code instead negates Lemb or minimizes a negative dot product, then Eq. 5 must be corrected. A second, independent check: modify only the sign of Lemb in the released code so that Eq. 5 is implemented literally and rerun Table 1; literal implementation should collapse attack success, confirming the sign error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that a task-agnostic perturbation of CLIP/ViT attention and embeddings disrupts downstream tasks—depends on the optimization objective stated in Sections 3.1–3.4. Eq. 1 sets z* = arg min_z Lcomb and Eq. 2 uses gradient descent. Eq. 5 defines Lemb = ||E_gt - E_adv||_2, and Eq. 8 minimizes α L_atn + β L_emb. Minimizing an L2 distance to the clean embedding pulls the adversarial embedding toward the clean one; since z=0 already attains Lemb=0, the attack should leave embeddings essentially unchanged and preserve downstream accuracy. Yet Table 1 reports Ours(Lemb) with roughly 98% attack success on classification, and Tables 2–4 report large drops from the embedding-only variant. Text around Eq. 5 says the goal is 'minimizing the dot product between adversarial embeddings and clean embeddings,' which is a different, similarity-decreasing objective. Thus the paper either implements a loss with the opposite sign (maximizing the L2 distance or minimizing a negative dot product) or negates Eq. 5's gradient during optimization. The stated math does not support the results; the intended objective is an unstated assumption. This is an internal inconsistency, not merely disagreement with prior work, and it is the load-bearing step for all downstream-task claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a task-agnostic adversarial attack on vision foundation models, targeting both the attention mechanism and the image embedding of CLIP and Vision Transformer backbones. The attack is optimized with a combined loss and is claimed to transfer to multiple downstream tasks, including zero-shot classification, image/text retrieval, captioning, semantic segmentation, and depth estimation, without access to labels or text. Experiments report large performance drops on all these tasks, and the paper positions the method as an improvement over multimodal attack baselines while attacking only the image modality.","tokens_in":15098,"tokens_out":3684,"duration_ms":44911,"significance":"If the stated objective is corrected and the empirical results hold, the paper would make a useful contribution: it demonstrates a single image-only perturbation that disrupts several downstream tasks built on shared vision backbones, and it explicitly tests transferability. The paper is also commendable for evaluating five task families and for making code available. The main concern is that the optimization objective as written in Eq. (5) is inconsistent with the reported attack success; this must be resolved before the empirical claims can be accepted. The absence of a comparison to the closest task-agnostic baseline, EmbedAttack, also weakens the claimed state-of-the-art status.","major_comments":[{"comment":"The embedding loss is defined as an L2 norm to be minimized under z* = arg min_z Lcomb. Since z = 0 already attains Lemb = 0, the stated objective cannot increase the distance between clean and adversarial embeddings; it should instead preserve or even reduce the perturbation's effect on the embedding. The reported 97.9% attack success for Ours(Lemb) in Table 1 and the large drops in Table 2 are therefore inconsistent with the printed math. The prose around Eq. (5) refers to 'minimizing the dot product' between embeddings, which is a different, similarity-decreasing objective. Please state the exact objective actually optimized (e.g., maximizing ||E_gt - E_adv||_2 or minimizing a negative cosine similarity), update Eq. (5) and Eq. (8) accordingly, or provide the optimization code. This is load-bearing because all downstream-task claims derive from this objective.","section":"Section 3.3, Eq. (5) and Eq. (8)"},{"comment":"EmbedAttack [22] is introduced as the first task-agnostic attack on ViT backbones and is the closest baseline to the proposed method, but it is never compared on classification or retrieval, despite the paper claiming state-of-the-art results. The experiments should include EmbedAttack (or at least an image-only embedding-perturbation baseline equivalent to Ours(Lemb)) on the same tasks and budgets. Without this, the claimed advantage over existing task-agnostic attacks is not established.","section":"Section 2, Tables 1 and 2"},{"comment":"The captioning experiments state that 'Adversarially perturbed images are generated using the CLIP-ViT-L model' while evaluating BLIP-2. It is unclear whether the attack is optimized on CLIP and then transferred to BLIP-2, or whether BLIP-2's own visual encoder is attacked. These are different threat models (transfer versus white-box), and the distinction matters for interpreting the captioning results. Please specify the exact protocol and, if the former, clarify how the attack is aligned with BLIP-2's backbone.","section":"Section 4, Table 3"}],"minor_comments":[{"comment":"The term 'Vision Transfomers' should be corrected to 'Vision Transformers'.","section":"Abstract and Section 1"},{"comment":"The balancing rule beta = alpha * |L_atn / L_emb| is undefined if L_emb is zero early in optimization; please state a numerical safeguard (e.g., adding a small epsilon to the denominator).","section":"Section 3.4"},{"comment":"In the FLICKR row for Ours(Lcomb), the value '0.03598' appears to be a typo for BLEU-1 (likely 0.3598); please verify all entries for consistency.","section":"Table 3"},{"comment":"The table formatting is hard to read because numbers are run together (e.g., 'Sep-Attack2.50 0.40 0.10'), and the dataset name is misspelled as 'Flicr30K'. Please fix the formatting and ensure all column entries are clearly separated.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The Eq. (5) sign issue is the central technical concern; if the authors clarify the implementation and add the EmbedAttack comparison, the paper could become publishable. The empirical corpus is broad, but the missing baseline and the inconsistency in the captioning threat model need to be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a real empirical finding buried under a load-bearing math error. The idea is simple and worth taking seriously: attack both the attention maps and the final embeddings of a vision backbone, image-only and label-free, and one perturbation disrupts five different downstream tasks. The attention-orthogonality loss is new, and the breakdown of which task type responds to attention vs. embedding damage is a genuinely useful observation. Credit where due: they cover a wide set of tasks, they release code, and the abstract's threat model is practical.\n\nThe problem is that the stated objective cannot produce the results. Eq. 5 defines Lemb as an L2 distance between clean and adversarial embeddings, and Eq. 8 minimizes it. At z=0, Lemb=0, so the gradient is zero and the embedding loss does nothing—or, if it does anything, it pulls the adversarial embedding back toward the clean one, which should preserve downstream accuracy. Yet Tables 1–4 report that the embedding-only variant achieves near-total attack success. The text around Eq. 5 says they want to minimize the dot product between the two embeddings, which is a different, similarity-decreasing objective. So either the equations are wrong or the implementation negates the gradient. As printed, the paper is internally inconsistent and not reproducible. That is a major issue, not a typo.\n\nOther soft spots are more ordinary. The closest prior work, EmbedAttack, is mentioned in the related work but never compared against, which weakens the novelty claim. There are no error bars anywhere, and the transferability table (Table 7) is hard to read because the format mixes task directions oddly. The abstract's phrase \"state-of-the-art results\" overreaches given these gaps.\n\nThat said, I do not think the work is dishonest or hopeless. The attention loss is well motivated, and the differential task sensitivity is a real empirical contribution that the authors apparently did not set out to find. If the code confirms the intended loss is maximization (or negative dot product), the core claim likely survives. A serious referee should engage with this, not desk-reject it, but the revision needs to correct the objective, add EmbedAttack, and report variance. As it stands, I would not cite it.","headline":"The empirical picture is probably real, but the paper's stated objective (minimize an L2 embedding distance) would preserve embeddings, so the reported attack success cannot come from the math as written; fix the sign, add EmbedAttack, and it becomes a credible contribution.","tokens_in":15600,"tokens_out":2090,"would_cite":false,"duration_ms":25944,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single label-free perturbation can disrupt five vision tasks built on CLIP, ViT, and DINOv2 backbones.","keywords":["adversarial attack","vision transformer","CLIP","DINOv2","attention manipulation","task-agnostic attack","embedding disruption","foundation model robustness"],"falsifier":"Run the released code or re-implement the attack exactly as written, minimizing $\\|E_{\\mathrm{gt}} - E_{\\mathrm{adv}}\\|_2$, and measure downstream classification or retrieval accuracy; if the accuracy does not drop substantially, the central claim fails. Alternatively, inspect the optimizer update sign in the code: gradient descent on an $L_2$ distance would decrease the distance, contradicting the reported attack success.","tokens_in":14634,"feed_emoji":"🎯","tokens_out":4328,"duration_ms":48456,"temperature":0.7,"pith_summary":"This paper claims that a single adversarial perturbation computed purely from a vision transformer's attention and embeddings can degrade a broad set of downstream tasks, without access to labels or text. The authors target CLIP, ViT, and DINOv2 backbones and evaluate on classification, image/text retrieval, captioning, depth estimation, and semantic segmentation. If the claim holds, it would show that foundation models inherit a shared vulnerability: whatever task is built on top, damaging attention and the embedding degrades it. The paper also reports that attention-focused disruption hits fine-grained spatial tasks harder, while embedding disruption hits semantic tasks harder.","feed_headline":"One label-free perturbation disrupts five vision tasks","feed_subtitle":"A single attention-plus-embedding attack degrades classification, retrieval, captioning, depth and segmentation on CLIP and ViT models.","key_machinery":"The attack uses two losses. The attention loss $\\mathcal{L}_{\\mathrm{atn}}$ is the dot product, summed over heads and token positions, between the clean attention matrix and the perturbed attention matrix at the last layer, so minimizing it makes perturbed attention orthogonal to clean attention and redistributes focus. The embedding loss $\\mathcal{L}_{\\mathrm{emb}}$ is written in Eq.\\ 5 as the $L_2$ norm between clean and adversarial embeddings, while the accompanying text describes the goal as minimizing the dot product to push embeddings apart. The combined objective $\\mathcal{L}_{\\mathrm{comb}} = \\alpha \\mathcal{L}_{\\mathrm{atn}} + \\beta \\mathcal{L}_{\\mathrm{emb}}$, with $\\beta = |\\mathcal{L}_{\\mathrm{atn}}/\\mathcal{L}_{\\mathrm{emb}}|$ at each iteration, is optimized with Weighted Adam and gradient descent over 250 iterations under a budget $\\epsilon$.","core_discovery":"The central discovery is a task-agnostic adversarial attack on vision foundation models. The perturbation is optimized to make the attention matrix of the last transformer layer nearly orthogonal to the clean attention map, while also shifting the final embedding; the combined loss is minimized with projected gradient descent under an $\\ell_\\infty$ budget. With this single image-only perturbation the authors report near-total accuracy collapse in zero-shot classification, high attack success in cross-modal retrieval, severe caption degradation, and large RMSE and mIoU drops in depth and segmentation models based on DINOv2. The attack needs no downstream labels and no text branch, unlike prior multimodal attacks.","pith_inferences":["If the sign discrepancy in Eq.\\ 5 is confirmed, the reported results likely come from an unstated maximization or a negative dot-product loss; the qualitative conclusion about attention vulnerability may still hold, but the exact mechanism would differ from what the equations describe.","The task-dependent split between attention and embedding attacks suggests a general principle: tasks that rely on spatial layout are more sensitive to attention redistribution, while tasks that rely on global semantics are more sensitive to representation shifts, which could inform robustness testing of future backbones.","A testable extension is to ablate the combined loss with a corrected sign to isolate which component carries the effect; the paper's separate $\\mathcal{L}_{\\mathrm{atn}}$ and $\\mathcal{L}_{\\mathrm{emb}}$ results already hint that the attention term alone drives captioning collapse.","The white-box threat model could be probed against black-box scenarios by querying only attention maps or using transfer from a surrogate encoder; the limited cross-model transfer reported suggests attention-based attacks on self-supervised encoders may behave differently across downstream heads."],"forward_implications":["A single image-only perturbation can degrade multiple tasks built on the same backbone, meaning a downstream deployer cannot assume task specificity protects against adversarial tampering.","Attacks that require neither labels nor text lower the barrier to attacking real deployments of CLIP-like models, since the adversary needs only white-box access to the visual encoder.","Attention-targeted attacks are more effective on fine-grained spatial tasks such as segmentation and depth, while embedding-targeted attacks are more effective on semantic tasks such as classification and retrieval, suggesting task-aware attack selection.","The method's transfer to other vision-language models such as ALBEF and TCL is limited, implying that while task-agnostic, the attack is not fully model-agnostic across different pretraining objectives."],"supporting_citations":[{"why":"CLIP is the primary multimodal foundation model attacked; the visual Transformer backbone is the target of the perturbation.","marker":"[36]"},{"why":"The Vision Transformer (ViT) is the architecture whose attention matrices and embeddings are directly manipulated by the attack.","marker":"[10]"},{"why":"DINOv2 supplies the self-supervised backbone used for the depth estimation and semantic segmentation experiments.","marker":"[35]"},{"why":"EmbedAttack is the prior task-agnostic embedding attack that this work contrasts with, and DepthPGD/SegPGD provide task-specific baselines for depth and segmentation.","marker":"[22]"},{"why":"Set-level Guidance Attack (SGA) is a multimodal baseline that the paper compares against in retrieval and captioning and in transfer experiments.","marker":"[29]"},{"why":"Co-Attack is a multimodal baseline that jointly perturbs image and text; its image-only setting is a direct competitor in retrieval and captioning.","marker":"[48]"},{"why":"BLIP-2 is the captioning model whose output metrics collapse under the attack, demonstrating the disruption of a downstream task built on a CLIP backbone.","marker":"[25]"}],"fun_headline_variants":["Label-free attack breaks five vision tasks","Attention attack on transformers disables five tasks","Single perturbation wreaks havoc on CLIP and ViT","No labels, just one patch: five vision tasks fail","Task-agnostic attack: one perturbation, five vision failures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack's reported success rests on the assumption that the implemented embedding loss actually pushes adversarial embeddings away from clean ones; if the $L_2$ norm in Eq.\\ 5 is minimized literally, the optimizer would pull the embeddings together and the claimed disruption would not occur.","fun_headline_variants_meta":{"raw":{"variants":["Label-free attack breaks five vision tasks","Attention attack on transformers disables five tasks","Single perturbation wreaks havoc on CLIP and ViT","No labels, just one patch: five vision tasks fail","Task-agnostic attack: one perturbation, five vision failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000827,"raw_usage":{"total_tokens":3570,"prompt_tokens":856,"completion_tokens":2714,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2639}},"tokens_in":472,"tokens_out":2714,"duration_ms":23227,"temperature":1.0,"reasoning_tokens":2639,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:05:41.107743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released code or re-implement the attack exactly as written, minimizing $\\|E_{\\mathrm{gt}} - E_{\\mathrm{adv}}\\|_2$, and measure downstream classification or retrieval accuracy; if the accuracy does not drop substantially, the central claim fails. Alternatively, inspect the optimizer update sign in the code: gradient descent on an $L_2$ distance would decrease the distance, contradicting the reported attack success.","supporting_citations":[{"cited_title":"Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks","cited_arxiv_id":"2407.12588","evidence_quote":"EmbedAttack is the prior task-agnostic embedding attack that this work contrasts with, and DepthPGD/SegPGD provide task-specific baselines for depth and segmentation."},{"cited_title":"Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models","cited_arxiv_id":null,"evidence_quote":"Set-level Guidance Attack (SGA) is a multimodal baseline that the paper compares against in retrieval and captioning and in transfer experiments."},{"cited_title":"Towards adversarial attack on vision-language pre-training models","cited_arxiv_id":null,"evidence_quote":"Co-Attack is a multimodal baseline that jointly perturbs image and text; its image-only setting is a direct competitor in retrieval and captioning."},{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"BLIP-2 is the captioning model whose output metrics collapse under the attack, demonstrating the disruption of a downstream task built on a CLIP backbone."}],"review_version":1}