{"id":"a7ebfcd7-4bc2-440e-ac4d-7d46bfccca81","arxiv_id":"2411.13136","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"TAPT adapts CLIP's prompts per test image using entropy minimization and ImageNet-based embedding alignment, but it is evaluated only against non-adaptive attacks.","lead":"This paper proposes TAPT, a method that rewrites CLIP's text and image prompts at test time to defend against adversarial images. The reported robustness jumps from near zero to about 50% accuracy, but the attacks tested do not know about the defense, so the real-world margin is uncertain.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The defense is evaluated only against attacks on the original CLIP, not on the TAPT pipeline, so the white-box robustness claim is untested; an adaptive attack is needed.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the evaluation does not account for adaptive attacks against TAPT despite the stated white-box threat model. I agree with this assessment. TAPT is differentiable through the prompt optimization, and the defense's stochastic augmentations are a recognized challenge that requires expectation-over-transformation attacks, not an excuse to omit them. Without an adaptive attack, the reported AutoAttack numbers measure robustness only against non-adaptive attacks that ignore the defense, which does not support the abstract's claim of enhanced zero-shot adversarial robustness under a white-box adversary. The paper's own limitation section does not address this omission. Since this is the central empirical claim, the REJECT verdict is appropriate and no adjustment is needed.","tokens_in":17123,"tokens_out":3268,"duration_ms":38982,"concrete_test":"Reproduce the TAPT-VLI row of Table 1 with an adaptive attack: generate AutoAttack or PGD-100 perturbations against the full TAPT inference function, using the cross-entropy after the per-sample prompt optimization as the attack loss. Backpropagate through the one-step prompt update (unrolling the AdamW step) and average gradients over at least 50 random augmentation sets to handle the stochastic view selection, at the same epsilon = 1/255. If the robust accuracy drops substantially (e.g., below 10% or close to the 17.2% APT-VLI baseline), the headline robustness gain is not supported under the stated white-box threat model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TAPT provides white-box adversarial robustness for CLIP. Section 3.1 defines the threat model as an attacker with full knowledge of the target model's architecture and parameters, optimizing Eq. (1) against the model. However, the attack generation in Section 4.2 appears to target the undefended CLIP: Table 1 reports PGD, DI, and AutoAttack results where the attacks are presumably generated on the vanilla model, not through TAPT's test-time prompt optimization. TAPT is a differentiable input-dependent defense: Algorithm 1 optimizes prompts on augmented views of the test image, and an attacker with full knowledge can backpropagate through this optimization to craft perturbations that account for the defense. The stochastic augmentations would require expectation-over-transformation attacks. Because no adaptive attack is reported, the reported robustness numbers (e.g., AutoAttack accuracy rising from 0.1% to 48.9%) may reflect a mismatch between the attack surrogate and the actual defended model. This is not a minor evaluation gap; it invalidates the claimed white-box guarantee under the paper's own threat model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Test-Time Adversarial Prompt Tuning (TAPT), an unsupervised test-time defense for CLIP that optimizes visual and textual prompts per test image by minimizing multi-view entropy and aligning test-embedding statistics with precomputed clean/adversarial statistics from ImageNet. It claims to improve zero-shot adversarial robustness over vanilla CLIP by at least 48.9% against AutoAttack and over APT baselines by at least 36.6%, evaluated on 11 datasets with ViT-B/16 and ViT-B/32 backbones.","tokens_in":17290,"tokens_out":8022,"duration_ms":72057,"significance":"If validated, TAPT would be a significant contribution: it is the first task-agnostic test-time defense for pre-trained VLMs, lifting CLIP's near-zero AutoAttack accuracy to roughly 50% while maintaining clean accuracy. The paper is well-structured, examines three prompt designs, provides ablations on TAPT steps, perturbation budgets, and reset intervals, and reports per-dataset numbers across 11 benchmarks. The method is conceptually simple and requires no downstream labels. However, the central white-box robustness claim is currently not supported by the evaluation, and the comparison to APT baselines may be confounded by an unequal data budget for initialization; these issues are load-bearing and need to be addressed before the significance can be assessed.","major_comments":[{"comment":"The white-box threat model in Section 3.1 gives the attacker full knowledge of the target model and has the attacker optimize Eq. (1) directly against that model. The evaluation in Table 1, however, appears to use adversarial examples generated against the undefended CLIP; the attack configuration in Section 4.2 never states that PGD-100, DI, or AutoAttack is run through TAPT's test-time prompt optimization. Since Algorithm 1 is differentiable and the defense includes stochastic augmentations, a white-box attacker could compute gradients through the optimization or use an expectation-over-transformation attack. Without such adaptive attack results, the reported AutoAttack accuracies (e.g., 48.9% for TAPT-V) do not establish robustness under the paper's own white-box threat model; they only measure transferability of attacks from the vanilla CLIP surrogate. This is load-bearing for the central claim and requires new experiments.","section":"Section 3.1 / Section 4.2"},{"comment":"The comparison against APT baselines is potentially unfair. Table 1's caption states that the APT baselines (APT-V, APT-VLJ, APT-VLI) were tuned on ImageNet under a 16-shot setting, while Section 4.1's Implementation Details says TAPT initializes its defensive prompt using APT on ImageNet trained for 100 epochs with a batch size of 32. If the TAPT initialization uses substantially more (or the full) ImageNet data than the 16-shot baselines, the reported average robustness improvement of 36.6% over APT-VLI conflates the benefit of test-time adaptation with a stronger initialization. The authors should state the exact training data budget for the TAPT initialization and either match the baselines' data budget or ablate the initialization effect.","section":"Section 4.2 / Section 4.1"},{"comment":"All results are single point estimates without error bars or repeated trials. Because TAPT involves stochastic augmentations and per-sample prompt optimization, the reported 'at least' improvements (e.g., 48.9% and 36.6%) could vary across runs. Providing means and standard deviations over multiple seeds is necessary to support the strength of the claim, especially for datasets with smaller margins such as Aircraft in Table 1.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"Line 6 of Algorithm 1 says 'Select the top τ entropy from A(x) to form Hτ(x)', which is inconsistent with the text's 'top-K views with the lowest entropy' and with the implementation's 'top 10% most confident predictions'. Please define whether τ is the fraction or the number of views and how the threshold is computed.","section":"Algorithm 1"},{"comment":"The phrase 'enhances ... by at least 48.9%' should be 'to at least 48.9%' or 'by 48.8 percentage points', since the vanilla AutoAttack accuracy is 0.1% and the TAPT-V average is 48.9%.","section":"Abstract"},{"comment":"The claim of 'zero-shot adversarial robustness' is overstated for ImageNet, since TAPT uses ImageNet-derived statistics and an APT initialization trained on ImageNet; the evaluation on the other 10 datasets is zero-shot, but ImageNet is in-distribution.","section":"Section 4.1"},{"comment":"The combined loss in Eq. (8) has no explicit weight for the entropy term relative to the alignment terms; if the two terms have different scales, the behavior of α may not match the description. Please report the relative scaling or normalization.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not mention code or data release. The proposed defense is conceptually interesting, but the white-box evaluation gap is the main obstacle; I would encourage the editor to request adaptive attacks and a data-budget-matched baseline comparison before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is that TAPT is a genuinely new idea—test-time prompt tuning as an adversarial defense for CLIP—but the evaluation doesn't actually attack the defended model. The white-box threat model in Section 3.1 assumes an attacker with full knowledge of the pipeline, and TAPT is differentiable, so an adaptive attacker could backpropagate through the per-sample prompt optimization. The experiments only use PGD, DI, and AutoAttack generated against the undefended CLIP. That makes the reported gains (e.g., AA accuracy from ~0.1% to ~49%) averages against the wrong surrogate. This is not a minor gap; it means the central claim of white-box robustness is untested. The authors need expectation-over-transformation or at least a standard adaptive attack.\n\nWhat's worth keeping is the method itself. The adversarial-clean alignment loss that matches test embedding statistics to pre-computed ImageNet statistics (both adversarial and clean) is a reasonable way to steer the prompt without labels. The multi-view entropy selection comes from TPT/PromptAlign, but combining it with the alignment loss is a sensible contribution. The paper writes all this up clearly, and the non-adaptive results are strong across 11 datasets and two backbones.\n\nThe soft spots beyond the threat model: no error bars in Tables 1 and 2, and the number of TAPT steps in the main results is inconsistent—the text says one step, but the ablation varies 0,1,2,4 without stating which produced Table 1. The abstract's 'at least 48.9%' is technically the average improvement over vanilla, but since the baseline is near zero, it reads as more impressive than it is. There's also no comparison to other test-time defenses for adversarial robustness (e.g., MEMO-style entropy minimization), which would help contextualize the gains. The lack of code is a practical downside but not a scientific flaw.\n\nOn balance, this is a paper for the adversarial robustness and VLM community. It deserves a serious referee because the idea is novel and the failure mode is well understood and potentially fixable. But in its current form, the robustness claims don't hold under the paper's own threat model. I'd send it to review with a strong request for adaptive attacks, and I expect a major revision or rejection otherwise.","headline":"TAPT is a novel test-time defense for CLIP, but the evaluation only attacks the undefended model, so the white-box robustness claim is unsupported.","tokens_in":17852,"tokens_out":3175,"would_cite":false,"duration_ms":28461,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that test-time tuning of bimodal prompts, driven by multi-view entropy and alignment to ImageNet embedding statistics, lifts CLIP's zero-shot AutoAttack accuracy from an average of about 0.1% to about 49% across 11…","keywords":["test-time defense","adversarial robustness","vision-language models","CLIP","prompt tuning","zero-shot learning","AutoAttack","multimodal"],"falsifier":"Run a white-box attack that computes gradients through the entire TAPT pipeline, including the per-sample prompt update and multi-view selection, and measure AutoAttack-style robustness on the same 11 datasets. If the average robust accuracy falls to near the undefended level (about 0.1%) rather than staying near the reported 48.9% to 53.8%, the central claim fails under the paper's stated white-box threat model.","tokens_in":1662,"feed_emoji":"🛡️","tokens_out":1539,"duration_ms":80374,"temperature":0.7,"pith_summary":"The paper proposes Test-Time Adversarial Prompt Tuning (TAPT), a defense that adapts CLIP's prompts for each test image on the fly to resist adversarial perturbations in the visual modality. The central claim is that this unsupervised, task-agnostic procedure raises CLIP's zero-shot robustness under AutoAttack from an average of about 0.1% to about 49% across 11 datasets, while largely preserving clean-image accuracy. TAPT also reports average robustness gains of 36.6 to 38.0 percentage points over existing adversarial prompt tuning baselines. If correct, this would give pre-trained vision-language models a practical, training-free route to robust inference on unseen tasks.","feed_headline":"Test-time prompt tuning lifts CLIP's AutoAttack accuracy to ~49%","feed_subtitle":"A task-agnostic defense raises zero-shot robustness across 11 datasets while keeping clean-image accuracy","key_machinery":"The machinery is a per-sample optimization loop over learnable prompt tokens appended to CLIP's image and text inputs. Two unsupervised losses drive the update: multi-view entropy, which averages prediction probabilities over the selected low-entropy augmented views and minimizes the entropy of that average, and adversarial-clean embedding alignment, which matches the mean and variance of the test image's intermediate embeddings, layer by layer, to pre-computed statistics of adversarial and clean ImageNet embeddings. The combined objective is $L_{\\mathrm{TAPT}} = L_{\\mathrm{entropy}} + \\alpha L_{\\mathrm{adv}} + (1-\\alpha)L_{\\mathrm{clean}}$ with $\\alpha = 0.5$, and the update is a single AdamW step per inference. The image and text encoders stay frozen; the prompt is the only moving part, which lets the defense adapt to each sample while leaving the pre-trained model intact.","core_discovery":"TAPT's central claim is that a prompt adapted to each test image can absorb the distribution shift introduced by adversarial perturbations, restoring alignment between the image embedding and its text embedding. Starting from a robust prompt initialized by adversarial prompt tuning on ImageNet, the method generates 64 augmented views of the test image, keeps the 10% with the lowest entropy in their averaged predictions, and performs one optimization step on the prompt. The step minimizes a combined loss: multi-view entropy of the averaged prediction plus $\\ell^1$ alignment of the mean and variance of intermediate embeddings against pre-computed adversarial and clean statistics from ImageNet. On the paper's evaluation, this lifts average AutoAttack accuracy from 0.1% for vanilla CLIP to 48.9% with visual-only prompts and 53.8% with V-L independent prompts on ViT-B/16, while clean accuracy remains at or above the adversarial prompt tuning baselines.","pith_inferences":["If the unadapted-attack assumption is the real source of the gains, then an adaptive attacker who differentiates through TAPT's prompt update would likely erode most of the reported margin; that test is absent from the paper.","The choice of ImageNet as the public dataset is a transferability bet: the alignment losses compare test embeddings to ImageNet-derived statistics, so datasets far from ImageNet's distribution may see smaller robustness gains than the average suggests.","The same mechanism could extend beyond image recognition: replacing image-embedding statistics with joint image-text statistics might defend multimodal attacks, and the per-sample reset policy could be relaxed in trusted settings where poisoning is not a concern.","TAPT's reliance on augmented views means the quality of the augmentation distribution matters; a testable extension would measure how robustness changes when augmentations are specialized to the target domain."],"forward_implications":["Zero-shot adversarial robustness becomes a test-time property: the same frozen CLIP can be defended on an unseen dataset with no labels or task-specific tuning.","The defense composes with existing adversarial prompt tuning: initializing from an ImageNet-tuned robust prompt is part of what makes a single adaptation step effective.","The V-L independent prompt design gives the largest average AutoAttack gains over its APT baseline, suggesting separate visual and textual prompts are easier to optimize jointly than shared ones.","Because prompts reset every sample, robustness does not accumulate across samples; the paper reports that longer reset intervals slightly improve ImageNet accuracy but hurt the ten zero-shot datasets and open a poisoning risk.","The time overhead is modest enough for per-image adaptation, at 0.095 to 0.166 seconds per image, so the defense is usable in latency-sensitive settings."],"supporting_citations":[{"why":"Supplies CLIP, the pre-trained vision-language model whose zero-shot inference is the target of the defense.","marker":"[33]"},{"why":"Defines AutoAttack, the strongest attack used in evaluation and the source of the paper's headline robustness figures.","marker":"[8]"},{"why":"Supplies PGD, the white-box attack used both for evaluation and for generating adversarial examples during prompt tuning.","marker":"[25]"},{"why":"Provides the DI black-box attack used to measure transfer-based robustness without querying the defended model.","marker":"[53]"},{"why":"One of the adversarial prompt tuning baselines and the source of the prompt initialization strategy for TAPT.","marker":"[22]"},{"why":"Adversarial prompt tuning baseline that, together with [22], defines the APT methods TAPT improves on.","marker":"[58]"},{"why":"Introduces test-time prompt tuning with multi-view entropy selection, which TAPT adapts for adversarial robustness.","marker":"[39]"},{"why":"Supplies the idea of aligning test embeddings to a reference distribution at test time, which TAPT extends to adversarial-clean alignment.","marker":"[1]"},{"why":"Provides the zero-shot adversarial robustness evaluation setup and perturbation budgets used in the experiments.","marker":"[27]"}],"fun_headline_variants":["Test-time prompts lift CLIP's AutoAttack accuracy from 0.1% to 49%","Adaptive prompts at inference time thwart adversarial attacks on CLIP","Zero-shot defense: test-time prompts raise CLIP's AutoAttack accuracy to 49%","One test-time prompt step boosts CLIP's adversarial accuracy to ~50%"],"cache_read_input_tokens":19968,"weakest_assumption_plain":"The evaluation assumes attackers craft perturbations against the original CLIP model, not against TAPT's own differentiable tuning process, even though a determined attacker could in principle optimize the perturbation through the full defense.","fun_headline_variants_meta":{"raw":{"variants":["Test-time prompts lift CLIP's AutoAttack accuracy from 0.1% to 49%","Adaptive prompts at inference time thwart adversarial attacks on CLIP","Zero-shot defense: test-time prompts raise CLIP's AutoAttack accuracy to 49%","One test-time prompt step boosts CLIP's adversarial accuracy to ~50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000504,"raw_usage":{"total_tokens":2467,"prompt_tokens":955,"completion_tokens":1512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1436}},"tokens_in":571,"tokens_out":1512,"duration_ms":12581,"temperature":1.0,"reasoning_tokens":1436,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:47:39.471606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a white-box attack that computes gradients through the entire TAPT pipeline, including the per-sample prompt update and multi-view selection, and measure AutoAttack-style robustness on the same 11 datasets. If the average robust accuracy falls to near the undefended level (about 0.1%) rather than staying near the reported 48.9% to 53.8%, the central claim fails under the paper's stated white-box threat model.","supporting_citations":[{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Supplies CLIP, the pre-trained vision-language model whose zero-shot inference is the target of the defense."},{"cited_title":"Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks","cited_arxiv_id":null,"evidence_quote":"Defines AutoAttack, the strongest attack used in evaluation and the source of the paper's headline robustness figures."},{"cited_title":"Towards deep learning models resistant to adversarial attacks","cited_arxiv_id":null,"evidence_quote":"Supplies PGD, the white-box attack used both for evaluation and for generating adversarial examples during prompt tuning."},{"cited_title":"One prompt word is enough to boost adversarial robustness for pre-trained vision-language models","cited_arxiv_id":null,"evidence_quote":"One of the adversarial prompt tuning baselines and the source of the prompt initialization strategy for TAPT."},{"cited_title":"Adversarial prompt tuning for vision-language models","cited_arxiv_id":null,"evidence_quote":"Adversarial prompt tuning baseline that, together with [22], defines the APT methods TAPT improves on."},{"cited_title":"Test- time prompt tuning for zero-shot generalization in vision- language models","cited_arxiv_id":null,"evidence_quote":"Introduces test-time prompt tuning with multi-view entropy selection, which TAPT adapts for adversarial robustness."},{"cited_title":"Align your prompts: Test-time prompting with distribution align- ment for zero-shot generalization","cited_arxiv_id":null,"evidence_quote":"Supplies the idea of aligning test embeddings to a reference distribution at test time, which TAPT extends to adversarial-clean alignment."},{"cited_title":"Understanding zero-shot adversarial robust- ness for large-scale models","cited_arxiv_id":null,"evidence_quote":"Provides the zero-shot adversarial robustness evaluation setup and perturbation budgets used in the experiments."}],"review_version":1}