Pith. sign in

REVIEW 2 major objections 2 minor 143 references

SS-TPT scores augmented views for stability and suitability to guide prompt tuning and weighted prediction, raising adversarial robustness in vision-language models while cutting the cost of multi-view processing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.3

2026-06-27 22:23 UTC pith:3CNSP6AQ

load-bearing objection SS-TPT pairs weak-aug stability with feature density to guide view selection in test-time VLM prompt tuning, but the link to actual adversarial trustworthiness remains unshown. the 2 major comments →

arxiv 2606.06943 v1 pith:3CNSP6AQ submitted 2026-06-05 cs.CV cs.AI

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

classification cs.CV cs.AI
keywords test-time adaptationadversarial robustnessprompt tuningvision-language modelsaugmented viewsstability score
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SS-TPT to make vision-language models like CLIP more resistant to adversarial attacks during test time. It assigns each augmented view two scores: stability, which checks if predictions stay the same under small changes, and suitability, which checks how densely the view sits among other views in feature space. These scores then steer a consistency loss during prompt adaptation and determine how much each view contributes to the final prediction. The method aims to keep the benefits of many views without paying the full speed penalty of prior approaches. A reader would care because it targets the practical barrier that has kept robust test-time defenses from seeing wide use.

Core claim

SS-TPT evaluates the quality of each augmented view via two complementary scores: stability, measuring prediction invariance to weak augmentations, and suitability, measuring feature-space density among views. These stability and suitability scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones.

What carries the argument

Stability and suitability (SS) scores on augmented views, which control an SS-guided consistency loss for prompt tuning and an SS-weighted prediction at inference time.

Load-bearing premise

The two SS scores reliably mark which augmented views are trustworthy enough to steer prompt adaptation and final predictions without opening new failure modes.

What would settle it

A test set or attack where views that receive high stability and suitability scores produce lower accuracy than either using every view equally or selecting views at random.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SS-TPT significantly outperforms prior state-of-the-art methods on robustness.
  • It achieves superior robustness-throughput trade-offs across diverse datasets.
  • Performance holds when the number of augmented views is varied.
  • The approach shows practicality and generality without model-specific tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the SS scores prove reliable, the same weighting idea could be added to other test-time adaptation techniques beyond prompt tuning.
  • The method might transfer to image-only models that lack the language component.
  • Fewer required views would directly lower memory and energy costs at deployment.
  • The scores could be checked against new attack families to see whether they remain predictive.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper proposes SS-TPT, a test-time prompt tuning approach for improving adversarial robustness of vision-language models such as CLIP. It computes two scores per augmented view—stability (prediction invariance under weak augmentations) and suitability (feature-space density)—and uses them to drive an SS-guided consistency loss during prompt adaptation as well as an SS-weighted ensemble prediction at inference. The central claim is that this guidance yields superior robustness-throughput trade-offs compared with prior state-of-the-art test-time adaptation methods across multiple datasets and different numbers of views.

Significance. If the empirical claims hold after addressing the correlation concern below, the work would provide a practical engineering advance for deploying robust VLMs under realistic compute budgets. The dual use of the same scores for both adaptation and inference is a coherent design choice, and the public code release is a clear strength for reproducibility.

major comments (2)
  1. [§3] §3 (method): Stability is defined exclusively via invariance to weak augmentations, yet the central claim requires that this quantity reliably identifies views that remain accurate under adversarial perturbations. No correlation analysis, per-view accuracy plots under attack, or ablation isolating the contribution of the SS scores versus other implementation choices is presented; without such evidence the robustness gains cannot be attributed to the proposed guidance rather than confounding factors.
  2. [§4] §4 (experiments): The manuscript asserts outperformance “across diverse datasets and varying numbers of views,” but the provided text supplies no attack strengths, baseline implementations, statistical tests, or ablations that remove the SS weighting. These details are load-bearing for the robustness-throughput claim and must be supplied with concrete numbers and controls.
minor comments (2)
  1. [§3] Notation for the two SS scores and the precise form of the consistency loss should be introduced with an equation rather than prose only.
  2. Figure captions should explicitly state the attack norm and strength used for the reported robustness numbers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments correctly identify areas where additional empirical evidence would strengthen the attribution of gains to the SS guidance mechanism. We will revise the manuscript to address both points with new analyses and details.

read point-by-point responses
  1. Referee: [§3] §3 (method): Stability is defined exclusively via invariance to weak augmentations, yet the central claim requires that this quantity reliably identifies views that remain accurate under adversarial perturbations. No correlation analysis, per-view accuracy plots under attack, or ablation isolating the contribution of the SS scores versus other implementation choices is presented; without such evidence the robustness gains cannot be attributed to the proposed guidance rather than confounding factors.

    Authors: We agree that a direct link between weak-augmentation stability and adversarial-view accuracy must be demonstrated rather than assumed. In the revised manuscript we will add (i) scatter plots and Pearson/Spearman correlations between per-view stability scores and adversarial accuracy, (ii) per-view accuracy plots under PGD attack, and (iii) an ablation that replaces the SS-guided loss and weighting with uniform averaging while keeping all other implementation choices fixed. These additions will allow readers to quantify how much of the reported robustness improvement is attributable to the SS scores. revision: yes

  2. Referee: [§4] §4 (experiments): The manuscript asserts outperformance “across diverse datasets and varying numbers of views,” but the provided text supplies no attack strengths, baseline implementations, statistical tests, or ablations that remove the SS weighting. These details are load-bearing for the robustness-throughput claim and must be supplied with concrete numbers and controls.

    Authors: We acknowledge that the experimental section must be expanded for full reproducibility and attribution. The revision will include: explicit attack parameters (PGD-10, ε=4/255, step size 1/255), code-level descriptions of all baseline re-implementations, standard deviations over three random seeds together with paired t-test p-values, and a dedicated ablation table that removes the SS weighting (both in the consistency loss and in the ensemble) while reporting accuracy and throughput (images/sec) for 1-, 4-, 8-, and 16-view settings on ImageNet and CIFAR-10. These concrete numbers and controls will directly support the robustness-throughput claims. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical engineering contribution with independent experimental validation

full rationale

The paper introduces SS-TPT as a test-time adaptation method that defines stability (prediction invariance to weak augmentations) and suitability (feature-space density) scores to guide consistency loss and weighted prediction. No equations, derivations, or first-principles claims are shown that reduce the robustness-throughput results to fitted inputs by construction, self-citations, or renamed known patterns. Central claims rest on empirical comparisons across datasets, which remain falsifiable outside the method definition itself. This is a standard non-circular empirical contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no equations, hyperparameters, or modeling assumptions; therefore the ledger is empty. Full text would be needed to list any fitted thresholds, loss coefficients, or domain assumptions about feature distributions.

pith-pipeline@v0.9.1-grok · 5714 in / 1142 out tokens · 21431 ms · 2026-06-27T22:23:47.581214+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations. Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown and a clear robustness-throughput trade-off. To address this challenge, we present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT), evaluating the quality of each augmented view via two complementary scores: (1) stability, measuring prediction invariance to weak augmentations, and (2) suitability, measuring feature-space density among views. These stability and suitability (SS) scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones. Extensive experiments demonstrate that SS-TPT significantly outperforms prior state-of-the-art methods, achieving superior robustness-throughput trade-offs across diverse datasets and varying numbers of views, thereby demonstrating both strong practicality and generality. Our code is available at https://github.com/sunoh-kim/SS-TPT.

Figures

Figures reproduced from arXiv: 2606.06943 by Daeho Um, Sunoh Kim.

Figure 1
Figure 1. Figure 1: Robustness-throughput trade-off across test-time de￾fenses, averaged over 10 fine-grained classification datasets. Un￾like prior methods that either sacrifice computational efficiency or robustness, our approach consistently achieves the best of both worlds, delivering higher robustness at faster speeds, even with few augmented views. +∆%p indicates the robustness improvement over the strongest previous me… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of per-image test time and adversarial robust￾ness on the Caltech101 dataset across CLIP, R-TPT with different numbers of augmented views (63 and 15 views), and our method with 15 views. These results highlight the trade-off that more views improve robustness but slow down inference, while our approach achieves strong robustness with fast inference. Wang et al., 2025; Sheng et al., 2025) commonl… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of SS-TPT. From a test image, multiple views are generated through augmentations A. Each view is evaluated by two quality scores: (i) stability, measuring invariance under weak augmentations A ′ , and (ii) suitability, assessing feature-space density among views. These scores produce SS weights that guide both adaptation and inference: (1) an SS-guided consistency loss aligns predictions toward mo… view at source ↗
Figure 4
Figure 4. Figure 4: Stability scores, suitability scores, and combined SS scores for the original view (index 0, indicated by the red dot) and its augmented views on (a) clean ImageNet, (b) ImageNet under attack, and (c-d) distribution-shifted variants: ImageNet-S and ImageNet-R. The natural original in (a) has the highest SS score, while the perturbed original in (b) has a near-zero score, and the distribution-shifted origin… view at source ↗
Figure 5
Figure 5. Figure 5: Average SS scores for original and adversarially attacked images on (a) ImageNet, (b) ImageNet-S, and (c) ImageNet-R. Scores are averaged over 500 random samples per dataset. 0.1 0.2 0.3 0.5 1.0 2.0 3.0 5.0 Loss Balance ( ) 79.5 80.0 80.5 81.0 Adversarial Robustness (a) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Trade-off in SS Weighting ( ) 80.0 80.5 81.0 Adversarial Robustness (b) 70.0 72.5 75.0 77.5 80… view at source ↗
Figure 6
Figure 6. Figure 6: Ablation study on the Caltech101 dataset showing (a) the impact of the loss balance λ, (b) the trade-off between stability and suitability, and (c) robustness under variations in prompt templates. 25.8%, improving over R-TPT by +15.7 points and over TTC by +12.1 points. Clean accuracy is 53.3%, establishing the highest performance among all adaptation methods. Test-time efficiency. As shown in Tab. 5, SS-T… view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of adversarial accuracy under different reference selection strategies for Lscons on the Caltech101 dataset. The SS-based selection methods consistently outperform random, average, and confidence-based selection methods. variance to stochastic perturbations is particularly beneficial. Crucially, leveraging both scores consistently outperforms relying on either one alone (i.e., α=0 or α=1). Sensi… view at source ↗
Figure 9
Figure 9. Figure 9: Robustness across different optimized parameter con￾figurations on the Caltech101 dataset. The results consistently demonstrate state-of-the-art performance, indicating that our ap￾proach remains effective not only for text prompt tuning but also for other optimization schemes, such as encoder tuning and visual prompt tuning. Comparison to training-time defenses. Tab. 11 shows that our test-time method ach… view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of clean accuracy (Acc.) and adversarial accuracy (Rob.) under PGD attack (ϵ = 4/255, step 10) across two architectures: EVA02-B-16 and OpenCLIP(VIT-B-32, LAION pretrained). Results are evaluated on three fine-grained datasets. generality of SS-TPT, we conduct additional experiments on different vision-language model architectures. Tab. 13 shows that SS-TPT achieves the highest adversarial accu… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

143 extracted references · 5 canonical work pages · 3 internal anchors

  1. [1]

    CVPR , year=

    When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustness , author=. CVPR , year=

  2. [2]

    IEEE Transactions on Multimedia , year =

    Finding Optimal Video Moment without Training: Gaussian Boundary Optimization for Weakly Supervised Video Grounding , author =. IEEE Transactions on Multimedia , year =

  3. [3]

    MM , year=

    Learnable Negative Proposals Using Dual-Signed Cross-Entropy Loss for Weakly Supervised Video Moment Localization , author=. MM , year=

  4. [4]

    AAAI , year=

    Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Grounding , author=. AAAI , year=

  5. [5]

    2022 , booktitle =

    Kim, Sunoh and Ha, Taegil and Yun, Kimin and Choi, Jin Young , title =. 2022 , booktitle =

  6. [6]

    AVSS , year=

    Enhancing Weakly Supervised Video Grounding via Diverse Inference Strategies for Boundary and Prediction Selection , author=. AVSS , year=

  7. [7]

    Position-aware Location Regression Network for Temporal Video Grounding , year=

    Kim, Sunoh and Yun, Kimin and Choi, Jin Young , booktitle=. Position-aware Location Regression Network for Temporal Video Grounding , year=

  8. [8]

    ICML , year=

    Learning transferable visual models from natural language supervision , author=. ICML , year=

  9. [9]

    IEEE transactions on pattern analysis and machine intelligence , year=

    Vision-language models for vision tasks: A survey , author=. IEEE transactions on pattern analysis and machine intelligence , year=

  10. [10]

    Machine Intelligence Research , year=

    Vlp: A survey on vision-language pre-training , author=. Machine Intelligence Research , year=

  11. [11]

    ACL , year=

    Clip models are few-shot learners: Empirical studies on vqa and visual entailment , author=. ACL , year=

  12. [12]

    CVPR , year=

    Regionclip: Region-based language-image pretraining , author=. CVPR , year=

  13. [13]

    CVPR , year=

    Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment , author=. CVPR , year=

  14. [14]

    ICML , year=

    Scaling up visual and vision-language representation learning with noisy text supervision , author=. ICML , year=

  15. [15]

    NeurIPS , year=

    Visual instruction tuning , author=. NeurIPS , year=

  16. [16]

    ICML , year=

    Zero-shot text-to-image generation , author=. ICML , year=

  17. [17]

    NeurIPS , year=

    Photorealistic text-to-image diffusion models with deep language understanding , author=. NeurIPS , year=

  18. [18]

    Transactions on Machine Learning Research , year=

    Coca: Contrastive captioners are image-text foundation models , author=. Transactions on Machine Learning Research , year=

  19. [19]

    ICCV , year=

    Enhancing adversarial robustness via test-time transformation ensembling , author=. ICCV , year=

  20. [20]

    AAAI , year=

    Combating adversaries with anti-adversaries , author=. AAAI , year=

  21. [21]

    CVPR , year=

    On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach , author=. CVPR , year=

  22. [22]

    ICLR , year=

    Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models , author=. ICLR , year=

  23. [23]

    CVPR , year=

    Pre-trained model guided fine-tuning for zero-shot adversarial robustness , author=. CVPR , year=

  24. [24]

    NeurIPS , year=

    On evaluating adversarial robustness of large vision-language models , author=. NeurIPS , year=

  25. [25]

    ICCV , year=

    Test-Time Prompt Tuning for Zero-Shot Depth Completion , author=. ICCV , year=

  26. [26]

    ICCV , year=

    Hierarchical Variational Test-Time Prompt Generation for Zero-Shot Generalization , author=. ICCV , year=

  27. [27]

    NeurIPS , year=

    Test-time prompt tuning for zero-shot generalization in vision-language models , author=. NeurIPS , year=

  28. [28]

    ICCV , year=

    Diverse data augmentation with diffusions for effective test-time prompt tuning , author=. ICCV , year=

  29. [29]

    NeurIPS , year=

    Swapprompt: Test-time prompt adaptation for vision-language models , author=. NeurIPS , year=

  30. [30]

    AAAI , year=

    Robust test-time adaptation for zero-shot prompt tuning , author=. AAAI , year=

  31. [31]

    Test-Time Adaptation with

    Shuai Zhao and Xiaohan Wang and Linchao Zhu and Yi Yang , booktitle=. Test-Time Adaptation with

  32. [32]

    ICLR , year=

    C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion , author=. ICLR , year=

  33. [33]

    NeurIPS , year=

    Historical test-time prompt tuning for vision foundation models , author=. NeurIPS , year=

  34. [34]

    Pattern Recognition , year=

    CTPT: Continual Test-time Prompt Tuning for vision-language models , author=. Pattern Recognition , year=

  35. [35]

    ICLR , year=

    Dynaprompt: Dynamic test-time prompt tuning , author=. ICLR , year=

  36. [36]

    CVPR , year=

    O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models , author=. CVPR , year=

  37. [37]

    CVPR , year=

    Tapt: Test-time adversarial prompt tuning for robust inference in vision-language models , author=. CVPR , year=

  38. [38]

    AAAI , year=

    Black-Box Test-Time Prompt Tuning for Vision-Language Models , author=. AAAI , year=

  39. [39]

    International Journal of Computer Vision , year=

    CAT-TPT: Class-Agnostic Text-based Test-time Prompt Tuning for Vision-Language Models , author=. International Journal of Computer Vision , year=

  40. [40]

    CVPR , year=

    R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning , author=. CVPR , year=

  41. [41]

    CVPR , year=

    Hierarchical Knowledge Prompt Tuning for Multi-task Test-Time Adaptation , author=. CVPR , year=

  42. [42]

    arXiv preprint arXiv:2210.04831 (2022)

    Visual prompt tuning for test-time domain adaptation , author=. arXiv preprint arXiv:2210.04831 , year=

  43. [43]

    ACM MM , year=

    Vpa: Fully test-time visual prompt adaptation , author=. ACM MM , year=

  44. [44]

    ACM MM , year=

    Towards Robustness Prompt Tuning with Fully Test-Time Adaptation for CLIP's Zero-Shot Generalization , author=. ACM MM , year=

  45. [45]

    ICCV , year=

    Why is prompt tuning for vision-language models robust to noisy labels? , author=. ICCV , year=

  46. [46]

    CVPR , year=

    Argue: Attribute-guided prompt tuning for vision-language models , author=. CVPR , year=

  47. [47]

    CVPR , year=

    Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning , author=. CVPR , year=

  48. [48]

    ICCV , year=

    Knowledge-aware prompt tuning for generalizable vision-language models , author=. ICCV , year=

  49. [49]

    ICCV , year=

    Distribution-aware prompt tuning for vision-language models , author=. ICCV , year=

  50. [50]

    CVPR , year=

    Adversarial Domain Prompt Tuning and Generation for Single Domain Generalization , author=. CVPR , year=

  51. [51]

    CVPR , year=

    MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning , author=. CVPR , year=

  52. [52]

    CVPR , year=

    Scene map-based prompt tuning for navigation instruction generation , author=. CVPR , year=

  53. [53]

    CVPR , year=

    Tcp: Textual-based class-aware prompt tuning for visual-language model , author=. CVPR , year=

  54. [54]

    CVPR , year=

    Visual-language prompt tuning with knowledge-guided context optimization , author=. CVPR , year=

  55. [55]

    CVPR , year=

    One prompt word is enough to boost adversarial robustness for pre-trained vision-language models , author=. CVPR , year=

  56. [56]

    ECCV , year=

    Adversarial prompt tuning for vision-language models , author=. ECCV , year=

  57. [57]

    NeurIPS , year=

    Few-shot adversarial prompt learning on vision-language models , author=. NeurIPS , year=

  58. [58]

    CVPR , year=

    Conditional prompt learning for vision-language models , author=. CVPR , year=

  59. [59]

    International Journal of Computer Vision , year=

    Learning to prompt for vision-language models , author=. International Journal of Computer Vision , year=

  60. [60]

    CVPR , year=

    Maple: Multi-modal prompt learning , author=. CVPR , year=

  61. [61]

    NeurIPS , year=

    The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models , author=. NeurIPS , year=

  62. [62]

    ICCV , year=

    Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models , author=. ICCV , year=

  63. [63]

    ICLR , year=

    Tent: Fully test-time adaptation by entropy minimization , author=. ICLR , year=

  64. [64]

    ICML , year=

    Test-time training with self-supervision for generalization under distribution shifts , author=. ICML , year=

  65. [65]

    ECCV , year=

    Tip-adapter: Training-free clip-adapter for better vision-language modeling , author=. ECCV , year=

  66. [66]

    CVPR , year=

    Bayesian test-time adaptation for vision-language models , author=. CVPR , year=

  67. [67]

    CVPR , year=

    COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation , author=. CVPR , year=

  68. [68]

    CVPR , year=

    Realistic test-time adaptation of vision-language models , author=. CVPR , year=

  69. [69]

    CVPR , year=

    Effortless active labeling for long-term test-time adaptation , author=. CVPR , year=

  70. [70]

    CVPR , year=

    Scap: Transductive test-time adaptation via supportive clique-based attribute prompting , author=. CVPR , year=

  71. [71]

    CVPR , year=

    Clip is strong enough to fight back: Test-time counterattacks towards zero-shot adversarial robustness of clip , author=. CVPR , year=

  72. [72]

    NeurIPS , year=

    Enhancing CLIP Robustness via Cross-Modality Alignment , author=. NeurIPS , year=

  73. [73]

    AAAI , year=

    Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference , author=. AAAI , year=

  74. [74]

    CVPR , year=

    On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? , author=. CVPR , year=

  75. [75]

    CVPR , year=

    Efficient test-time adaptation of vision-language models , author=. CVPR , year=

  76. [76]

    ICCV , year=

    Improving adversarial robustness of masked autoencoders via test-time frequency-domain prompting , author=. ICCV , year=

  77. [77]

    CVPR , year=

    Tesla: Test-time self-learning with automatic adversarial augmentation , author=. CVPR , year=

  78. [78]

    CVPR , year=

    Robust mean teacher for continual and gradual test-time adaptation , author=. CVPR , year=

  79. [79]

    NeurIPS , year=

    Frustratingly easy test-time adaptation of vision-language models , author=. NeurIPS , year=

  80. [80]

    ICLR , year=

    Efficiently learning at test-time: Active fine-tuning of llms , author=. ICLR , year=

Showing first 80 references.