REVIEW 2 major objections 2 minor 143 references
SS-TPT scores augmented views for stability and suitability to guide prompt tuning and weighted prediction, raising adversarial robustness in vision-language models while cutting the cost of multi-view processing.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-27 22:23 UTC pith:3CNSP6AQ
load-bearing objection SS-TPT pairs weak-aug stability with feature density to guide view selection in test-time VLM prompt tuning, but the link to actual adversarial trustworthiness remains unshown. the 2 major comments →
SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
SS-TPT evaluates the quality of each augmented view via two complementary scores: stability, measuring prediction invariance to weak augmentations, and suitability, measuring feature-space density among views. These stability and suitability scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones.
What carries the argument
Stability and suitability (SS) scores on augmented views, which control an SS-guided consistency loss for prompt tuning and an SS-weighted prediction at inference time.
Load-bearing premise
The two SS scores reliably mark which augmented views are trustworthy enough to steer prompt adaptation and final predictions without opening new failure modes.
What would settle it
A test set or attack where views that receive high stability and suitability scores produce lower accuracy than either using every view equally or selecting views at random.
If this is right
- SS-TPT significantly outperforms prior state-of-the-art methods on robustness.
- It achieves superior robustness-throughput trade-offs across diverse datasets.
- Performance holds when the number of augmented views is varied.
- The approach shows practicality and generality without model-specific tuning.
Where Pith is reading between the lines
- If the SS scores prove reliable, the same weighting idea could be added to other test-time adaptation techniques beyond prompt tuning.
- The method might transfer to image-only models that lack the language component.
- Fewer required views would directly lower memory and energy costs at deployment.
- The scores could be checked against new attack families to see whether they remain predictive.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SS-TPT, a test-time prompt tuning approach for improving adversarial robustness of vision-language models such as CLIP. It computes two scores per augmented view—stability (prediction invariance under weak augmentations) and suitability (feature-space density)—and uses them to drive an SS-guided consistency loss during prompt adaptation as well as an SS-weighted ensemble prediction at inference. The central claim is that this guidance yields superior robustness-throughput trade-offs compared with prior state-of-the-art test-time adaptation methods across multiple datasets and different numbers of views.
Significance. If the empirical claims hold after addressing the correlation concern below, the work would provide a practical engineering advance for deploying robust VLMs under realistic compute budgets. The dual use of the same scores for both adaptation and inference is a coherent design choice, and the public code release is a clear strength for reproducibility.
major comments (2)
- [§3] §3 (method): Stability is defined exclusively via invariance to weak augmentations, yet the central claim requires that this quantity reliably identifies views that remain accurate under adversarial perturbations. No correlation analysis, per-view accuracy plots under attack, or ablation isolating the contribution of the SS scores versus other implementation choices is presented; without such evidence the robustness gains cannot be attributed to the proposed guidance rather than confounding factors.
- [§4] §4 (experiments): The manuscript asserts outperformance “across diverse datasets and varying numbers of views,” but the provided text supplies no attack strengths, baseline implementations, statistical tests, or ablations that remove the SS weighting. These details are load-bearing for the robustness-throughput claim and must be supplied with concrete numbers and controls.
minor comments (2)
- [§3] Notation for the two SS scores and the precise form of the consistency loss should be introduced with an equation rather than prose only.
- Figure captions should explicitly state the attack norm and strength used for the reported robustness numbers.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback. The comments correctly identify areas where additional empirical evidence would strengthen the attribution of gains to the SS guidance mechanism. We will revise the manuscript to address both points with new analyses and details.
read point-by-point responses
-
Referee: [§3] §3 (method): Stability is defined exclusively via invariance to weak augmentations, yet the central claim requires that this quantity reliably identifies views that remain accurate under adversarial perturbations. No correlation analysis, per-view accuracy plots under attack, or ablation isolating the contribution of the SS scores versus other implementation choices is presented; without such evidence the robustness gains cannot be attributed to the proposed guidance rather than confounding factors.
Authors: We agree that a direct link between weak-augmentation stability and adversarial-view accuracy must be demonstrated rather than assumed. In the revised manuscript we will add (i) scatter plots and Pearson/Spearman correlations between per-view stability scores and adversarial accuracy, (ii) per-view accuracy plots under PGD attack, and (iii) an ablation that replaces the SS-guided loss and weighting with uniform averaging while keeping all other implementation choices fixed. These additions will allow readers to quantify how much of the reported robustness improvement is attributable to the SS scores. revision: yes
-
Referee: [§4] §4 (experiments): The manuscript asserts outperformance “across diverse datasets and varying numbers of views,” but the provided text supplies no attack strengths, baseline implementations, statistical tests, or ablations that remove the SS weighting. These details are load-bearing for the robustness-throughput claim and must be supplied with concrete numbers and controls.
Authors: We acknowledge that the experimental section must be expanded for full reproducibility and attribution. The revision will include: explicit attack parameters (PGD-10, ε=4/255, step size 1/255), code-level descriptions of all baseline re-implementations, standard deviations over three random seeds together with paired t-test p-values, and a dedicated ablation table that removes the SS weighting (both in the consistency loss and in the ensemble) while reporting accuracy and throughput (images/sec) for 1-, 4-, 8-, and 16-view settings on ImageNet and CIFAR-10. These concrete numbers and controls will directly support the robustness-throughput claims. revision: yes
Circularity Check
No circularity; empirical engineering contribution with independent experimental validation
full rationale
The paper introduces SS-TPT as a test-time adaptation method that defines stability (prediction invariance to weak augmentations) and suitability (feature-space density) scores to guide consistency loss and weighted prediction. No equations, derivations, or first-principles claims are shown that reduce the robustness-throughput results to fitted inputs by construction, self-citations, or renamed known patterns. Central claims rest on empirical comparisons across datasets, which remain falsifiable outside the method definition itself. This is a standard non-circular empirical contribution.
Axiom & Free-Parameter Ledger
read the original abstract
Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations. Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown and a clear robustness-throughput trade-off. To address this challenge, we present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT), evaluating the quality of each augmented view via two complementary scores: (1) stability, measuring prediction invariance to weak augmentations, and (2) suitability, measuring feature-space density among views. These stability and suitability (SS) scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones. Extensive experiments demonstrate that SS-TPT significantly outperforms prior state-of-the-art methods, achieving superior robustness-throughput trade-offs across diverse datasets and varying numbers of views, thereby demonstrating both strong practicality and generality. Our code is available at https://github.com/sunoh-kim/SS-TPT.
Figures
Reference graph
Works this paper leans on
-
[1]
CVPR , year=
When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustness , author=. CVPR , year=
-
[2]
IEEE Transactions on Multimedia , year =
Finding Optimal Video Moment without Training: Gaussian Boundary Optimization for Weakly Supervised Video Grounding , author =. IEEE Transactions on Multimedia , year =
-
[3]
MM , year=
Learnable Negative Proposals Using Dual-Signed Cross-Entropy Loss for Weakly Supervised Video Moment Localization , author=. MM , year=
-
[4]
AAAI , year=
Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Grounding , author=. AAAI , year=
-
[5]
2022 , booktitle =
Kim, Sunoh and Ha, Taegil and Yun, Kimin and Choi, Jin Young , title =. 2022 , booktitle =
2022
-
[6]
AVSS , year=
Enhancing Weakly Supervised Video Grounding via Diverse Inference Strategies for Boundary and Prediction Selection , author=. AVSS , year=
-
[7]
Position-aware Location Regression Network for Temporal Video Grounding , year=
Kim, Sunoh and Yun, Kimin and Choi, Jin Young , booktitle=. Position-aware Location Regression Network for Temporal Video Grounding , year=
-
[8]
ICML , year=
Learning transferable visual models from natural language supervision , author=. ICML , year=
-
[9]
IEEE transactions on pattern analysis and machine intelligence , year=
Vision-language models for vision tasks: A survey , author=. IEEE transactions on pattern analysis and machine intelligence , year=
-
[10]
Machine Intelligence Research , year=
Vlp: A survey on vision-language pre-training , author=. Machine Intelligence Research , year=
-
[11]
ACL , year=
Clip models are few-shot learners: Empirical studies on vqa and visual entailment , author=. ACL , year=
-
[12]
CVPR , year=
Regionclip: Region-based language-image pretraining , author=. CVPR , year=
-
[13]
CVPR , year=
Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment , author=. CVPR , year=
-
[14]
ICML , year=
Scaling up visual and vision-language representation learning with noisy text supervision , author=. ICML , year=
-
[15]
NeurIPS , year=
Visual instruction tuning , author=. NeurIPS , year=
-
[16]
ICML , year=
Zero-shot text-to-image generation , author=. ICML , year=
-
[17]
NeurIPS , year=
Photorealistic text-to-image diffusion models with deep language understanding , author=. NeurIPS , year=
-
[18]
Transactions on Machine Learning Research , year=
Coca: Contrastive captioners are image-text foundation models , author=. Transactions on Machine Learning Research , year=
-
[19]
ICCV , year=
Enhancing adversarial robustness via test-time transformation ensembling , author=. ICCV , year=
-
[20]
AAAI , year=
Combating adversaries with anti-adversaries , author=. AAAI , year=
-
[21]
CVPR , year=
On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach , author=. CVPR , year=
-
[22]
ICLR , year=
Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models , author=. ICLR , year=
-
[23]
CVPR , year=
Pre-trained model guided fine-tuning for zero-shot adversarial robustness , author=. CVPR , year=
-
[24]
NeurIPS , year=
On evaluating adversarial robustness of large vision-language models , author=. NeurIPS , year=
-
[25]
ICCV , year=
Test-Time Prompt Tuning for Zero-Shot Depth Completion , author=. ICCV , year=
-
[26]
ICCV , year=
Hierarchical Variational Test-Time Prompt Generation for Zero-Shot Generalization , author=. ICCV , year=
-
[27]
NeurIPS , year=
Test-time prompt tuning for zero-shot generalization in vision-language models , author=. NeurIPS , year=
-
[28]
ICCV , year=
Diverse data augmentation with diffusions for effective test-time prompt tuning , author=. ICCV , year=
-
[29]
NeurIPS , year=
Swapprompt: Test-time prompt adaptation for vision-language models , author=. NeurIPS , year=
-
[30]
AAAI , year=
Robust test-time adaptation for zero-shot prompt tuning , author=. AAAI , year=
-
[31]
Test-Time Adaptation with
Shuai Zhao and Xiaohan Wang and Linchao Zhu and Yi Yang , booktitle=. Test-Time Adaptation with
-
[32]
ICLR , year=
C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion , author=. ICLR , year=
-
[33]
NeurIPS , year=
Historical test-time prompt tuning for vision foundation models , author=. NeurIPS , year=
-
[34]
Pattern Recognition , year=
CTPT: Continual Test-time Prompt Tuning for vision-language models , author=. Pattern Recognition , year=
-
[35]
ICLR , year=
Dynaprompt: Dynamic test-time prompt tuning , author=. ICLR , year=
-
[36]
CVPR , year=
O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models , author=. CVPR , year=
-
[37]
CVPR , year=
Tapt: Test-time adversarial prompt tuning for robust inference in vision-language models , author=. CVPR , year=
-
[38]
AAAI , year=
Black-Box Test-Time Prompt Tuning for Vision-Language Models , author=. AAAI , year=
-
[39]
International Journal of Computer Vision , year=
CAT-TPT: Class-Agnostic Text-based Test-time Prompt Tuning for Vision-Language Models , author=. International Journal of Computer Vision , year=
-
[40]
CVPR , year=
R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning , author=. CVPR , year=
-
[41]
CVPR , year=
Hierarchical Knowledge Prompt Tuning for Multi-task Test-Time Adaptation , author=. CVPR , year=
-
[42]
arXiv preprint arXiv:2210.04831 (2022)
Visual prompt tuning for test-time domain adaptation , author=. arXiv preprint arXiv:2210.04831 , year=
-
[43]
ACM MM , year=
Vpa: Fully test-time visual prompt adaptation , author=. ACM MM , year=
-
[44]
ACM MM , year=
Towards Robustness Prompt Tuning with Fully Test-Time Adaptation for CLIP's Zero-Shot Generalization , author=. ACM MM , year=
-
[45]
ICCV , year=
Why is prompt tuning for vision-language models robust to noisy labels? , author=. ICCV , year=
-
[46]
CVPR , year=
Argue: Attribute-guided prompt tuning for vision-language models , author=. CVPR , year=
-
[47]
CVPR , year=
Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning , author=. CVPR , year=
-
[48]
ICCV , year=
Knowledge-aware prompt tuning for generalizable vision-language models , author=. ICCV , year=
-
[49]
ICCV , year=
Distribution-aware prompt tuning for vision-language models , author=. ICCV , year=
-
[50]
CVPR , year=
Adversarial Domain Prompt Tuning and Generation for Single Domain Generalization , author=. CVPR , year=
-
[51]
CVPR , year=
MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning , author=. CVPR , year=
-
[52]
CVPR , year=
Scene map-based prompt tuning for navigation instruction generation , author=. CVPR , year=
-
[53]
CVPR , year=
Tcp: Textual-based class-aware prompt tuning for visual-language model , author=. CVPR , year=
-
[54]
CVPR , year=
Visual-language prompt tuning with knowledge-guided context optimization , author=. CVPR , year=
-
[55]
CVPR , year=
One prompt word is enough to boost adversarial robustness for pre-trained vision-language models , author=. CVPR , year=
-
[56]
ECCV , year=
Adversarial prompt tuning for vision-language models , author=. ECCV , year=
-
[57]
NeurIPS , year=
Few-shot adversarial prompt learning on vision-language models , author=. NeurIPS , year=
-
[58]
CVPR , year=
Conditional prompt learning for vision-language models , author=. CVPR , year=
-
[59]
International Journal of Computer Vision , year=
Learning to prompt for vision-language models , author=. International Journal of Computer Vision , year=
-
[60]
CVPR , year=
Maple: Multi-modal prompt learning , author=. CVPR , year=
-
[61]
NeurIPS , year=
The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models , author=. NeurIPS , year=
-
[62]
ICCV , year=
Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models , author=. ICCV , year=
-
[63]
ICLR , year=
Tent: Fully test-time adaptation by entropy minimization , author=. ICLR , year=
-
[64]
ICML , year=
Test-time training with self-supervision for generalization under distribution shifts , author=. ICML , year=
-
[65]
ECCV , year=
Tip-adapter: Training-free clip-adapter for better vision-language modeling , author=. ECCV , year=
-
[66]
CVPR , year=
Bayesian test-time adaptation for vision-language models , author=. CVPR , year=
-
[67]
CVPR , year=
COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation , author=. CVPR , year=
-
[68]
CVPR , year=
Realistic test-time adaptation of vision-language models , author=. CVPR , year=
-
[69]
CVPR , year=
Effortless active labeling for long-term test-time adaptation , author=. CVPR , year=
-
[70]
CVPR , year=
Scap: Transductive test-time adaptation via supportive clique-based attribute prompting , author=. CVPR , year=
-
[71]
CVPR , year=
Clip is strong enough to fight back: Test-time counterattacks towards zero-shot adversarial robustness of clip , author=. CVPR , year=
-
[72]
NeurIPS , year=
Enhancing CLIP Robustness via Cross-Modality Alignment , author=. NeurIPS , year=
-
[73]
AAAI , year=
Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference , author=. AAAI , year=
-
[74]
CVPR , year=
On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? , author=. CVPR , year=
-
[75]
CVPR , year=
Efficient test-time adaptation of vision-language models , author=. CVPR , year=
-
[76]
ICCV , year=
Improving adversarial robustness of masked autoencoders via test-time frequency-domain prompting , author=. ICCV , year=
-
[77]
CVPR , year=
Tesla: Test-time self-learning with automatic adversarial augmentation , author=. CVPR , year=
-
[78]
CVPR , year=
Robust mean teacher for continual and gradual test-time adaptation , author=. CVPR , year=
-
[79]
NeurIPS , year=
Frustratingly easy test-time adaptation of vision-language models , author=. NeurIPS , year=
-
[80]
ICLR , year=
Efficiently learning at test-time: Active fine-tuning of llms , author=. ICLR , year=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.