Pith. sign in

REVIEW 3 major objections 6 minor 24 references

A two-stage CLIP fine-tuning and confidence-guided self-training method raises both clean accuracy and adversarial robustness in unsupervised domain adaptation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:15 UTC pith:6AWI6SWL

load-bearing objection Practical CLIP robust-UDA recipe that recombines known pieces and posts real gains on three benchmarks; the 'RL' label is loose and pseudo-label quality under attack is never checked. the 3 major comments →

arxiv 2607.03600 v1 pith:6AWI6SWL submitted 2026-07-03 cs.CV cs.AI

A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning

Sushant Dagaji Desale , Rahul Mishra , Ashutosh Kumar Sinha This is my paper
classification cs.CV cs.AI
keywords unsupervised domain adaptationadversarial robustnessCLIPpseudo-labelingPGDsupervised fine-tuningconfidence thresholdvision-language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Unsupervised domain adaptation usually fails when the target domain is also under adversarial attack, because noisy pseudo-labels and the shift between clean source and perturbed target features destroy accuracy and robustness at once. This paper claims that a simple two-stage procedure on top of a frozen CLIP visual encoder solves the problem: first adversarially fine-tune only a linear head and the projection layer on labeled source data, then refine the model on unlabeled target data by accepting only high-confidence pseudo-labels whose threshold slowly decays. The resulting classifier is trained on mixed clean and PGD-perturbed batches drawn from both domains. On OfficeHome, PACS and VisDA the method lifts clean accuracy by roughly ten points and adversarial robustness by roughly sixteen points over prior robust UDA baselines, showing that partial unfreezing plus a curriculum of filtered pseudo-labels is enough to keep CLIP’s semantic priors while hardening the model against attacks.

Core claim

SFT+RL establishes that adversarially supervised fine-tuning of CLIP’s projection layer on source data, followed by reinforcement-style self-training that admits only high-confidence target pseudo-labels under a decaying threshold, yields a single linear classifier that simultaneously improves clean accuracy and PGD robustness across three standard domain-adaptation benchmarks.

What carries the argument

SFT+RL: the two-stage pipeline that first hardens a partially-unfrozen CLIP projection plus linear head with on-the-fly PGD on labeled source data, then repeatedly filters unlabeled target samples by a decaying confidence threshold τ and retrains the same head on the composite clean-plus-adversarial set.

Load-bearing premise

High-confidence pseudo-labels produced by the partially adapted CLIP head stay accurate enough under the same PGD attack used at test time that the decaying-threshold curriculum does not inject systematic label noise the later adversarial training cannot correct.

What would settle it

On any OfficeHome or PACS domain pair, measure the true accuracy of the filtered pseudo-labels under PGD-20 at the moment they are accepted; if that accuracy falls below roughly 70 percent while the reported robust accuracy still rises, the curriculum claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • CLIP-style vision-language models can be made adversarially robust for domain shift without full backbone fine-tuning or labeled target data.
  • A simple decaying confidence filter is sufficient to turn noisy target predictions into useful training signal under attack.
  • Partial unfreezing of only the projection layer preserves zero-shot semantic priors while still allowing adaptation to adversarial noise.
  • The same two-stage recipe can be applied to other frozen foundation encoders that already possess strong cross-domain features.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same curriculum could be extended to larger multimodal models by replacing the linear head with a lightweight adapter, reducing compute while retaining the robustness gains.
  • If the confidence threshold were itself learned by a small meta-network rather than hand-scheduled, the method might adapt automatically to harder domain gaps or stronger attacks.
  • Failure modes on sketch or synthetic domains suggest the approach still relies on CLIP’s pre-training distribution covering the target style; testing on truly out-of-distribution medical or satellite imagery would expose this limit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SFT+RL, a two-stage framework for adversarially robust unsupervised domain adaptation on top of a frozen CLIP ViT-B/32 visual encoder. Stage 1 (SFT) adversarially fine-tunes a linear classifier and partially unfreezes CLIP’s visual projection layer on labeled source data using 20-step PGD. Stage 2 (called “Reinforcement Learning” but implemented as confidence-driven self-training) generates pseudo-labels on unlabeled target data, retains only those whose max softmax exceeds a decaying threshold τ (starting at 0.85, floor 0.7), forms a composite source+high-confidence-target set, and continues adversarial training on clean/adversarial mixed batches. Evaluations on OfficeHome, PACS and VisDA under PGD-20 (ε=2/255 or 8/255) report average gains of roughly 10% clean accuracy and 15% robust accuracy over DANN, UDA+AT/TRADES/MART, ARTUDA, SRoUDA and DART; an ablation (Table 4) and sensitivity study (Table 5) support the contribution of both stages and of the decaying threshold.

Significance. If the reported gains hold under stronger scrutiny, the work supplies a practical, architecture-compatible recipe for hardening CLIP-style VLMs against simultaneous domain shift and white-box adversarial attack—an important and still under-explored intersection. The partial-unfreezing schedule, the explicit composite-batch adversarial objective, and the decaying-confidence curriculum are concrete, reproducible design choices that other groups can adopt or ablate. The paper already supplies an algorithmic listing (Algorithm 1), a public codebase link, and multi-dataset tables that make the empirical claim falsifiable. These engineering contributions are valuable even if the “RL” nomenclature is loose and even if some statistical reporting remains incomplete.

major comments (3)
  1. Section 3.3 and Algorithm 1 (lines 14–21): the central claim that the RL stage improves robustness rests on the assumption that high-confidence pseudo-labels remain accurate under the same PGD threat model later used at test time. The manuscript never reports the true accuracy (or class-wise error rate) of the filtered set D_conf on clean or adversarially perturbed target images, nor how that accuracy evolves across the three RL cycles. Without this diagnostic, it is impossible to rule out that the subsequent adversarial training simply hardens the model to systematically incorrect targets. A short table of pseudo-label precision/recall (or accuracy under PGD) per cycle would directly test the weakest assumption of the method.
  2. Tables 1–3 and the abstract’s headline averages: the paper states that all numbers are “averaged over three runs” yet never reports standard deviations or confidence intervals. Given that several domain pairs show large absolute jumps (e.g., OfficeHome Pr→Ar clean 43.7→62.1, PACS Sk→Ar clean 71.9→82.4), the absence of variance makes it impossible to judge whether the claimed 10.2 % / 15.8 % average improvements are statistically reliable or driven by a few high-variance transfers. Adding ±std or a simple paired significance test is a load-bearing reporting requirement for the central empirical claim.
  3. Section 3.3 title and surrounding prose repeatedly call the second stage “Reinforcement Learning,” yet the algorithm contains no policy, reward, value function or policy-gradient update—only confidence-thresholded self-training with a decaying τ. This nomenclature is misleading and risks conflating the method with genuine RL-based domain adaptation. Either rename the stage (e.g., “Confidence-Guided Self-Training”) or supply an explicit RL formulation that matches the implemented procedure.
minor comments (6)
  1. Abstract and §1 claim “average improvements of 10.2 % clean / 15.8 % robust”; the precise baseline against which these percentages are computed is never stated (DANN? best prior? mean of all priors?). Clarify the arithmetic.
  2. Figure 1 caption and t-SNE axes are hard to read; the three clusters (source / clean target / adversarial target) are not quantitatively separated (e.g., by silhouette or MMD). A short quantitative measure would strengthen the motivation.
  3. Inconsistent notation: Algorithm 1 uses “SFT+RF” while the rest of the paper uses “SFT+RL”; ε is set to 8/255 in the SFT generation equations but 2/255 in the experimental protocol. Harmonize.
  4. Table 3 (VisDA) uses ε=8/255 while Tables 1–2 use ε=2/255; a brief justification for the different threat models would help readers compare across datasets.
  5. Several citations appear truncated or mistyped (e.g., “tomm-ude” for OfficeHome, missing page numbers). Clean the bibliography.
  6. The claim of “cross-modal curriculum” and “updating the text encoder” (Introduction and contributions) is not reflected in Algorithm 1 or Figure 2, where the text encoder and CTP remain frozen. Either implement the claimed text-side update or remove the claim.

Circularity Check

0 steps flagged

No circularity: empirical method paper whose accuracy/robustness claims are measured on external public benchmarks against independent baselines, not derived by construction from fitted inputs or self-citation.

full rationale

SFT+RL is a two-stage engineering procedure (PGD adversarial fine-tuning of a linear head + CLIP projection on labeled source, followed by confidence-thresholded self-training on unlabeled target with decaying au and mixed clean/adversarial batches). The load-bearing claims are purely empirical averages (+10.2 % clean, +15.8 % robust accuracy) reported in Tables 1–3 and Figure 4 against public datasets (OfficeHome, PACS, VisDA) and external baselines (DANN, UDA+AT/TRADES/MART, ARTUDA, SRoUDA, DART). No equation equates a reported accuracy to a free parameter by definition; the decaying-threshold curriculum (Alg. 1, Eq. 11) and composite loss (Eq. 10) are design choices whose quality is tested by ablation (Table 4) rather than assumed true by construction. There are no uniqueness theorems, no self-citations that close a logical loop, and no renaming of a known closed-form result presented as a derivation. The skeptic’s concern about unmeasured pseudo-label accuracy under PGD is a validity/assumption risk, not circularity. The derivation chain is therefore self-contained and non-circular.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The method rests on standard adversarial-training and self-training assumptions plus a handful of hand-chosen hyper-parameters that control the curriculum and the attack strength. No new physical entities are postulated; the free parameters are ordinary ML knobs.

free parameters (3)
  • initial confidence threshold τ = 0.85 → 0.7
    Set by hand to 0.85 and decayed by 0.9 each cycle down to a floor of 0.7; directly controls which pseudo-labels enter training.
  • PGD attack budget (ε, α, steps) = ε=2/255 (or 8/255), 20 steps
    Fixed to ε=2/255 or 8/255, α=0.5/255 or 2/255, 20 steps for both training and evaluation; chosen rather than derived.
  • learning-rate schedule and unfreezing epoch = 1e-3 / 1e-5, unfreeze after epoch 2
    Classifier lr=1e-3 then 1e-5, projection unfrozen after epoch 2; pure design choices.
axioms (3)
  • domain assumption CLIP ViT-B/32 visual embeddings remain a useful semantic prior after only the final projection layer is lightly adapted.
    Invoked throughout Section 3.1–3.2; never proved, only empirically observed.
  • domain assumption High-confidence softmax predictions on unlabeled target data are sufficiently correct to serve as training labels under the same threat model.
    Core premise of the RL stage (Section 3.3).
  • standard math ℓ∞-bounded PGD is an adequate proxy for the adversarial threat of interest.
    Standard in the adversarial-robustness literature; used for both training and evaluation.

pith-pipeline@v1.1.0-grok45 · 19082 in / 2536 out tokens · 18392 ms · 2026-07-12T01:15:46.104814+00:00 · methodology

0 comments
read the original abstract

Adversarial robustness in Unsupervised Domain Adaptation (UDA) remains a significant challenge due to noisy pseudo labels and inherent distributional shifts between the clean source and adversarially perturbed target domains. Existing approaches often fail to achieve an optimal trade-off between robustness and accuracy, as pseudo-labels generated by domain-adapted models tend to introduce classification errors under adversarial attacks. In this work, we propose \textbf{SFT+RL}, a two-stage robust UDA framework that integrates Supervised Fine Tuning (SFT) and Reinforcement Learning (RL) on top of CLIP's pre-trained visual encoder. In the SFT stage, we adversarially fine-tune a linear classifier using PGD-based perturbations over the labelled source domain while partially unfreezing CLIP's projection layer. It allows adaptation to adversarial noise while preserving CLIP's rich semantic priors. We introduce a confidence-guided pseudo-labeling strategy in the RL stage to annotate unlabeled target samples progressively. Pseudo labels are filtered using a decaying confidence threshold to balance quality and coverage, and the model is trained on a composite dataset formed by combining clean source samples with high-confidence target samples. Adversarial training is applied to mixed batches of clean and adversarial examples to enhance cross-domain robustness. Comprehensive evaluations on three benchmark datasets OfficeHome~\cite{tomm-ude}, PACS~\cite{pacs}, and VisDA~\cite{visda} demonstrate the effectiveness of our approach. Notably, \textbf{SFT+RL} achieves average improvements of \textbf{10.2\%} in clean accuracy and \textbf{15.8\%} in adversarial robustness across all three datasets, outperforming existing state-of-the-art methods.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 4 linked inside Pith

  1. [1]

    Bhagwatkar, S

    R. Bhagwatkar, S. Nayak, P . Bashivan, and I. Rish. Improving adversar- ial robustness in vision-language models with architecture and prompt design. In Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 17003–17020, 2024

    2024

  2. [2]

    Ganin, E

    Y . Ganin, E. Ustinova, H. Ajakan, P . Germain, H. Larochelle, F. Lavi- olette, M. March, and V . Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research , 17(59):1–35, 2016

    2016

  3. [3]

    T. Gao, A. Fisch, and D. Chen. Making pre-trained language models robust to spelling errors. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2735–2750, 2021

    2021

  4. [4]

    Huang, Y

    H. Huang, Y . Wang, S. Erfani, Q. Gu, J. Bailey, and X. Ma. Explor- ing architectural ingredients of adversarially robust deep neural net- works. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P . Liang, and J. W. V aughan, editors,Advances in Neural Information Processing Systems , volume 34, pages 5545–5559, 2021

    2021

  5. [5]

    G. Kang, L. Jiang, Y . Y ang, and A. G. Hauptmann. Contrastive adap- tation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4893–4902, 2019

    2019

  6. [6]

    P . W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsub- ramani, W. Hu, M. Y asunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International confer- ence on machine learning , pages 5637–5664. PMLR, 2021

    2021

  7. [7]

    D. Li, Y . Y ang, Y .-Z. Song, and T. M. Hospedales. Deeper, Broader and Artier Domain Generalization. In Proceedings of the IEEE In- ternational Conference on Computer Vision (ICCV) , pages 5542–5550. IEEE, 2017

    2017

  8. [8]

    D. Li, Y . Y ang, Y .-Z. Song, and T. Hospedales. Learning to general- ize: Meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), 2018

    2018

  9. [9]

    Liang, T

    P . Liang, T. Wang, and Y . Zhang. We need more training data: Eval- uating domain adaptation in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) , 2020

    2020

  10. [10]

    Lo and V

    S.-Y . Lo and V . Patel. Exploring adversarially robust training for un- supervised domain adaptation. In Proceedings of the Asian Conference on Computer Vision, pages 4093–4109, 2022

    2022

  11. [11]

    Madry, A

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017

    Pith/arXiv arXiv 2017

  12. [12]

    Madry, A

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR) , pages 1–23, 2018

    2018

  13. [13]

    X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017

    Pith/arXiv arXiv 2017

  14. [14]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P . Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Con- ference on Machine Learning (ICML) , pages 8748–8763, 2021

    2021

  15. [15]

    Saito, D

    K. Saito, D. Kim, S. Sclaroff, T. Darrell, and K. Saenko. Semi- supervised domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8050–8058, 2019

    2019

  16. [16]

    Schlarmann, N

    C. Schlarmann, N. D. Singh, F. Croce, and M. Hein. Robust clip: Unsu- pervised adversarial fine-tuning of vision embeddings for robust large vision-language models. arXiv preprint arXiv:2402.12336, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Tanwisuth, Y

    K. Tanwisuth, Y . Aytar, and A. Zisserman. Textual redundancy in vision-language models. In Proceedings of the 40th International Con- ference on Machine Learning (ICML) , 2023

    2023

  18. [18]

    V enkateswara, J

    H. V enkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 5018–5027, 2017

    2017

  19. [19]

    H. Wang, G. Wang, Y . Li, D. Zhang, and L. Lin. Transferable, control- lable, and inconspicuous adversarial attacks on person re-identification with deep mis-ranking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 342–351, 2020

    2020

  20. [20]

    Wang and W

    M. Wang and W. Deng. Deep visual domain adaptation: A survey. Neu- rocomputing, 312:135–153, 2018

    2018

  21. [21]

    Y . Wang, H. Hazimeh, N. Ponomareva, A. Kurakin, I. Hammoud, and R. Arora. Dart: A principled approach to adversarially robust unsuper- vised domain adaptation. arXiv preprint arXiv:2402.11120, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Y ang, R

    J. Y ang, R. Xu, R. Li, X. Qi, X. Shen, G. Li, and L. Lin. An adversarial perturbation oriented domain adaptation approach for semantic segmen- tation. Proceedings of the AAAI Conference on Artificial Intelligence , 34(07):12613–12620, 2020

    2020

  23. [23]

    Zhang, Y

    H. Zhang, Y . Y u, J. Jiao, E. Xing, L. E. Ghaoui, and M. Jordan. Theo- retically principled trade-off between robustness and accuracy. In Inter- national Conference on Machine Learning (ICML) , pages 7472–7482, 2019

    2019

  24. [24]

    Zhu, J.-L

    W. Zhu, J.-L. Yin, B.-H. Chen, and X. Liu. SRoUDA: Meta Self- Training for Robust Unsupervised Domain Adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3852–3860, 2023

    2023