Pith. sign in

REVIEW 4 major objections 4 minor 26 references

Open Set Domain Adaptation with Vision-language models via Gradient-aware Separation

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims unknown target-domain samples in open-set domain adaptation can be detected from the L2 norm of CLIP prompt gradients.

desk verdict Genuinely new gradient-norm observation for CLIP prompt tuning, but the derivation is invalid as written and the experiments don't seem to test the claimed mechanism; not ready for serious review. read the letter →

arxiv 2505.13507 v1 pith:WRY3AB6T submitted 2025-05-16 cs.LG cs.CV

Haoyang Chen This is my paper · ORCID
classification cs.LGcs.CV
keywords opensetdomainadaptationCLIPgradientnormpromptlearningunknownsampledetectionvision-languagemodelspseudo-labelingOffice-Home
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a method for open-set domain adaptation with CLIP, where the target domain contains classes the source never saw. Its central claim is that unknown samples can be separated from shared-class samples by the L2 norm of the loss gradient with respect to the learnable text prompts: unknowns produce larger gradient norms. The paper derives this ordering from a chain-rule identity, explains why it reverses the classic GradNorm ordering, and uses the norm as a threshold-based unknown detector that requires no separate detector network or unknown labels. The framework also includes domain-conditioned prompt tuning and a split pseudo-label loss, with experiments on Office-Home reporting consistent improvement over zero-shot CLIP and CoOp.

What carries the argument

The central object is the L2 norm of the gradient of a KL-divergence loss between the uniform distribution and the model's softmax output, taken with respect to CLIP's learnable prompt embeddings. The paper's identity for this gradient is $\frac{\partial l}{\partial w} = \frac{\partial v}{\partial w}^\top \mathrm{diag}(z)\left(p - \frac{1}{K}\mathbf{1}\right)$, in which $z$ is the image feature, $p$ is the softmax of the class similarities, and $K$ is the number of classes. The identity turns a softmax-confidence statement into a geometric statement: a peaked $p$ leaves few nonzero entries in the gradient and shrinks its norm, while a flatter $p$ spreads nonzero entries across prompt columns and enlarges the norm. This is the mechanism that makes unknown detection threshold-based and free of unknown-class supervision.

What would settle it

Compute the exact gradient of the KL-divergence loss with respect to the learnable prompts using CLIP's normalized cosine similarity, including the Jacobian of $z/\|z\|$ and $v/\|v\|$, and measure the L2 norms for held-out known and unknown target images; if unknown samples do not consistently yield larger gradient norms than shared-class samples across the source-target splits of Office-Home, the central detection rule is falsified.

Watch

Extended reading notes

Core claim

The discovery is that the gradient norm flips direction relative to GradNorm once prompts are learnable. For CLIP with soft prompts, the loss-gradient with respect to prompt weights contains the factor $\partial v/\partial w^\top$ and a block-diagonal image-feature matrix acting on $p - \frac{1}{K}\mathbf{1}$, where $p$ is the softmax of CLIP's cosine similarities. When target images are drawn from unknown classes, temperature-scaled softmax probabilities are flatter, so more entries of $p - \frac{1}{K}\mathbf{1}$ are non-negligible and the L2 gradient norm becomes larger. Hence the paper's rule: large gradient norm means unknown. A threshold calibrated so that 90% of source samples are called known transfers the separation to the target domain, and the known/unknown division is then used to apply cross-entropy to shared-class pseudo-labels and KL divergence toward uniform for unknowns.

Load-bearing premise

The derivation in Section 3.1 treats CLIP's cosine similarity as a plain inner product $z^\top v$, but CLIP actually uses L2-normalized features whose gradient includes projection and scaling terms that Eq. (6) omits; if those terms are included, the paper's Eq. (7) does not follow, and the claimed ordering of gradient norms is not established by the paper's own mathematics.

Editorial extensions

If this is right

  • If the ordering holds, unknown target classes can be flagged from a scalar gradient norm calibrated on the source domain alone, eliminating the need for a separately trained unknown detector and for labeled unknown data.
  • The same gradient-norm score can be reused across the 12 Office-Home source-target tasks, since thresholds set at 90% source recall transfer to the target domain without additional tuning.
  • Separating target samples before alignment lets the objective apply cross-entropy only to confident shared-class pseudo-labels and push uncertain samples toward a uniform distribution, reducing error accumulation in later adaptation steps.
  • Because CoOp's quality degrades under domain shift, conditioning prompts on a domain-discrepancy measure becomes a necessary ingredient for prompt-based open-set domain adaptation.
  • The reported gains over zero-shot CLIP and CoOp on AUROC, FPR95, and CCR@FPR10 indicate that prompt learning can be made robust under open-set domain shift when paired with gradient-aware separation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to use the prompt-gradient norm as a prompt- or test-time adaptation signal beyond open-set domain adaptation, for example in continual learning or source-free adaptation; the paper does not test these settings.
  • Because the separation is driven by softmax flatness, interventions such as temperature scaling or entropy maximization could widen the known-unknown gradient gap; this is a testable prediction that the paper leaves implicit.
  • The reversal of GradNorm's ordering suggests that the direction of gradient-based OOD scores is architecture- and parameterization-dependent, so a same-score pipeline may need a per-module sign check when moved to other prompt-tuned encoders.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an open-set domain adaptation (OSDA) method based on CLIP prompt learning. The two main components are domain-conditioned textual prompt tuning and a gradient-norm-based detector for unknown target-domain samples. The central claim, stated in Section 3.1, is that the L2 norm of the loss gradient with respect to the learnable prompt parameters is larger for unknown-class samples than for shared-class samples, so a source-calibrated threshold on this norm can separate known and unknown target samples. The method then applies cross-entropy loss to the pseudo-labeled shared class and KL divergence regularization to the detected unknown class. Experiments on Office-Home compare against zero-shot CLIP and CoOp, and an ablation study is reported. The paper also includes a derivation of the gradient formula in Section 3.1.

Significance. If the gradient-norm ordering claim were valid, the paper would introduce a conceptually interesting and computationally simple unknown-detection criterion for CLIP-based OSDA, avoiding the need for a separately trained unknown detector. The paper also clearly states its assumptions and provides a detailed gradient-flow diagram. However, the central derivation is mathematically incomplete, the qualitative argument for the ordering is contradicted by the norm of the stated vector, and the experimental results do not consistently support the claimed advantage. The proposed method is therefore not established as a reliable contribution to OSDA.

major comments (4)
  1. [Section 3.1, Eq. (6)] The derivation of the gradient norm ordering is not valid because Eq. (6) treats CLIP's cosine similarity as a plain inner product. In CLIP, the similarity is f_i = z^T v_i with both z and v_i normalized. The derivative of f_i with respect to v_i includes the projection term (z - f_i v_i)/||v_i||, which is omitted from the block-diagonal form in Eq. (6). This missing term changes the magnitude of the gradient and can reverse the claimed ordering between known and unknown samples, so Eq. (7) does not follow from the premises.
  2. [Section 3.1, Eq. (7)] Even if Eq. (6) were accepted, the norm argument leading to the conclusion that unknown samples have larger gradient norms is unsupported. The vector M in Eq. (7) has norm proportional to ||p - (1/K)1||. For a peaked in-distribution probability, say p_i = 0.99 and K = 65, this norm is approximately 0.98, whereas for a diffuse unknown distribution with three entries of 0.3 the norm is approximately 0.5. The paper's claim that the Jacobian N amplifies the diffuse case more than the peaked case is not backed by any property of N, whose norm is left uncharacterized. Thus the central theoretical claim of Section 3.1 is not established.
  3. [Tables 1-4] The abstract and introduction state that the method 'consistently outperforms' CLIP and CoOp, but several rows in the results tables contradict this. For example, in Table 3 (Rw to Pr) Ours is worse than CoOp on all three metrics (Acc10 78.27 vs 80.79, FPR95 42.07 vs 40.94, AUROC 91.96 vs 92.29), and in Table 4 (Cl to Ar) Ours has lower Acc10 than CLIP (62.54 vs 63.19). Additionally, the experiments compare only against CLIP and CoOp, with no comparison to existing OSDA methods, so the claim of superiority in the open-set domain adaptation setting is not demonstrated.
  4. [Table 5 and Section 3.2] The ablation study does not isolate the gradient-norm selector, which is the key novelty of the method. Table 5 reports rows for CLIP, '+CE', and '+KL' separately, but no row corresponds to the full method (CE + KL with gradient-based separation) or to a control where the gradient-norm threshold is replaced by another detection score. The text claims that 'ablation studies confirm the gradient norm's critical role,' but the presented experiments cannot support that claim.
minor comments (4)
  1. [General] The manuscript contains numerous typographical and grammatical errors, including 'Learnabl e', 'cosin e', 'pseudo label s', and 'the results of experiment is opposite of GradNorm'. A careful proofreading pass is needed.
  2. [Section 3.2, Eq. (8)] Eq. (8) is incomplete and inconsistent with the hyperparameter description. The term 𝛼𝒑𝑦̂ is not defined, and the variable γ, which is said to be set to 0.001 in Section 4.2, does not appear in Eq. (8). The loss formulation should be stated explicitly with all terms and variables defined.
  3. [Section 3.1, citation [16]] The statement that CLIP lacks a final fully connected layer is cited to reference [16], which is a paper about filtering random noise in language model pre-training data. This citation does not support the statement and appears to be a referencing error.
  4. [References] The reference list has inconsistencies and duplicates: references [19] and [20] are both the same Busto & Gall (2017) paper, and reference [25] duplicates the content of [15]. The numbering in the text also seems misaligned in places, for example reference [16] as noted above.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the derivation has correctness gaps but is not equivalent to its inputs by construction.

full rationale

Walking the claimed derivation chain in Sec. 3.1, the gradient-norm detector is not defined in terms of the unknown/known labels: the gradient norm is computed from Eq. (7) as a function of prompt parameters and softmax probabilities, and the decision threshold is calibrated on source data (90% ID) and carried over to target. This is a standard source-calibrated detector, not a fitted prediction of target unknown-ness. The method is benchmarked against CLIP and CoOp on Office-Home with AUROC/FPR95/CCR, so the headline results are not baked in by construction. The pseudo-label step in Sec. 3.2 is a transductive self-training loop, which can reinforce whatever separation the gradient metric produces, but it does not make the final evaluation equal to the loss definition; pseudo-labels are not ground-truth and the tables are scored against true labels. The main concern with the paper is correctness, not circularity: Eq. (6) drops the L2-normalization derivative of CLIP's cosine similarity, and the conclusion that the Jacobian N=∂v/∂w^T amplifies diffuse distributions more than peaked ones is asserted, not derived. These are mathematical/empirical gaps, not equation-identity reductions. No load-bearing self-citation or imported uniqueness claim appears. Hence no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or architectural entities are introduced; the method reuses CLIP and CoOp prompts. The free parameters are standard hyperparameters plus the source-calibrated threshold. The main hidden load is the normalization-free cosine-similarity derivative and the assumed transferability of the source threshold.

free parameters (5)
  • alpha = 0.1
    Weight for the target-domain cross-entropy loss on pseudo-labeled shared-class samples.
  • beta = 0.01
    Weight for the KL divergence toward uniform on target samples identified as unknown.
  • gamma = 0.001
    Listed among hyperparameters but does not appear in the loss equation (8), so its role is undefined.
  • source threshold for ID = 90th percentile of source gradient norms
    A fixed fraction of source samples are declared ID and the resulting gradient-norm threshold is applied to target samples.
  • number of learnable prompts = 4
    Prompt count chosen by hand, initialized with 'a photo of a'.
assumptions (4)
  • ad hoc to paper CLIP text-image similarity can be differentiated as a plain inner product z^T v.
    Eq. (6) sets the derivative of the cosine similarity to z^T, ignoring L2 normalization and its projection terms. This is the load-bearing approximation behind Eq. (7).
  • domain assumption Known-class samples have a dominant softmax probability near 0.99 while unknown samples have flatter distributions.
    Section 3.1 uses this empirical pattern to argue that the gradient-norm ordering flips relative to GradNorm. No statistical evidence is provided for this pattern across Office-Home.
  • domain assumption A gradient-norm threshold set on source domain samples transfers to the target domain.
    The paper uses the 90th percentile of source gradient norms as the ID/unknown boundary for target samples, assuming the distribution of known-class gradient norms is stable across domains.
  • domain assumption Unknown-class target outputs should be regularized toward a uniform distribution.
    The KL loss in Eq. (8) assumes uniform output is the right prior for unknown samples, which is not derived or measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Open Set Domain Adaptation with Vision-language models via Gradient-aware Separation." pith.science (2026). https://pith.science/paper/WRY3AB6T

@misc{pith2026250513507,
  author       = {Pith},
  title        = {Pith review of: Open Set Domain Adaptation with Vision-language models via Gradient-aware Separation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WRY3AB6T}},
  note         = {Machine review of arXiv:2505.13507}
}
read the original abstract

Open-Set Domain Adaptation (OSDA) confronts the dual challenge of aligning known-class distributions across domains while identifying target-domain-specific unknown categories. Current approaches often fail to leverage semantic relationships between modalities and struggle with error accumulation in unknown sample detection. We propose to harness Contrastive Language-Image Pretraining (CLIP) to address these limitations through two key innovations: 1) Prompt-driven cross-domain alignment: Learnable textual prompts conditioned on domain discrepancy metrics dynamically adapt CLIP's text encoder, enabling semantic consistency between source and target domains without explicit unknown-class supervision. 2) Gradient-aware open-set separation: A gradient analysis module quantifies domain shift by comparing the L2-norm of gradients from the learned prompts, where known/unknown samples exhibit statistically distinct gradient behaviors. Evaluations on Office-Home show that our method consistently outperforms CLIP baseline and standard baseline. Ablation studies confirm the gradient norm's critical role.

Figures

Figures reproduced from arXiv: 2505.13507 by the authors.

Figure 1
Figure 1. Illustration of covariate shift and open [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The gradient flow of learnable soft prompts. Black lines indicate forward process, and red [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

  1. [1]

    (2015, June)

    Ganin, Y., & Lempitsky, V. (2015, June). Unsupervised domain adaptation by backpropagation. In International conference on machine learning (pp. 1180-1189). PMLR

    work page 2015

  2. [2]

    Li, J., Yu, Z., Du, Z., Zhu, L., & Shen, H. T. (2024). A comprehensive survey on source-free domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence

    work page 2024

  3. [4]

    Ru, J., Tian, J., Xiao, C., Li, J., & Shen, H. T. (2023). Imbalanced open set domain adaptation via moving-threshold estimation and gradual alignment. IEEE Transactions on Multimedia, 26, 2504-2514

    work page 2023

  4. [5]

    Zhuang, X., Xie, Y., Deng, Y., Liang, L., Ru, J., Yin, Y., & Zou, Y. (2025). VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model. arXiv preprint arXiv:2501.12327

    arXiv 2025

  5. [6]

    Du, Z., Li, X., Li, F., Lu, K., Zhu, L., & Li, J. (2024). Domain -agnostic mutual prompting for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23375- 23384)

    work page 2024

  6. [7]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.,

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever , I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PmLR

    work page 2021

  7. [8]

    C., & Liu, Z

    Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022). Learning to prompt for vision -language models. International Journal of Computer Vision, 130(9), 2337-2348

    work page 2022

  8. [9]

    Ming, Y., Cai, Z., Gu, J., Sun, Y., Li, W., & Li, Y. (2022). Delving into out -of-distribution detection with vision - language representations. Advances in neural information processing systems, 35, 35087-35102

    work page 2022

Show all 26 references
  1. [10]

    Huang, R., Geng, A., & Li, Y. (2021). On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34, 677-689

    2021

  2. [11]

    Liu, W., Wang, X., Owens, J., & Li, Y. (2020). Energy -based out -of-distribution detection. Advances in neural information processing systems, 33, 21464-21475

    2020

  3. [12]

    & Zou, Y

    Yin, Y., Xie, Y., Yang, W., Yang, D., Ru, J., Zhuang, X., ... & Zou, Y. (2025). ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errors. arXiv preprint arXiv:2502.14627

    2025 arXiv

  4. [13]

    Fort, S., Ren, J., & Lakshminarayanan, B. (2021). Exploring the limits of out-of-distribution detection. Advances in neural information processing systems, 34, 7068-7081

    2021

  5. [14]

    Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024). Generalized out -of-distribution detection: A survey. International Journal of Computer Vision, 132(12), 5635-5662

    2024

  6. [16]

    Ru, J., Xie, Y., Zhuang, X., Yin, Y., & Zou, Y. (2025). Do we really have to filter out random noise in pre -training data for language models?. arXiv preprint arXiv:2502.06604

    2025 arXiv

  7. [17]

    Venkateswara, H., Eusebio, J., Chakraborty, S., & Panchanathan, S. (2017). Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5018-5027)

    2017

  8. [18]

    H., & Budiharto, W

    Triestyarso, A., Kartowisastro, I. H., & Budiharto, W. (2024). Investigating Catastrophic Forgetting of Deep Learning Models within Office 31 Dataset. IEEE Access

    2024

  9. [19]

    (2024, March)

    Zhuang, X., Cheng, X., & Zou, Y. (2024, March). Towards explainable joint models via information theory for multiple intent detection and slot filling. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 17, pp. 19786-19794)

    2024

  10. [20]

    Busto, P. P. , & Gall, J. . (2017). Open set domain adaptation. IEEE

    2017

  11. [21]

    Panareda Busto, P., & Gall, J. (2017). Open set domain adaptation. In Proceedings of the IEEE international conference on computer vision (pp. 754-763)

    2017

  12. [22]

    , Cao, Z

    Liu, H. , Cao, Z. , Long, M. , Wang, J. , & Yang, Q. . (2020). Separate to Adapt: Open Set Domain Adaptation via Progressive Separation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE

    2020

  13. [23]

    Lu, J., Batra, D., Parikh, D., & Lee, S. (2019). Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32

    2019

  14. [24]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.,

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PmLR

    2021

  15. [25]

    (2022, June)

    Sun, Y., Ming, Y., Zhu, X., & Li, Y. (2022, June). Out -of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning (pp. 20827-20840). PMLR

    2022

  16. [26]

    Chen, J., Li, J., Qu, X., Wang, J., Wan, J., & Xiao, J. (2023). Gaia: Delving into gradient -based attribution abnormality for out -of-distribution detection. Advances in Neural Information Processing Systems, 36, 79946 - 79958

    2023

  17. [27]

    Wang, H., Li, Z., Feng, L., & Zhang, W. (2022). Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4921-4930)

    2022

  18. [28]

    Nie, J., Zhang, Y., Fang, Z., Liu, T., Han, B., & Tian, X. (2024). Out-of-distribution detection with negative prompts. In The twelfth international conference on learning representations

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.