{"id":"0cdae25d-59e0-4b7d-b5b8-36b7d1d38b0d","arxiv_id":"2505.13507","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A gradient-norm heuristic applied to CLIP prompt weights is proposed to detect unknown classes in open-set domain adaptation, with Office-Home experiments against CLIP and CoOp.","lead":"This paper proposes a CLIP-based method for open-set domain adaptation that uses the gradient norm of learned prompt vectors to distinguish known from unknown target samples. It reports gains over plain CLIP and CoOp on Office-Home, but the comparisons are narrow and the gradient derivation omits cosine-normalization terms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6) drops the L2-normalization derivative in CLIP's cosine similarity, and even granting Eq. (6), Eq. (7) does not imply larger gradient norms for unknown samples; the Jacobian is uncharacterized and the raw p-1/K norm is larger for peaked ID distributions. Experiments do not repair the gap.","rationale":"The reader's REJECT verdict is reinforced. The central claim is the gradient-norm ordering that underlies the open-set separation mechanism. The only analytical support is Section 3.1. Eq. (6) is incorrect for normalized CLIP embeddings: the missing projection term is not negligible and can change the sign or order of the gradient-norm comparison. Independently, even accepting Eq. (6), Eq. (7) does not logically imply larger norms for unknowns because the Jacobian N is uncharacterized and the raw vector p - (1/K)1 has larger norm for peaked ID distributions than for diffuse unknown distributions. The paper's sparsity argument is an informal assertion, not a bound. The experiments do not compensate: Tables 1-4 contain multiple rows where Ours is worse than CoOp or CLIP, contradicting the claimed consistent superiority. Table 5 does not isolate the gradient selector, so the claimed critical role is untested. No code, data, or formal verification is provided, leaving the math as the only support. A correct autograd re-computation is the decisive check. Since the reader already reached REJECT and this stress-test identifies the same central weakness with additional detail, the verdict does not need to change.","tokens_in":8826,"tokens_out":10189,"duration_ms":101015,"concrete_test":"On Office-Home with CLIP ViT-B/16 and the paper's CoOp settings (4 prompts, SGD, same alpha/beta), compute for each target sample both (a) the paper's Eq. (7) gradient using ∂f/∂v = z^T and (b) the true autograd gradient of D_KL(u||p) with respect to the learnable prompts, computed through CLIP's normalized cosine similarity and text-encoder Jacobian. Measure AUROC of each gradient norm for separating novel classes from shared classes averaged over the 12 splits. If ||g_true|| does not exceed an entropy-based selector or ranks ID above unknown, the central claim fails. Also add a Table-5 ablation row with entropy-based unknown selection in place of the gradient-norm threshold; if performance does not drop, the 'critical role' is not demonstrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 claims that the gradient norm of the loss with respect to learnable prompts is larger for unknown than for shared-class samples, and this is the sole theoretical basis for the detection threshold. The derivation is not valid. Eq. (6) sets ∂f/∂v to a block-diagonal of z^T, arguing cosine similarity is an inner product. In CLIP, f_i = z^T v_i with normalized v_i; the derivative of f_i with respect to the underlying text-encoder output includes the projection term (z - f_i v_i)/||v_i||, which Eq. (6) omits. For an in-distribution sample with f_i ≈ 0.99, this term has norm sqrt(1 - f_i^2) ≈ 0.14, far from ||z|| = 1, so the missing term changes the gradient magnitude and can reverse the claimed ordering. Second, even if Eq. (6) were correct, Eq. (7) gives g = N M with M = (p - (1/K)1) block-scaled by z. The norm of M is ||p - (1/K)1||, which is larger for a peaked ID distribution (≈0.98 for p_i = 0.99) than for a diffuse unknown distribution (≈0.49 for three classes at 0.3). The paper's assertion that N amplifies the latter more than the former is unsubstantiated; no property of the text-encoder Jacobian N = ∂v/∂w^T is derived or bounded. The experimental section does not repair this: Table 5 omits the full method and does not isolate the gradient-norm selector, and Tables 1-4 contain rows where Ours is worse than both baselines (e.g., Table 3 Rw→Pr: Acc10 78.27 vs CoOp 80.79, FPR95 42.07 vs 40.94, AUROC 91.96 vs 92.29). The central claim is therefore unsupported by the paper's mathematics and not convincingly rescued by its experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an open-set domain adaptation (OSDA) method based on CLIP prompt learning. The two main components are domain-conditioned textual prompt tuning and a gradient-norm-based detector for unknown target-domain samples. The central claim, stated in Section 3.1, is that the L2 norm of the loss gradient with respect to the learnable prompt parameters is larger for unknown-class samples than for shared-class samples, so a source-calibrated threshold on this norm can separate known and unknown target samples. The method then applies cross-entropy loss to the pseudo-labeled shared class and KL divergence regularization to the detected unknown class. Experiments on Office-Home compare against zero-shot CLIP and CoOp, and an ablation study is reported. The paper also includes a derivation of the gradient formula in Section 3.1.","tokens_in":9298,"tokens_out":4547,"duration_ms":42804,"significance":"If the gradient-norm ordering claim were valid, the paper would introduce a conceptually interesting and computationally simple unknown-detection criterion for CLIP-based OSDA, avoiding the need for a separately trained unknown detector. The paper also clearly states its assumptions and provides a detailed gradient-flow diagram. However, the central derivation is mathematically incomplete, the qualitative argument for the ordering is contradicted by the norm of the stated vector, and the experimental results do not consistently support the claimed advantage. The proposed method is therefore not established as a reliable contribution to OSDA.","major_comments":[{"comment":"The derivation of the gradient norm ordering is not valid because Eq. (6) treats CLIP's cosine similarity as a plain inner product. In CLIP, the similarity is f_i = z^T v_i with both z and v_i normalized. The derivative of f_i with respect to v_i includes the projection term (z - f_i v_i)/||v_i||, which is omitted from the block-diagonal form in Eq. (6). This missing term changes the magnitude of the gradient and can reverse the claimed ordering between known and unknown samples, so Eq. (7) does not follow from the premises.","section":"Section 3.1, Eq. (6)"},{"comment":"Even if Eq. (6) were accepted, the norm argument leading to the conclusion that unknown samples have larger gradient norms is unsupported. The vector M in Eq. (7) has norm proportional to ||p - (1/K)1||. For a peaked in-distribution probability, say p_i = 0.99 and K = 65, this norm is approximately 0.98, whereas for a diffuse unknown distribution with three entries of 0.3 the norm is approximately 0.5. The paper's claim that the Jacobian N amplifies the diffuse case more than the peaked case is not backed by any property of N, whose norm is left uncharacterized. Thus the central theoretical claim of Section 3.1 is not established.","section":"Section 3.1, Eq. (7)"},{"comment":"The abstract and introduction state that the method 'consistently outperforms' CLIP and CoOp, but several rows in the results tables contradict this. For example, in Table 3 (Rw to Pr) Ours is worse than CoOp on all three metrics (Acc10 78.27 vs 80.79, FPR95 42.07 vs 40.94, AUROC 91.96 vs 92.29), and in Table 4 (Cl to Ar) Ours has lower Acc10 than CLIP (62.54 vs 63.19). Additionally, the experiments compare only against CLIP and CoOp, with no comparison to existing OSDA methods, so the claim of superiority in the open-set domain adaptation setting is not demonstrated.","section":"Tables 1-4"},{"comment":"The ablation study does not isolate the gradient-norm selector, which is the key novelty of the method. Table 5 reports rows for CLIP, '+CE', and '+KL' separately, but no row corresponds to the full method (CE + KL with gradient-based separation) or to a control where the gradient-norm threshold is replaced by another detection score. The text claims that 'ablation studies confirm the gradient norm's critical role,' but the presented experiments cannot support that claim.","section":"Table 5 and Section 3.2"}],"minor_comments":[{"comment":"The manuscript contains numerous typographical and grammatical errors, including 'Learnabl e', 'cosin e', 'pseudo label s', and 'the results of experiment is opposite of GradNorm'. A careful proofreading pass is needed.","section":"General"},{"comment":"Eq. (8) is incomplete and inconsistent with the hyperparameter description. The term 𝛼𝒑𝑦̂ is not defined, and the variable γ, which is said to be set to 0.001 in Section 4.2, does not appear in Eq. (8). The loss formulation should be stated explicitly with all terms and variables defined.","section":"Section 3.2, Eq. (8)"},{"comment":"The statement that CLIP lacks a final fully connected layer is cited to reference [16], which is a paper about filtering random noise in language model pre-training data. This citation does not support the statement and appears to be a referencing error.","section":"Section 3.1, citation [16]"},{"comment":"The reference list has inconsistencies and duplicates: references [19] and [20] are both the same Busto & Gall (2017) paper, and reference [25] duplicates the content of [15]. The numbering in the text also seems misaligned in places, for example reference [16] as noted above.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript's central theoretical derivation is flawed, the experiments contradict the headline claim, and the ablation does not isolate the proposed mechanism. In my view, these are load-bearing issues that cannot be fixed by local revisions within the current scope. The paper also does not compare with any existing OSDA method and appears to have multiple citation errors, which together suggest the manuscript is not yet ready for publication in this journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper's core observation—that in CoOp-style prompt tuning, the L2 norm of the loss gradient with respect to the prompt parameters is higher for unknown-class samples than for shared-class samples—is genuinely new and worth a footnote. But the derivation that supposedly proves it does not hold up, and the experiments appear to undercut the method they claim to validate.\n\nThe new part: applying GradNorm-style gradient analysis to CLIP prompt parameters is a legitimate application I have not seen before. The sign inversion (unknown > known) is curious, and the chain-rule setup in Eqs. (3)-(7) is an honest attempt to understand it.\n\nThe soft spots, in order. Eq. (6) treats CLIP's cosine similarity as a plain inner product z^T v. CLIP's similarity is L2-normalized; the derivative of f_i with respect to the text-encoder output includes the projection term (z - f_i v_i)/||v_i||. That term is not small—for an ID sample with f_i ≈ 0.99, its norm is about 0.14, not ||z|| = 1. So the derivation of Eq. (7) fails at the first step. Second, even if Eq. (6) were correct, the norm of (p - 1/K) appearing in Eq. (7) is larger for a peaked ID distribution than for a diffuse unknown distribution, so the claimed ordering does not follow from the formula; the Jacobian N = ∂v/∂w^T is left uncharacterized. Third, the experiments seem to contradict the method: Section 4.2 says the scoring function used in all tests is the temperature-scaled maximum softmax. If that is true, the reported AUROC and FPR95 numbers do not measure gradient-norm separation at all. Fourth, the baselines are only zero-shot CLIP and CoOp—none of the OSDA methods discussed in the related work appear in the tables—and in several rows (e.g., Table 3, Rw→Pr) the proposed method is worse than CoOp on all three metrics. The ablation table (Table 5) omits the full method, so the gradient selector's contribution is not isolated.\n\nThe mathematical flaw is load-bearing, not a matter of polish. As written, the paper's central claim is unproven. There may be an empirical effect worth chasing, but this paper does not pin it down.\n\nWho is this for? A reader interested in prompt-based OOD detection could skim the observation, but should not rely on the derivation. It is not ready for serious peer review in this form. My recommendation: desk reject, and invite a resubmission only if the authors fix the derivation and evaluate the gradient-norm score directly.","headline":"Genuinely new gradient-norm observation for CLIP prompt tuning, but the derivation is invalid as written and the experiments don't seem to test the claimed mechanism; not ready for serious review.","tokens_in":9811,"tokens_out":3921,"would_cite":false,"duration_ms":36593,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims unknown target-domain samples in open-set domain adaptation can be detected from the L2 norm of CLIP prompt gradients.","keywords":["open set domain adaptation","CLIP","gradient norm","prompt learning","unknown sample detection","vision-language models","pseudo-labeling","Office-Home"],"falsifier":"Compute the exact gradient of the KL-divergence loss with respect to the learnable prompts using CLIP's normalized cosine similarity, including the Jacobian of $z/\\|z\\|$ and $v/\\|v\\|$, and measure the L2 norms for held-out known and unknown target images; if unknown samples do not consistently yield larger gradient norms than shared-class samples across the source-target splits of Office-Home, the central detection rule is falsified.","tokens_in":8615,"feed_emoji":"🔍","tokens_out":7160,"duration_ms":66551,"temperature":0.7,"pith_summary":"This paper proposes a method for open-set domain adaptation with CLIP, where the target domain contains classes the source never saw. Its central claim is that unknown samples can be separated from shared-class samples by the L2 norm of the loss gradient with respect to the learnable text prompts: unknowns produce larger gradient norms. The paper derives this ordering from a chain-rule identity, explains why it reverses the classic GradNorm ordering, and uses the norm as a threshold-based unknown detector that requires no separate detector network or unknown labels. The framework also includes domain-conditioned prompt tuning and a split pseudo-label loss, with experiments on Office-Home reporting consistent improvement over zero-shot CLIP and CoOp.","feed_headline":"Gradient size reveals unknown classes in domain-shifted images","feed_subtitle":"A CLIP prompt-tuning method separates novel target classes from shared ones using L2 gradient norms, outperforming both baselines on…","key_machinery":"The central object is the L2 norm of the gradient of a KL-divergence loss between the uniform distribution and the model's softmax output, taken with respect to CLIP's learnable prompt embeddings. The paper's identity for this gradient is $\\frac{\\partial l}{\\partial w} = \\frac{\\partial v}{\\partial w}^\\top \\mathrm{diag}(z)\\left(p - \\frac{1}{K}\\mathbf{1}\\right)$, in which $z$ is the image feature, $p$ is the softmax of the class similarities, and $K$ is the number of classes. The identity turns a softmax-confidence statement into a geometric statement: a peaked $p$ leaves few nonzero entries in the gradient and shrinks its norm, while a flatter $p$ spreads nonzero entries across prompt columns and enlarges the norm. This is the mechanism that makes unknown detection threshold-based and free of unknown-class supervision.","core_discovery":"The discovery is that the gradient norm flips direction relative to GradNorm once prompts are learnable. For CLIP with soft prompts, the loss-gradient with respect to prompt weights contains the factor $\\partial v/\\partial w^\\top$ and a block-diagonal image-feature matrix acting on $p - \\frac{1}{K}\\mathbf{1}$, where $p$ is the softmax of CLIP's cosine similarities. When target images are drawn from unknown classes, temperature-scaled softmax probabilities are flatter, so more entries of $p - \\frac{1}{K}\\mathbf{1}$ are non-negligible and the L2 gradient norm becomes larger. Hence the paper's rule: large gradient norm means unknown. A threshold calibrated so that 90% of source samples are called known transfers the separation to the target domain, and the known/unknown division is then used to apply cross-entropy to shared-class pseudo-labels and KL divergence toward uniform for unknowns.","pith_inferences":["A natural extension is to use the prompt-gradient norm as a prompt- or test-time adaptation signal beyond open-set domain adaptation, for example in continual learning or source-free adaptation; the paper does not test these settings.","Because the separation is driven by softmax flatness, interventions such as temperature scaling or entropy maximization could widen the known-unknown gradient gap; this is a testable prediction that the paper leaves implicit.","The reversal of GradNorm's ordering suggests that the direction of gradient-based OOD scores is architecture- and parameterization-dependent, so a same-score pipeline may need a per-module sign check when moved to other prompt-tuned encoders."],"forward_implications":["If the ordering holds, unknown target classes can be flagged from a scalar gradient norm calibrated on the source domain alone, eliminating the need for a separately trained unknown detector and for labeled unknown data.","The same gradient-norm score can be reused across the 12 Office-Home source-target tasks, since thresholds set at 90% source recall transfer to the target domain without additional tuning.","Separating target samples before alignment lets the objective apply cross-entropy only to confident shared-class pseudo-labels and push uncertain samples toward a uniform distribution, reducing error accumulation in later adaptation steps.","Because CoOp's quality degrades under domain shift, conditioning prompts on a domain-discrepancy measure becomes a necessary ingredient for prompt-based open-set domain adaptation.","The reported gains over zero-shot CLIP and CoOp on AUROC, FPR95, and CCR@FPR10 indicate that prompt learning can be made robust under open-set domain shift when paired with gradient-aware separation."],"supporting_citations":[{"why":"Supplies the pretrained CLIP image and text encoders whose joint embedding and learnable-prompt gradient form the basis of the method.","marker":"[7]"},{"why":"Defines the CoOp learnable soft-prompt setting that the paper adapts for domain-conditioned prompts and uses as a comparison baseline.","marker":"[8]"},{"why":"Introduces gradient-norm OOD detection (GradNorm), which the paper reverses for prompt-learning CLIP and builds its own metric against.","marker":"[10]"},{"why":"Provides the temperature-scaled maximum softmax scoring used as the test-time scoring function and as a degraded baseline under distribution shift.","marker":"[9]"}],"fun_headline_variants":["CLIP gradient norms flag unknown classes","Gradient size separates novel classes in domain shift","Prompt gradients reveal open-set unknowns","Gradient norm predicts novel classes in target domains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation in Section 3.1 treats CLIP's cosine similarity as a plain inner product $z^\\top v$, but CLIP actually uses L2-normalized features whose gradient includes projection and scaling terms that Eq. (6) omits; if those terms are included, the paper's Eq. (7) does not follow, and the claimed ordering of gradient norms is not established by the paper's own mathematics.","fun_headline_variants_meta":{"raw":{"variants":["CLIP gradient norms flag unknown classes","Gradient size separates novel classes in domain shift","Prompt gradients reveal open-set unknowns","Gradient norm predicts novel classes in target domains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1811,"prompt_tokens":900,"completion_tokens":911,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":857}},"tokens_in":516,"tokens_out":911,"duration_ms":7438,"temperature":1.0,"reasoning_tokens":857,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:56:00.322273+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the exact gradient of the KL-divergence loss with respect to the learnable prompts using CLIP's normalized cosine similarity, including the Jacobian of $z/\\|z\\|$ and $v/\\|v\\|$, and measure the L2 norms for held-out known and unknown target images; if unknown samples do not consistently yield larger gradient norms than shared-class samples across the source-target splits of Office-Home, the central detection rule is falsified.","supporting_citations":[{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.,","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained CLIP image and text encoders whose joint embedding and learnable-prompt gradient form the basis of the method."},{"cited_title":"C., & Liu, Z","cited_arxiv_id":null,"evidence_quote":"Defines the CoOp learnable soft-prompt setting that the paper adapts for domain-conditioned prompts and uses as a comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces gradient-norm OOD detection (GradNorm), which the paper reverses for prompt-learning CLIP and builds its own metric against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the temperature-scaled maximum softmax scoring used as the test-time scoring function and as a degraded baseline under distribution shift."}],"review_version":1}