Pith. sign in

REVIEW 5 major objections 4 minor 35 references

The paper claims that a single energy-change score, ΔEnergy, can simultaneously serve as a state-of-the-art OOD detector and, when optimized, improve OOD generalization under covariate shifts, and it proves this in a unified few-shot prompt

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:09 UTC pith:DYL2KMPD

load-bearing objection The empirical ΔEnergy method is plausible and the experiments are extensive, but the theoretical claims do not hold: Theorem 3.5's proof is incorrect and the other theorems do not apply in the experimental regime. the 5 major comments →

arxiv 2510.11296 v3 pith:DYL2KMPD submitted 2025-10-13 cs.CV cs.LG

Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization

Lin Zhu , Yifeng Yang , Xinbing Wang , Qinying Gu , Nanyang Ye This is my paper
classification cs.CV cs.LG
keywords out-of-distribution detectionOOD generalizationvision-language modelsenergy-based scoreprompt tuningCLIPcovariate shiftsemantic shift
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper attempts to show that one quantity—the change in a vision-language model's energy score when the top cosine similarities are reset to zero—can serve both as a state-of-the-art out-of-distribution (OOD) detector and, when optimized as a training loss, as a regularizer that improves generalization to covariate-shifted domains. The authors prove that this 'ΔEnergy' score separates in-distribution from open-set OOD samples more strongly than the maximum softmax probability (MCM) baseline, and they introduce a bound-maximization objective (EBM) claimed to enlarge ΔEnergy for closed-set classes while aligning Hessians across domains, which they tie to OOD generalization. If correct, a single few-shot prompt-tuning procedure would improve both semantic-shift detection and covariate-shift accuracy simultaneously, without extra prompts or negative labels. The empirical results show large AUROC gains over prior methods on ImageNet-based benchmarks.

Core claim

The central claim is that re-aligning vision-language modalities—specifically, zeroing out the top-c cosine similarities between an image's feature and the class text features—produces an energy change that is systematically larger for in-distribution images than for open-set OOD images, and that this gap exceeds the gap produced by the standard MCM score. The paper further claims that minimizing an energy-based loss on masked image features (EBM) increases a lower bound on ΔEnergy for closed-set classes and, at the local optimum, makes the Hessian of the classification loss consistent between the original and masked domains; by assuming the masked domain approximates real covariate-shifted

What carries the argument

The central object is ΔEnergy, defined as the difference between the energy score computed after resetting the top-c cosine similarities to zero and the original energy score. The masking operation that retains only the top p% of the element-wise product of image and top-1 text features creates a 'masked domain' used by the EBM loss; the proof that EBM yields domain-consistent Hessians rests on equating the gradient of the energy difference to a difference of Hessians of the classification loss.

Load-bearing premise

The OOD-generalization guarantee requires that the synthetic masked feature domain, created by zeroing top-product elements, faithfully approximates real covariate-shifted target domains; nothing in the proof controls the distance between masked-source and target-domain Hessians.

What would settle it

A direct test: train with EBM on a source domain, evaluate Hessian alignment on a held-out target domain where the shift is caused by a different mechanism (e.g., blur or color jitter rather than style transfer). If the bound's epsilon_1 term is large in practice, the OOD generalization gain should disappear or reverse; measuring the actual generalization gap against the predicted bound would settle the claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • ΔEnergy provides a zero-shot OOD score that provably has a lower false-positive rate than MCM under a condition on the maximum similarity.
  • The EBM loss increases the lower bound of ΔEnergy for closed-set classes, enhancing detection of semantic-shifted OOD.
  • The EBM loss produces domain-consistent Hessians, bounding the OOD generalization gap under covariate shift.
  • A single prompt-tuning procedure with EBM improves both OOD accuracy and OOD detection AUROC on ImageNet and PACS/VLCS benchmarks, outperforming prior methods.
  • No additional negative labels or extra prompts are needed; computation cost is comparable to CoOp.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The masking heuristic (zeroing elements where the product with the top text feature is positive) may be interpreted as forcing the model to rely on background or non-discriminative features; if so, the EBM loss is a form of feature decorrelation that could be extended to other modalities or to unsupervised domain adaptation.
  • The proof's bridge from masked domain to target domain is the load-bearing step; one testable extension is to measure whether the Hessian alignment actually transfers to unseen domains beyond the synthetic mask, e.g., by evaluating on datasets with different shift types not in the training set.
  • The ΔEnergy score could be combined with other OOD scores (e.g., negative-label based) in a straightforward additive way; the paper already hints at this with CSP+ΔEnergy.
  • Since ΔEnergy relies on the top-c similarity being lower for OOD, its advantage may be largest in hard OOD settings where ID and OOD are close; on easy benchmarks the gain is smaller, consistent with the paper's own observation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes ΔEnergy, a zero-shot OOD score for CLIP that measures the change in the energy score when the top-c cosine similarities are set to zero, and EBM, a prompt-tuning loss that minimizes E2−E0 where E2 is the energy computed from masked image features. The authors claim that ΔEnergy provably outperforms MCM for OOD detection (Theorems 3.2–3.3), that EBM maximizes a lower bound of ΔEnergy (Theorem 3.4), and that EBM leads to domain-consistent Hessians that bound the OOD generalization gap (Theorem 3.5 and Proposition 3.6). Empirically, the paper reports consistent improvements over a broad set of baselines on ImageNet-style OOD detection, hard OOD detection, conventional OOD benchmarks, and few-shot OOD generalization on PACS/VLCS.

Significance. If the empirical results are taken at face value, the paper makes a useful practical contribution: a simple, nearly free post-hoc score that improves semantic-shift detection, and a computationally efficient prompt-tuning loss that appears to improve both OOD detection and OOD generalization. The experimental scope is broad, the reported gains are consistent, and the method adds little overhead. However, the advertised theoretical guarantees are not established. The proof of Theorem 3.5 drops a nonzero second-derivative term, Proposition 3.6 relies on an unvalidated proxy assumption connecting masked source features to natural target domains, and Theorem 3.3's condition is not satisfied at the τ=0.01 used in all experiments. Because the central advertised contribution is the claim that EBM is 'theoretically proven' to improve both detection and generalization, these issues are load-bearing rather than cosmetic.

major comments (5)
  1. [Appendix E, Eq. (32)] The proof of Theorem 3.5 does not establish the claimed Hessian consistency. From Eq. (27), LΔE = a0 − a1, where a0 and a1 are the unmasked and masked log-sum-exp averages. Stationarity of LΔE only gives ∇a0 = ∇a1. Using Eq. (31), the difference of Hessians is ĤS − ĤS′ = (∇²a0 − ∇²a1) − (1/N)Σ(∇²S(i) − ∇²S_m(i)). Eq. (32) simply drops the ∇²a0 − ∇²a1 term. Stationarity of first derivatives does not make second derivatives equal. Thus Theorem 3.5 is unproved, and the chain from EBM to domain-consistent Hessians to the OOD generalization bound in Proposition 3.6 has no valid support.
  2. [Appendix F, Eq. (35)] The OOD generalization bound is not established. The proof assumes ||zI(xi) − z~I(xi)|| ≤ ε1 between source and target image features, but then asserts |θᵀ(ĤT − ĤS)θ| ≤ O(ε1) 'since we optimize EBM on masked features'. The EBM loss is defined on a synthetic masked domain S′ (masking zI⊙h1), and nothing in the derivation controls the Hessian distance between S′ and a natural covariate-shifted target domain T. The masked-feature construction does not by itself imply closeness to ImageNet-R/Sketch/V2 or other target domains. Proposition 3.6 therefore needs the synthetic-domain proxy as an explicit, separately justified assumption; as written, the OOD generalization guarantee is unsupported.
  3. [Theorem 3.3 and §4.1] The FPR guarantee is not applicable to the experimental setting. Theorem 3.3 requires ŝ_y1(x′) ≤ τ ln 2. In §4.1 and §4.2 the temperature is set to τ = 0.01, so τ ln 2 ≈ 0.0069. The scores are cosine similarities of normalized CLIP features, which are typically an order of magnitude larger (often 0.2–0.3) on ImageNet. Thus the condition fails for the very configuration tested in Tables 1–4. The 'provably lower FPR than MCM' claim does not cover the reported results; the authors need either to state explicitly that the theorem applies only to low-temperature regimes satisfying the condition, or to test the method in a regime where the condition is met.
  4. [Appendix B / Theorem B.1] The proof of Theorem 3.2 does not cover the c=2 setting used in all experiments. The proof begins 'Without loss of generality, we set c=1', but the comparison in Theorem B.1 and the amplification property are only worked out for c=1. The c=2 case, which is what Table 1 and the tuning experiments use, is not proven. In addition, the core step assumes Σ_{i≠ŷ1} e^{s_i(xID)/τ} ≈ Σ_{i≠ŷ1} e^{s_i(xOOD)/τ}; this non-maximal-sum equality is asserted to be 'reasonable' without evidence. It is not generally true for semantic-shift OOD samples, whose non-max similarity profile can be systematically different. Without controlling this term, the claim that ΔEnergy amplifies the ID/OOD gap does not follow from the max-similarity gap alone.
  5. [Tables 3–4] The empirical comparison for tuning-based OOD detection is asymmetric. In Table 3, all baseline methods are evaluated with the MCM (or GL-MCM) score, while the proposed EBM is evaluated with ΔEnergy. Since ΔEnergy already outperforms MCM in the zero-shot setting (Tables 1–2), part of the apparent improvement may be attributable to the change in scoring function rather than to the EBM objective. Table 4 does include an 'EBM + MCM' row, which is a useful control, but the main Setup-I comparison does not report EBM + MCM or baseline + ΔEnergy. For a fair claim that EBM improves detection, the paper should report both scoring functions for all methods.
minor comments (4)
  1. [Table 2] The header 'A VG' appears to be a formatting artifact; the average column should be labeled clearly, e.g., 'Average'.
  2. [Section 3.2, Eq. (6)] The masking proportion p is described as retaining 'the top p-proportion elements in zI⊙h1', but p is searched in [0.4, 0.6]. It would help to state explicitly whether p is the fraction retained or the fraction zeroed, and to add an ablation for p on both detection and generalization, since p directly controls the masked domain S′.
  3. [Appendix D] The proof of Theorem 3.4 uses s_j(x) and s′_j(x) without restating that s′_j is the similarity of the masked feature; a brief definition inside the proof would improve readability.
  4. [General] Several theorems in the main text are stated without their proof conditions, e.g., Theorem 3.5's ε assumption is in the statement but the proof of Proposition 3.6 introduces ε1 with a different definition; the relationship between ε and ε1 should be clarified.

Circularity Check

1 steps flagged

EBM's ΔEnergy detection gain is partly built into the objective; OOD-generalization proof has a non-circular gap.

specific steps
  1. fitted input called prediction [Section 3.2, Eq. (4), Theorem 3.4]
    "Specifically, motivated by further enlarging ∆Energy, we propose to minimize the following term: L∆E = 1/N Σ [E2(xi) − E0(xi)] ... Theorem 3.4 implies that if the change in the VLM's predictions after re-alignment is not too small and satisfies the condition as shown in Equation 7, minimizing L∆E can increase the lower bound of ∆Energy for closed-set classes."

    The evaluation score is ∆Energy = E1 − E0 (Eqs. 2–3), while the EBM training loss is L∆E = E2 − E0, with E2 obtained by the same masked re-alignment mechanism. Theorem 3.4 proves ∆Energy ≥ −L∆E under a condition, so minimizing L∆E directly maximizes a lower bound of the very score used to measure OOD detection. The reported improvement in ∆Energy-based AUROC/FPR95 after EBM training is therefore substantially baked into the objective, not an independent prediction. The circularity is partial: E2 ≠ E1, the bound is conditional, and EBM also improves MCM-based detection (Table 4) and OOD classification accuracy, which are independent checks.

full rationale

The zero-shot ∆Energy score is a well-defined transform of CLIP similarities, and Theorems 3.2–3.3 are conditional comparisons against MCM under the same max-similarity separation assumptions used by MCM; these are not circular. The main circularity risk is in EBM: Eq. (4) defines L∆E = E2 − E0, and Theorem 3.4 proves a lower bound ∆Energy ≥ −L∆E. Since the reported detection metric is exactly ∆Energy (E1 − E0), the improvement in ∆Energy-based AUROC after minimizing L∆E is substantially built into the objective. I therefore score this as partial (5): it is a bound, not an identity, E1 ≠ E2, the theorem is conditional on Eq. (7), and two independent results—EBM also improves MCM-based detection (Table 4) and OOD classification accuracy (Tables 3–4)—show the method has content beyond the score itself. The OOD-generalization claim (Theorem 3.5 / Proposition 3.6) is not circular but is under-supported: Appendix E's Eq. (32) drops the ∇²L∆E term when passing from stationarity of ∇L∆E to equality of Hessians, and Appendix F's O(ε1) link assumes the masked-source domain approximates the target domain; these are correctness risks, not reductions to inputs. No load-bearing self-citation or imported uniqueness theorem was found.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

The central theoretical structure rests on several unverified or unstated assumptions: equal non-max similarity sums, ID max-similarity dominance, a temperature condition not met at τ=0.01, and the key identification of a synthetic masked domain with real covariate shifts. The EBM loss weight and score hyperparameters are tuned on validation/test-like data.

free parameters (4)
  • temperature τ = 0.01
    Scaling temperature for energy and MCM scores; chosen by hand, and the value violates the condition required by Theorem 3.3 for typical CLIP similarities.
  • top-c similarities c = 2
    Number of max similarities reset to zero; selected via ablation on the same ImageNet OOD benchmark reported in Table 9.
  • masking proportion p = 0.4–0.6
    Retains top p-proportion of elements in the product feature; tuned per setup and used in EBM.
  • EBM loss weight λ0 = 1 or 2 depending on setup
    Searched in [0.1,0.5,1.0] for Setup-I and set to 2 for Setup-II; selected on validation.
axioms (6)
  • domain assumption Non-max similarity sums are approximately equal for ID and OOD samples
    Used in Appendix B / Theorem B.1 to make ΔEnergy monotone in the max similarity; not verified on the benchmark data.
  • domain assumption Maximum cosine similarity of in-distribution data exceeds that of OOD data
    Theorems 3.2/B.1 assume this; it is the property the score is meant to test, so the proof does not establish discriminability from scratch.
  • domain assumption Top-1 similarity satisfies s1 ≤ τ ln 2
    Required for FPR theorem (Thm 3.3); at the experimental τ=0.01 this bound (≈0.0069) is far below typical CLIP max similarities.
  • ad hoc to paper Masked feature domain S' is a proxy for covariate-shifted target domain
    Theorem 3.5 aligns Hessians between source and masked features; Prop 3.6 transfers to target using only ||zI−z~I||≤ε1, without linking mask to shift.
  • standard math Hessian consistency is a valid predictor of OOD generalization
    Imported from Rame et al. (2022) / Hemati et al. (2023); load-bearing for Prop 3.6.
  • standard math Smoothness: small feature perturbation bounds Hessian differences via O(ε)
    Taylor expansion in Prop 3.6; sketched, not rigorously quantified.
invented entities (1)
  • synthetic masked domain S′ no independent evidence
    purpose: Proxy for covariate-shifted target domain in the Hessian-consistency proof
    No evidence that zeroing top-p product elements reproduces natural domain shifts; the mask is chosen for attention preservation (Appendix G, Fig. 3), not for matching any real shift.

pith-pipeline@v1.3.0-alltime-deepseek · 29554 in / 19651 out tokens · 132151 ms · 2026-08-04T10:09:06.221194+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of $\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization." pith.science (2026). https://pith.science/paper/DYL2KMPD

@misc{pith2026251011296,
  author       = {Pith},
  title        = {Pith review of: $\Delta \mathrmEnergy$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DYL2KMPD}},
  note         = {Machine review of arXiv:2510.11296}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent approaches for vision-language models (VLMs) have shown remarkable success in achieving fast downstream adaptation. When applied to real-world downstream tasks, VLMs inevitably encounter both the in-distribution (ID) data and out-of-distribution (OOD) data. The OOD datasets often include both covariate shifts (e.g., known classes with changes in image styles) and semantic shifts (e.g., test-time unseen classes). This highlights the importance of improving VLMs' generalization ability to covariate-shifted OOD data, while effectively detecting open-set semantic-shifted OOD classes. In this paper, inspired by the substantial energy change observed in closed-set data when re-aligning vision-language modalities (specifically by directly reducing the maximum cosine similarity to a low value), we introduce a novel OOD score, named {\Delta}Energy. {\Delta}Energy significantly outperforms the vanilla energy-based OOD score and provides a more reliable approach for OOD detection. Furthermore, {\Delta}Energy can simultaneously improve OOD generalization under covariate shifts, which is achieved by lower-bound maximization for {\Delta}Energy (termed EBM). EBM is theoretically proven to not only enhance OOD detection but also yields a domain-consistent Hessian, which serves as a strong indicator for OOD generalization. Based on this finding, we developed a unified fine-tuning framework that allows for improving VLMs' robustness in both OOD generalization and OOD detection. Extensive experiments on challenging OOD detection and generalization benchmarks demonstrate the superiority of our method, outperforming recent approaches by 10% to 25% in AUROC.

Figures

Figures reproduced from arXiv: 2510.11296 by Lin Zhu, Nanyang Ye, Qinying Gu, Xinbing Wang, Yifeng Yang.

Figure 1
Figure 1. Figure 1: (A) Illustration of ∆Energy for OOD detection. Significant differences in ∆Energy are observed between closed-set data and open-set OOD data when the maximum cosine similarity is cropped to zero. (B) Illustration of the ∆Energy for OOD generalization. We introduce the EBM method to achieve domain-consistent Hessians, which simultaneously triggers bound optimization for ∆Energy. More details are in Section … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed method. Based on the prompt-tuning approach, we freeze both [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The significant prediction difference between closed-set data and open-set OOD data when [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 22 linked inside Pith

  1. [1]

    In the table, OOD detection is measured by AUROC and FPR95 over 6 hard OOD detection datasets

    Since we optimize the EBM loss based on the unmasked image features and masked image features, we have the following approximation: θ⊤[ bHT (θ∗)− bHS (θ∗)](θ)θ ≤O(ε 1)(35) G More Experiment Results More experiment detailsWe present experiment details for the baseline models as follows: 23 Table 7:Hard OOD detection Results #2.Comparison with state-of-the-...

    arXiv 2024

  2. [2]

    Therefore, unless otherwise specified, we set c= 2 in ∆Energyfor all experiments

    ∆Energy achieves the overall best performance on both AUROC and FPR95 when c= 2 . Therefore, unless otherwise specified, we set c= 2 in ∆Energyfor all experiments. OOD generalization performance under concept shiftWe conduct experiments on the concept- shifted ImageNet-Superclass dataset (Xiao et al., 2024; Santurkar et al., 2020), where each im- age is a...

    2024

  3. [3]

    Conjugated semantic pool improves ood detection with pre-trained vision-language models.arXiv preprint arXiv:2410.08611,

    Mengyuan Chen, Junyu Gao, and Changsheng Xu. Conjugated semantic pool improves ood detection with pre-trained vision-language models.arXiv preprint arXiv:2410.08611,

    Pith/arXiv arXiv

  4. [6]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks.arXiv preprint arXiv:1610.02136,

    Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks.arXiv preprint arXiv:1610.02136,

    Pith/arXiv arXiv

  5. [8]

    A detailed comparison of computation cost is provided in Table

    It re-aligns vision-language representations in the latent space without introducing extra prompts, as required by vanilla CoOp (Zhou et al., 2021), thereby it is more efficient over methods such as LoCoOp (Miyai et al., 2024b) and GalLoP (Lafon et al., 2024). A detailed comparison of computation cost is provided in Table

    2021

  6. [10]

    Negative label guided ood detection with pretrained vision-language models.arXiv preprint arXiv:2403.20078,

    Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained vision-language models.arXiv preprint arXiv:2403.20078,

    Pith/arXiv arXiv

  7. [11]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023a. 11 Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. S...

    Pith/arXiv arXiv 2012

  8. [12]

    Recent advances in out-of- distribution detection with clip-like models: A survey.arXiv preprint arXiv:2505.02448,

    Chaohua Li, Enhao Zhang, Chuanxing Geng, and Songcan Chen. Recent advances in out-of- distribution detection with clip-like models: A survey.arXiv preprint arXiv:2505.02448,

    Pith/arXiv arXiv

  9. [14]

    Delving into out-of- distribution detection with vision-language representations.Advances in neural information processing systems, 35:35087–35102, 2022a

    Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations.Advances in neural information processing systems, 35:35087–35102, 2022a. Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. InInternational Conference on Mac...

    Pith/arXiv arXiv

  10. [15]

    Generalized out-of-distribution detection and beyond in vision language model era: A survey.arXiv preprint arXiv:2407.21794, 2024a

    12 Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Irie, Shafiq Joty, Yixuan Li, Hai Li, et al. Generalized out-of-distribution detection and beyond in vision language model era: A survey.arXiv preprint arXiv:2407.21794, 2024a. Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot out-of-distribution d...

    Pith/arXiv arXiv

  11. [17]

    A less biased evaluation of out-of-distribution sample detectors.arXiv preprint arXiv:1809.04729,

    Alireza Shafaei, Mark Schmidt, and James J Little. A less biased evaluation of out-of-distribution sample detectors.arXiv preprint arXiv:1809.04729,

    Pith/arXiv arXiv

  12. [18]

    Clipn for zero-shot ood detection: Teaching clip to say no

    Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1802–1812, 2023a. Haoliang Wang, Chen Zhao, Yunhui Guo, Kai Jiang, and Feng Chen. Towards effective seman- tic ood detection in unseen domains: A domain generalizat...

    Pith/arXiv arXiv

  13. [19]

    Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms.CoRR, abs/2106.03721,

    Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms.CoRR, abs/2106.03721,

    Pith/arXiv arXiv

  14. [20]

    Hybrid models for open set recognition

    Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 102–117. Springer,

    2020

  15. [22]

    Learning to prompt for vision- language models.arXiv preprint arXiv:2109.01134,

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models.arXiv preprint arXiv:2109.01134,

    Pith/arXiv arXiv

  16. [23]

    Decoop: Robust prompt tuning with out-of-distribution detection.arXiv preprint arXiv:2406.00345,

    Zhi Zhou, Ming Yang, Jiang-Xin Shi, Lan-Zhe Guo, and Yu-Feng Li. Decoop: Robust prompt tuning with out-of-distribution detection.arXiv preprint arXiv:2406.00345,

    Pith/arXiv arXiv

  17. [24]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023a

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023a. Lin Zhu, Xinbing Wang, Chenghu Zhou, and Nanyang Ye. Bayesian cross-modal alignment learning for few-shot out-of-distribution generalization. InProceedings of the AAAI Conference on Artificial ...

    Pith/arXiv arXiv

  18. [25]

    15 A Related Works Robust fine-tuning methods for VLMFor training efficiency, there have been many lightweight CLIP-based fine-tuning methods to enhance generalization performance via prompt tuning (Singha et al., 2023; Huang et al., 2022; Khattak et al., 2023a; Wang et al., 2023c; Wasim et al., 2023; Goswami et al., 2024; Huang et al.,

    2023

  19. [26]

    Prompt tuning methods aim to get better vision-language alignment via only fine-tuning the input prompts

    or adapter tuning (Gondal et al., 2024; Zhang et al., 2023b; Song et al., 2023). Prompt tuning methods aim to get better vision-language alignment via only fine-tuning the input prompts. For example, with only few-shot samples for learning, CoOp (Zhou et al.,

    2024

  20. [27]

    Adapter-tuning is another popular lightweight fine-tuning method, like CLIP-Adapter (Gao et al., 2023), Tip-Adapter (Zhang et al., 2021a)

    is proposed to achieve generalization on unseen classes via conditional prompt learning. Adapter-tuning is another popular lightweight fine-tuning method, like CLIP-Adapter (Gao et al., 2023), Tip-Adapter (Zhang et al., 2021a). Both of them inject a lightweight bottleneck architecture after the image encoder or text encoder and perform residual-style feat...

    2023

  21. [28]

    to handle OOD generalization and OOD detection simultaneously, these approaches are typically limited to traditional vision models or have been evaluated with a narrow set of post-hoc functions for OOD Detection. Thus, when fine-tuning VLMs for downstream tasks, the challenge of improving the models’ generalization ability to closed-set OOD data while sim...

    2018

  22. [29]

    and training-time regularization (Narayanaswamy et al., 2023; Bai et al., 2023; Malinin and Gales, 2018; Du et al., 2022b,a; Ming et al., 2022b). The former typically resort to post-hoc functions to recognize open-set without altering the DNN training process, like density estimation (Zhang et al., 2020), uncertainty modeling (Gal and Ghahramani, 2016), a...

    2023

  23. [30]

    1 + es ˆy1 (x′)/τ −e ˜sˆy1 (x′)/τ P i̸=ˆy1 esi(x′)/τ +e ˜sˆy1 (x′)/τ # = log

    have explored the capability of zero-shot or few-shot OOD detection based on VLMs. More details about the CLIP-based OOD detection methods can be seen in these surveys (Miyai et al., 2024a; Li et al., 2025). However, while these studies primarily focus on handling semantic-shifted datasets, our research aims to simultaneously improve both OOD detection fo...

    2025

  24. [32]

    end”, and set the class-specific context (CSC) as “False

    without modification. For the tuning-based methods, based on the code of CoOp (Zhou et al., 2021), we train models with SGD optimizer with a learning rate of 2e−2 . The batch size is set to 32 for all tuning-based experiments. For the specific hyperparameter for each method, we follow the setting of the original paper. For the tuning-based methods for imp...

    2021

  25. [33]

    This difference offers a novel approach to distinguishing between closed-set and open-set classes

    method, we adopt image adapter only with the residual ratio of 0.5 for Setup-I and 24 Figure 3: The significant prediction difference between closed-set data and open-set OOD data when vision-language re-alignment is applied to the zero-shot CLIP model (Radford et al., 2021). This difference offers a novel approach to distinguishing between closed-set and...

    2021

  26. [2014]

    Siren: Shaping representations for detecting out-of-distribution objects.Advances in Neural Information Processing Systems, 35:20434–20449, 2022a

    Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detecting out-of-distribution objects.Advances in Neural Information Processing Systems, 35:20434–20449, 2022a. Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. V os: Learning what you don’t know by virtual outlier synthesis.arXiv preprint arXiv:2202.01197, 2022b. 10...

    Pith/arXiv arXiv

  27. [2016]

    Scaling out-of-distribution detection for real-world settings

    Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132,

    Pith/arXiv arXiv 1911

  28. [2017]

    Learning transferable nega- tive prompts for out-of-distribution detection

    Tianqi Li, Guansong Pang, Xiao Bai, Wenjun Miao, and Jin Zheng. Learning transferable nega- tive prompts for out-of-distribution detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17584–17594, 2024a. Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised p...

    Pith/arXiv arXiv

  29. [2018]

    Learning transferable visual models from natural language supervision.CoRR, abs/2103.00020,

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision.CoRR, abs/2103.00020,

    Pith/arXiv arXiv

  30. [2019]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021a. Dan Hendrycks, Kevin Zhao, ...

    Pith/arXiv arXiv

  31. [2020]

    Openood v1.5: Enhanced benchmark for out-of-distribution detection.arXiv preprint arXiv:2306.09301, 2023a

    Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distribution detection.arXiv preprint arXiv:2306.09301, 2023a. 14 Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongshe...

    Pith/arXiv arXiv

  32. [2021]

    Food-101–mining discriminative compo- nents with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative compo- nents with random forests. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461. Springer,

    2014

  33. [2022]

    Dense outlier detection and open-set recognition based on training with noisy negative images.arXiv preprint arXiv:2101.09193,

    Petra Bevandi´c, Ivan Krešo, Marin Orši´c, and Siniša Šegvi´c. Dense outlier detection and open-set recognition based on training with noisy negative images.arXiv preprint arXiv:2101.09193,

    Pith/arXiv arXiv

  34. [2023]

    In search of lost domain generalization.arXiv preprint arXiv:2007.01434,

    Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization.arXiv preprint arXiv:2007.01434,

    Pith/arXiv arXiv 2007

  35. [2024]

    Unsupervised prompt learning for vision-language models

    Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649,

    Pith/arXiv arXiv