Pith. sign in

REVIEW 5 major objections 6 minor 68 references

Decouple before Align: Visual Disentanglement Enhances Prompt Tuning

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Decoupling images into foreground and background before text alignment redirects CLIP's attention from context to the object, improving few-shot accuracy and generalization with no inference-time cost.

desk verdict Useful, simple regularization for CLIP prompt tuning that deserves review, but the missing code, missing error bars, and a nonsensical background prompt template need to be sorted out before the SOTA claims can be taken at face value. read the letter →

arxiv 2508.00395 v1 pith:WNS5VCE3 submitted 2025-08-01 cs.CV cs.AI

Fei Zhang , Tianfei Zhou , Jiangchao Yao , Ya Zhang , Ivor W. Tsang , Yanfeng Wang This is my paper
classification cs.CVcs.AI
keywords prompttuningCLIPvisualdisentanglementforeground-backgrounddecouplingfew-shotclassificationbase-to-novelgeneralizationtripletlosssegmentationmasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompt tuning adapts CLIP to a new task by learning a few prompt tokens, but the paper argues that standard image-text alignment is information-asymmetric: an image carries rich background context while its paired text names only the object. Aligning the two as if they were symmetric makes the model's attention drift toward context, which shows up as misclassified samples whose attention maps skip the region of interest. The paper's proposal, DAPT (decouple before align), first splits each training image into a foreground part and a background part using either a coarse model-generated attention mask or a fine off-the-shelf segmentation mask, then aligns the foreground with the class text, aligns the background with hand-crafted background class names, and adds a pull-push triplet loss that keeps the original image feature near the foreground and away from the background. The claimed result is that these training-only regularizations sharpen CLIP's attention on the object, improving few-shot accuracy, base-to-novel generalization, and data efficiency across 11 benchmarks, with no extra computation at inference. If true, DAPT is a general, architecture-free regularizer that can be bolted onto existing prompt-tuning frameworks.

What carries the argument

The carrying object is the visual triplet $(I, I_f, I_b)$ produced by a binary semantic mask $M$, together with the symmetric alignment losses built on it. $I_f$ is the masked foreground, $I_b$ is the masked background, and the triplet loss $L_v = \sum_i \max(\|\tilde{Z}_I^i - \tilde{Z}_f^i\|_1 - \|\tilde{Z}_I^i - \tilde{Z}_b^i\|_1 + \alpha, 0)$ is what pulls the original image representation toward the object and away from the context. Foreground-text alignment $L_f$ ties $I_f$ to the class prompts, while background-text alignment $L_b$ ties $I_b$ to a fixed vocabulary of 25 background class names, giving the model a way to keep useful context without letting it dominate. The mask itself can come from a coarse, self-generated attention map or a fine, external segmentation map, and the paper argues the pull-push term makes the method tolerant to mask errors because a poor mask degrades into an erasing-style perturbation rather than a wrong supervisory signal.

What would settle it

Run DAPT on a dataset where the 25 background class names are systematically wrong for the imagery (for example, aerial or medical images with no sky, ground, or wall), measure the agreement of $y_i^b$ with human background labels, and compare DAPT's gain against the same model with $L_b$ removed or with $y_i^b$ replaced by random labels. If the gain persists despite near-chance pseudo-labels, or vanishes only when $L_b$ is removed, the background-alignment term is not doing the causal work attributed to it.

Watch

Extended reading notes

Core claim

The central claim is that the failure of prompt-tuned CLIP on hard samples is caused by biased attention from coarse image-text alignment, and that explicitly decoupling the visual input before alignment restores symmetry between modalities. DAPT separates each image $I$ into foreground $I_f = M \odot I$ and background $I_b = (1-M) \odot I$ using a semantic mask $M$, and then optimizes four losses together: the usual classification loss, a foreground-text alignment $L_f$ that scores the foreground against the class texts, a background-text alignment $L_b$ that scores the background against 25 background class names such as sky and ground, and a visual triplet loss $L_v$ that pulls the prompted feature of the original image toward the foreground feature and pushes it away from the background feature. The authors' claim is that this decouple-before-align recipe directs CLIP's attention toward the region of interest in previously misclassified images, improves both in-domain recognition and novel-class generalization, works with imperfect masks, and costs nothing at inference time because only the original image and class texts are used at test time.

Load-bearing premise

The load-bearing premise is that the background pseudo-labels in Eq. (5), assigned by the same frozen CLIP model matching each background image against 25 hand-picked class names, are accurate enough that training the background-text alignment loss $L_b$ teaches useful context rather than reinforcing wrong context; the paper never measures this pseudo-label accuracy.

Editorial extensions

If this is right

  • Adding $L_v + L_f + L_b$ to an existing prompt-tuning framework (MaPLe, PromptSRC, or PromptKD) raises both base and novel class accuracy, reaching 84.45 harmonic mean when DAPT is combined with PromptKD.
  • In the data-efficient setting, DAPT-G and DAPT-S match MaPLe's full-data performance using only about 50% of the training data, with random sample selection and no coreset methods.
  • Because decoupling happens only during training, inference uses exactly the same inputs as vanilla prompt tuning; the reported overhead is about 0.2 seconds per 40 images for external segmentation masks and about 5.13 seconds per epoch for on-the-fly mask generation.
  • The decouple-before-align loss transfers to single-modal prompting baselines: adding it to CoOp improves few-shot accuracy by up to +2.85 and to VPT by up to +3.67.
  • The method extends to multi-object recognition: on VOC12, DAPT-S improves 16-shot mAP to 96.88 and base-to-novel harmonic mean to 90.73.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the coarse mask variant needs only signals derived from CLIP itself, the same recipe could apply to any frozen vision-language model without external segmentation, making the decoupling a default plug-in for prompt-tuning pipelines rather than a special case.
  • The 25 background class names are fixed across all datasets; a natural extension is to choose background vocabulary per domain (for example, texture words for DTD or land-cover words for EuroSAT), which the paper's own results suggest is where the largest gap remains.
  • The information-asymmetry diagnosis is not limited to classification: image-text retrieval and visual question answering face the same context-overload problem, so a decouple-before-align loss might transfer to those tasks, though the paper does not test them.
  • A direct test of the mechanism would be to measure whether the background-alignment loss's benefit tracks the pseudo-label accuracy of $y_i^b$; if it does, then better background vocabularies or label cleaning should yield further gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper proposes DAPT, a prompt-tuning framework for CLIP that explicitly decouples the input image into foreground and background components before performing vision-language alignment. The foreground is obtained by masking the image with either Grad-CAM (DAPT-G) or SEEM (DAPT-S) masks, and the background is its complement. Three additional losses are introduced: foreground-text alignment (Lf), background-text alignment (Lb), and a visual triplet pull-push term (Lv). The method is evaluated on few-shot classification, base-to-novel generalization, cross-data transfer, and data-efficient learning across 11+ benchmarks, with consistent positive gains over MaPLe and further improvements when plugged into PromptSRC and PromptKD. The central claim is that coarse image-text alignment causes biased attention toward context, and that decoupling before alignment yields a symmetrical modality alignment that improves recognition and generalization.

Significance. If the method holds, DAPT offers a generally applicable, architecture-free regularization for CLIP prompt tuning that requires no inference-time cost and can be plugged into existing frameworks. The paper provides extensive experiments, including ablations that decompose each loss term and mask-quality analyses that suggest robustness to imperfect segmentation. The idea of using visual disentanglement to address modality asymmetries in prompt tuning is intuitive and potentially impactful for the community. However, the significance is tempered by the unverified semantics of the background prompt template, the absence of variance estimates for headline numbers, and an explicitly incomplete ablation note; these issues need to be resolved before the claims can be fully credited.

major comments (5)
  1. [Section 4.3, Eq. (5) and Section 5.1] The background-text alignment uses the prompt template "a clean origami [BACKGROUND NAME]" for classes such as sky, ground, and wall. This phrase is semantically incoherent as a description of a background region. Since the background text features Z_U in Eq. (5) and the pseudo-labels y_i^b are both derived from this template, L_b does not appear to perform the "background-text alignment" described in the paper. The attribution of L_b's +1.60 novel-class improvement (Table 5) to "generalized contextual knowledge" is therefore not supported by the text as written. Please either justify this template as an intentional choice (e.g., with a citation to [39] showing its empirical effectiveness) or provide an ablation using a coherent template such as "a photo of [BACKGROUND NAME]" and report whether the L_b benefit persists. Additionally, please report the accuracy of the background pseudo-labels on the 11 datasets, since the pseudo-labeling step is load-bearing for L_b.
  2. [Section 5.1, Table 1, Figure 5, Tables 2-4] The paper reports that results are the average of three runs but provides no standard deviations or confidence intervals. Several of the headline gains are below 1.5 points (e.g., DAPT-S vs. MaPLe harmonic mean +0.80 in Table 1; many cross-data deltas in Table 2 are between +0.2 and +0.7). Given the known run-to-run variability in few-shot prompt tuning, these differences may not be statistically meaningful. Please report standard deviations for all main tables and figures, and state whether the improvements are consistent across the three runs. This is essential for readers to assess the reliability of the central claims.
  3. [Section 5.6, Table 8] The text after Table 8 contains the incomplete sentence "We will add this analysis." This indicates that the ablation study is not finished at the time of submission. The mask-quality robustness claim (Section 4.3, "Mask Quality") is partly supported by Tables 7 and 9, but the presence of an unfinished note weakens the completeness of the experimental section. Please complete the missing analysis or remove the sentence from the final version.
  4. [Section 5.4.1, Table 1] In Table 1, DAPT-G achieves a harmonic mean of 78.38, which is below MaPLe's 78.55. The sentence "DAPT consistently outperforms others in recognizing both base and novel class images" is therefore not accurate for DAPT-G. The paper should either qualify this claim by distinguishing DAPT-G and DAPT-S or present a statistical comparison before making such a broad statement.
  5. [Section 4.4 and Section 5.4] The paper repeatedly describes DAPT as "architecture-free." While it is true that no architectural modifications are made to CLIP, DAPT-S requires an external segmentation model (SEEM) to generate masks, and the method introduces several additional loss hyper-parameters. Please clarify that "architecture-free" refers to the backbone architecture of CLIP, and acknowledge the external dependency of DAPT-S (as also noted in the text of Section 5.6). This will prevent misleading readers about the deployment requirements.
minor comments (6)
  1. [Section 2.1] Typo: "embebddings" should be "embeddings".
  2. [Section 5.1] Dataset name "SUN39" should be "SUN397."
  3. [Section 5.4.1] Typo: "PropmptKD" should be "PromptKD".
  4. [Throughout] The word "elation" appears in several places (e.g., "performance elation") and is likely intended to mean "elevation" or "improvement." Please correct.
  5. [References, [41]] Reference [41] for ImageNet cites Gao et al., "Large-scale unsupervised semantic segmentation," which does not appear to be the correct source for the ImageNet dataset. Please verify and correct the citation.
  6. [Figure 3] The caption states "The visual triplets are generated via either a coarse or fine-grained mask," but the figure appears to show only one mask path. Consider clarifying the two variants in the figure or caption.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: DAPT is evaluated on held-out labels with mask-free inference; the self-referential Grad-CAM masks and background pseudo-labels are disclosed self-training loops, not equation-level reductions.

full rationale

No equation in the paper reduces to its own input by construction. The four loss terms in Eq. (7) operate on masked visual inputs (If = M ⊙ I and Ib = (1−M) ⊙ I) and prompted text features, and none is defined as the accuracy metric the paper claims to improve. Few-shot, base-to-novel, and data-efficient results are obtained by evaluating the prompted model on held-out labels with mask-free inference: as the paper states, "During the inference stage, we only input the original testing image and the corresponding foreground-class texts for evaluation." The two self-referential elements are real feedback loops but not circular derivations. The on-the-fly Grad-CAM mask in Eq. (3) is generated from the model being trained, and the background pseudo-label in Eq. (5) is generated by the same frozen CLIP whose similarity is then aligned by Lb. However, neither is hidden or presented as an external prediction; both are disclosed auxiliary regularizers. Lb is a self-distillation target rather than an identity: the prompted features eZ_i_b and eZU differ from the frozen features Z_b and Z_U that produced y_i^b, and the background loss is never the reported result. Consequently, the paper does not exhibit self-definition, fitted-input-called-prediction, or imported-uniqueness circularity. The incomplete analysis note in Section 5.6 ("We will add this analysis") and the semantically odd "a clean origami [BACKGROUND NAME]" template are completeness and correctness concerns, not derivation-circularity concerns; they do not raise the circularity score.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method adds no new physical entities. It does rely on several hand-set hyperparameters and two self-supervision signals, Grad-CAM masks and background pseudo-labels, that are generated by the same CLIP model being tuned. The strongest ledger concern is the assumption that those self-generated signals are accurate enough to provide useful supervision, which the paper tests only indirectly.

free parameters (7)
  • Loss weight gamma_v for visual pull-push term = 0.6 for settings I and II; 0.4 for setting III
    Chosen to balance Lv in Eq. (7); the setting III value is adjusted after observing base-to-novel performance (Sections 5.1 and 5.6).
  • Loss weight gamma_f for foreground-text alignment = 0.4 for all settings
    Chosen for Lf in Eq. (7); ablation in Table 5 shows sensitivity but the paper does not tune it per dataset.
  • Loss weight gamma_b for background-text alignment = 0.1 for settings I and II; 0.5 for setting III
    Chosen for Lb in Eq. (7); the paper increases it in setting III to improve novel class generalization, which is a post hoc adjustment.
  • Triplet margin alpha = 5.0
    Margin in Eq. (6); hand-chosen, with no sensitivity study reported.
  • Grad-CAM threshold beta = 0.5
    Threshold for the binary semantic mask in Eq. (3) and Section 4.2; hand-chosen.
  • Number of learnable prompts m = 2
    Number of visual and textual prompt tokens inherited from the MaPLe baseline [11] and treated as fixed.
  • Background class vocabulary = 25 hand-picked classes, listed in Section 5.1
    Hand-crafted class set used as the pseudo-label space for Lb in Eq. (5), following [39]; not validated per dataset.
assumptions (4)
  • domain assumption CLIP's frozen encoders and cosine similarity define a usable classifier for downstream tasks.
    Section 3, Eq. (1), relies on the standard CLIP zero-shot assumption that image-text cosine similarity ranks classes correctly.
  • domain assumption The semantic mask M separates the ROI foreground from the context background well enough that If and Ib carry useful decoupled semantics.
    Section 4.2 introduces the masks; they are tested indirectly via erasing and mask-source ablations, but not independently guaranteed per dataset.
  • domain assumption Background pseudo-labels y_i^b = argmax_j sim(Zb, Z_U^j) are sufficiently accurate to supervise Lb.
    Section 4.3, Eq. (5) uses these labels; no pseudo-label accuracy analysis is supplied.
  • ad hoc to paper A margin-based triplet loss with L1 norm and margin alpha is an effective regularizer for prompted CLIP features.
    Section 4.3, Eq. (6) assumes the L1 geometry of CLIP features supports the intended pull-push semantics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decouple before Align: Visual Disentanglement Enhances Prompt Tuning." pith.science (2026). https://pith.science/paper/WNS5VCE3

@misc{pith2026250800395,
  author       = {Pith},
  title        = {Pith review of: Decouple before Align: Visual Disentanglement Enhances Prompt Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNS5VCE3}},
  note         = {Machine review of arXiv:2508.00395}
}
read the original abstract

Prompt tuning (PT), as an emerging resource-efficient fine-tuning paradigm, has showcased remarkable effectiveness in improving the task-specific transferability of vision-language models. This paper delves into a previously overlooked information asymmetry issue in PT, where the visual modality mostly conveys more context than the object-oriented textual modality. Correspondingly, coarsely aligning these two modalities could result in the biased attention, driving the model to merely focus on the context area. To address this, we propose DAPT, an effective PT framework based on an intuitive decouple-before-align concept. First, we propose to explicitly decouple the visual modality into the foreground and background representation via exploiting coarse-and-fine visual segmenting cues, and then both of these decoupled patterns are aligned with the original foreground texts and the hand-crafted background classes, thereby symmetrically strengthening the modal alignment. To further enhance the visual concentration, we propose a visual pull-push regularization tailored for the foreground-background patterns, directing the original visual representation towards unbiased attention on the region-of-interest object. We demonstrate the power of architecture-free DAPT through few-shot learning, base-to-novel generalization, and data-efficient learning, all of which yield superior performance across prevailing benchmarks. Our code will be released at https://github.com/Ferenas/DAPT.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 54 canonical work pages

  1. [39]

    Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation,

    Y. Lin, M. Chen, W. Wang, B. Wu, K. Li, B. Lin, H. Liu, and X. He, “Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 15 305–15 314

    work page 2023

  2. [1]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P . Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763

    work page 2021

  3. [2]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Confer- ence on Machine Learning . PMLR, 2022, pp. 12 888– 12 900

    work page 2022

  4. [3]

    Open-vocabulary object detection via vision and language knowledge distillation,

    X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” in International Conference on Learning Rep- resentations, 2022

    work page 2022

  5. [4]

    Denseclip: Language-guided dense prediction with context-aware prompting,

    Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu, “Denseclip: Language-guided dense prediction with context-aware prompting,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 082–18 091

    work page 2022

  6. [5]

    Bert: Pre-training of deep bidirectional transformers for lan- guage understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for lan- guage understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  7. [6]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

    work page 1901

  8. [7]

    Learning to prompt for vision-language models,

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” International Journal of Computer Vision, vol. 130, no. 9, pp. 2337–2348, 2022

    work page 2022

Show all 68 references
  1. [8]

    Conditional prompt learning for vision-language models,

    ——, “Conditional prompt learning for vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 816– 16 825

    2022

  2. [9]

    Visual prompt tuning,

    M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in European Conference on Computer Vision. Springer, 2022, pp. 709–727

    2022

  3. [10]

    Prompt- aligned gradient for prompt tuning,

    B. Zhu, Y. Niu, Y. Han, Y. Wu, and H. Zhang, “Prompt- aligned gradient for prompt tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 659–15 669

    2023

  4. [11]

    Maple: Multi-modal prompt learning,

    M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, “Maple: Multi-modal prompt learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19 113–19 122

    2023

  5. [12]

    What does clip know about a red circle? visual prompt engineering for vlms,

    A. Shtedritski, C. Rupprecht, and A. Vedaldi, “What does clip know about a red circle? visual prompt engineering for vlms,” arXiv preprint arXiv:2304.06712, 2023

    2023 arXiv

  6. [13]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,” arXiv preprint arXiv:2310.11441 , 2023

    2023 arXiv

  7. [14]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2017, pp. 618–626

    2017

  8. [15]

    Segment everything everywhere all at once,

    X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Gao, and Y. J. Lee, “Segment everything everywhere all at once,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  9. [16]

    Plot: Prompt learning with optimal transport for vision- language models,

    G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang, “Plot: Prompt learning with optimal transport for vision- language models,” in The Eleventh International Confer- ence on Learning Representations, 2023

    2023

  10. [17]

    Visual-language prompt tuning with knowledge-guided context optimization,

    H. Yao, R. Zhang, and C. Xu, “Visual-language prompt tuning with knowledge-guided context optimization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6757–6767

    2023

  11. [18]

    Texts as images in prompt tuning for multi-label image recognition,

    Z. Guo, B. Dong, Z. Ji, J. Bai, Y. Guo, and W. Zuo, “Texts as images in prompt tuning for multi-label image recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 2808–2817

    2023

  12. [19]

    Self-regulating prompts: Foundational model adaptation without forgetting,

    M. U. Khattak, S. T. Wasim, M. Naseer, S. Khan, M.- H. Yang, and F. S. Khan, “Self-regulating prompts: Foundational model adaptation without forgetting,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 190–15 200

    2023

  13. [20]

    Promptkd: Unsupervised prompt distil- lation for vision-language models,

    Z. Li, X. Li, X. Fu, X. Zhang, W. Wang, S. Chen, and J. Yang, “Promptkd: Unsupervised prompt distil- lation for vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 617–26 626

    2024

  14. [21]

    Domain prompt learning with quaternion networks,

    Q. Cao, Z. Xu, Y. Chen, C. Ma, and X. Yang, “Domain prompt learning with quaternion networks,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 637–26 646

    2024

  15. [22]

    Generalized domain prompt learning for accessible scientific vision-language models,

    Q. Cao, Y. Chen, L. Lu, H. Sun, Z. Zeng, X. Yang, and D. Zhang, “Generalized domain prompt learning for accessible scientific vision-language models,” Nexus, vol. 2, no. 2, 2025

    2025

  16. [23]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rol- land, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al. , “Segment anything,” arXiv preprint arXiv:2304.02643, 2023

    2023 arXiv

  17. [24]

    Complementary patch for weakly supervised semantic segmentation,

    F. Zhang, C. Gu, C. Zhang, and Y. Dai, “Complementary patch for weakly supervised semantic segmentation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 7242–7251

    2021

  18. [25]

    Uncovering prototyp- ical knowledge for weakly open-vocabulary semantic JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, OCTOBER 2024 14 segmentation,

    F. Zhang, T. Zhou, B. Li, H. He, C. Ma, T. Zhang, J. Yao, Y. Zhang, and Y. Wang, “Uncovering prototyp- ical knowledge for weakly open-vocabulary semantic JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, OCTOBER 2024 14 segmentation,” Advances in Neural Information Processing Syst...

    2024

  19. [26]

    Attrseg: open-vocabulary semantic segmentation via at- tribute decomposition-aggregation,

    C. Ma, Y. Yuhuan, C. Ju, F. Zhang, Y. Zhang, and Y. Wang, “Attrseg: open-vocabulary semantic segmentation via at- tribute decomposition-aggregation,” Advances in neural information processing systems, vol. 36, pp. 10 258–10 270, 2023

    2023

  20. [27]

    Diffusionseg: Adapting diffusion to- wards unsupervised object discovery,

    C. Ma, Y. Yang, C. Ju, F. Zhang, J. Liu, Y. Wang, Y. Zhang, and Y. Wang, “Diffusionseg: Adapting diffusion to- wards unsupervised object discovery,” arXiv preprint arXiv:2303.09813, 2023

    2023 arXiv

  21. [28]

    Open-vocabulary semantic segmentation via attribute decomposition-aggregation,

    C. Ma, Y. Yang, C. Ju, F. Zhang, Y. Zhang, and Y. Wang, “Open-vocabulary semantic segmentation via attribute decomposition-aggregation,” arXiv preprint arXiv:2309.00096, 2023

    2023 arXiv

  22. [29]

    Probabilistic conformal distillation for enhanc- ing missing modality robustness,

    M. Chen, F. Zhang, Z. Zhao, J. Yao, Y. Zhang, and Y. Wang, “Probabilistic conformal distillation for enhanc- ing missing modality robustness,” Advances in Neural Information Processing Systems, vol. 37, pp. 36 218–36 242, 2024

    2024

  23. [30]

    Context: Driving in-context learning for text removal and segmentation,

    F. Zhang, P . Zhang, B. Yang, F. Huang, Y. Wang, and Y. Zhang, “Context: Driving in-context learning for text removal and segmentation,” arXiv preprint arXiv:2506.03799, 2025

    2025 arXiv

  24. [31]

    G4seg: Generation for inexact segmentation refinement with diffusion models,

    T. Zhang, F. Zhang, J. Yao, Y. Zhang, and Y. Wang, “G4seg: Generation for inexact segmentation refinement with diffusion models,” arXiv preprint arXiv:2506.01539, 2025

    2025 arXiv

  25. [32]

    Fine- grained visual prompting,

    L. Yang, Y. Wang, X. Li, X. Wang, and J. Yang, “Fine- grained visual prompting,” Advances in Neural Informa- tion Processing Systems, vol. 36, 2024

    2024

  26. [33]

    Alpha-clip: A clip model focusing on wherever you want,

    Z. Sun, Y. Fang, T. Wu, P . Zhang, Y. Zang, S. Kong, Y. Xiong, D. Lin, and J. Wang, “Alpha-clip: A clip model focusing on wherever you want,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 019–13 029

    2024

  27. [34]

    Vip-llava: Making large mul- timodal models understand arbitrary visual prompts,

    M. Cai, H. Liu, S. K. Mustikovela, G. P . Meyer, Y. Chai, D. Park, and Y. J. Lee, “Vip-llava: Making large mul- timodal models understand arbitrary visual prompts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 12 914–12 923

    2024

  28. [35]

    Learning deep features for discriminative localization,

    B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 2921–2929

    2016

  29. [36]

    Weakly-supervised semantic segmentation by iteratively mining common object features,

    X. Wang, S. You, X. Li, and H. Ma, “Weakly-supervised semantic segmentation by iteratively mining common object features,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 1354– 1362

    2018

  30. [37]

    Learning affinity from attention: end-to-end weakly-supervised semantic segmentation with transformers,

    L. Ru, Y. Zhan, B. Yu, and B. Du, “Learning affinity from attention: end-to-end weakly-supervised semantic segmentation with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 846–16 855

    2022

  31. [38]

    Seggpt: Segmenting everything in context,

    X. Wang, X. Zhang, Y. Cao, W. Wang, C. Shen, and T. Huang, “Seggpt: Segmenting everything in context,” arXiv preprint arXiv:2304.03284, 2023

    2023 arXiv

  32. [40]

    Learning to rank in person re-identification with metric ensembles,

    S. Paisitkriangkrai, C. Shen, and A. Van Den Hengel, “Learning to rank in person re-identification with metric ensembles,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 1846– 1855

    2015

  33. [41]

    Large-scale unsupervised semantic segmenta- tion,

    S. Gao, Z.-Y. Li, M.-H. Yang, M.-M. Cheng, J. Han, and P . Torr, “Large-scale unsupervised semantic segmenta- tion,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    2022

  34. [42]

    Learning gener- ative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,

    L. Fei-Fei, R. Fergus, and P . Perona, “Learning gener- ative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” in 2004 conference on computer vision and pattern recognition workshop. IEEE, 2004, pp. 178–178

    2004

  35. [43]

    Cats and dogs,

    O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and dogs,” in 2012 IEEE conference on computer vision and pattern recognition . IEEE, 2012, pp. 3498– 3505

    2012

  36. [44]

    3d object representations for fine-grained categorization,

    J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in Pro- ceedings of the IEEE international conference on computer vision workshops, 2013, pp. 554–561

    2013

  37. [45]

    Automated flower classification over a large number of classes,

    M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in 2008 Sixth Indian conference on computer vision, graphics & image processing. IEEE, 2008, pp. 722–729

    2008

  38. [46]

    Food- 101–mining discriminative components with random forests,

    L. Bossard, M. Guillaumin, and L. Van Gool, “Food- 101–mining discriminative components with random forests,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13. Springer, 2014, pp. 446–461

    2014

  39. [47]

    Fine-grained visual classification of aircraft,

    S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine-grained visual classification of aircraft,” arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  40. [48]

    Sun database: Large-scale scene recognition from abbey to zoo,

    J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database: Large-scale scene recognition from abbey to zoo,” in 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 2010, pp. 3485–3492

    2010

  41. [49]

    Ucf101: A dataset of 101 human actions classes from videos in the wild,

    K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402, 2012. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, OCTOBER 2024 15

    2012 arXiv

  42. [50]

    Describing textures in the wild,

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 3606–3613

    2014

  43. [51]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,

    P . Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 7, pp. 2217–2226, 2019

    2019

  44. [52]

    Do imagenet classifiers generalize to imagenet?

    B. Recht, R. Roelofs, L. Schmidt, and V . Shankar, “Do imagenet classifiers generalize to imagenet?” in Interna- tional conference on machine learning . PMLR, 2019, pp. 5389–5400

    2019

  45. [53]

    Learning robust global representations by penalizing local predic- tive power,

    H. Wang, S. Ge, Z. Lipton, and E. P . Xing, “Learning robust global representations by penalizing local predic- tive power,” Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  46. [54]

    Natural adversarial examples,

    D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural adversarial examples,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15 262–15 271

    2021

  47. [55]

    The many faces of robustness: A critical analysis of out-of-distribution generalization,

    D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo et al., “The many faces of robustness: A critical analysis of out-of-distribution generalization,” in Proceedings of the IEEE/CVF International Conference on Computer Vision...

    2021

  48. [56]

    Blip-2: Bootstrap- ping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrap- ping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning. PMLR, 2023, pp. 19 730– 19 742

    2023

  49. [57]

    Deepcore: A comprehen- sive library for coreset selection in deep learning,

    C. Guo, B. Zhao, and Y. Bai, “Deepcore: A comprehen- sive library for coreset selection in deep learning,” in International Conference on Database and Expert Systems Applications. Springer, 2022, pp. 181–195

    2022

  50. [58]

    A combinatorial strongly polynomial algorithm for minimizing submod- ular functions,

    S. Iwata, L. Fleischer, and S. Fujishige, “A combinatorial strongly polynomial algorithm for minimizing submod- ular functions,” Journal of the ACM (JACM), vol. 48, no. 4, pp. 761–777, 2001

    2001

  51. [59]

    Selection via proxy: Efficient data selection for deep learning,

    C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P . Bailis, P . Liang, J. Leskovec, and M. Zaharia, “Selection via proxy: Efficient data selection for deep learning,” arXiv preprint arXiv:1906.11829, 2019

    1906 arXiv

  52. [60]

    Glister: Generalization based data subset selection for efficient and robust learning,

    K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, and R. Iyer, “Glister: Generalization based data subset selection for efficient and robust learning,” inProceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 9, 2021, pp. 8110–8118

    2021

  53. [61]

    Deep learning on a data diet: Finding important examples early in training,

    M. Paul, S. Ganguli, and G. K. Dziugaite, “Deep learning on a data diet: Finding important examples early in training,” Advances in Neural Information Processing Systems, vol. 34, pp. 20 596–20 607, 2021

    2021

  54. [62]

    Active learning by acquiring contrastive examples,

    K. Margatina, G. Vernikos, L. Barrault, and N. Aletras, “Active learning by acquiring contrastive examples,” arXiv preprint arXiv:2109.03764, 2021

    2021 arXiv

  55. [63]

    A survey on label- efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction,

    W. Shen, Z. Peng, X. Wang, H. Wang, J. Cen, D. Jiang, L. Xie, X. Yang, and Q. Tian, “A survey on label- efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  56. [64]

    Freeseg: Unified, universal and open-vocabulary image segmentation,

    J. Qin, J. Wu, P . Yan, M. Li, R. Yuxi, X. Xiao, Y. Wang, R. Wang, S. Wen, X. Pan et al. , “Freeseg: Unified, universal and open-vocabulary image segmentation,” arXiv preprint arXiv:2303.17225, 2023

    2023 arXiv

  57. [65]

    Fine-grained visual prompting,

    L. Yang, Y. Wang, X. Li, X. Wang, and J. Yang, “Fine-grained visual prompting,” arXiv preprint arXiv:2306.04356, 2023

    2023 arXiv

  58. [66]

    Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization,

    S. Sagawa, P . W. Koh, T. B. Hashimoto, and P . Liang, “Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization,” arXiv preprint arXiv:1911.08731, 2019

    1911 arXiv

  59. [67]

    A compre- hensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes,

    M. Moayeri, P . Pope, Y. Balaji, and S. Feizi, “A compre- hensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 19 087–19 097

    2022

  60. [68]

    Noise or signal: The role of image backgrounds in object recognition,

    K. Xiao, L. Engstrom, A. Ilyas, and A. Madry, “Noise or signal: The role of image backgrounds in object recognition,” arXiv preprint arXiv:2006.09994, 2020. Fei Zhang is pursuing the Ph.D. degree in Shang- hai Jiao Tong University, Shanghai, China. He is also with Shanghai Inn...

    2006 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.