un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP

Hong Chang; Jiahe Zhao; Ruibing Hou; Shiguang Shan; Xilin Chen; Yinqi Li

REVIEW 4 major objections 4 minor 1 cited by

un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims CLIP's weak fine-detail vision can be repaired by finetuning its image encoder against the frozen unCLIP generator that inverts it, transferring detail-capturing knowledge while keeping language alignment.

desk verdict A genuinely new image-only CLIP fine-tuning method with real gains on detail benchmarks, but the 'alignment preserved' claim is undercut by a large zero-shot classification drop and needs more evidence. read the letter →

arxiv 2505.24517 v1 pith:XBGMKSBZ submitted 2025-05-30 cs.CV

Yinqi Li , Jiahe Zhao , Hong Chang , Ruibing Hou , Shiguang Shan , Xilin Chen This is my paper

classification cs.CV

keywords CLIPunvisualdetailcapturingdiffusionmodelsgenerativemodelinversionopen-vocabularysegmentationmultimodallargelanguageMMVP-VLMbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that CLIP's weak fine-detail vision can be repaired without any new text annotations, by finetuning its image encoder against a frozen generative model called unCLIP. unCLIP was trained to invert the original CLIP image encoder: it takes a CLIP image embedding and generates an image back from it, so its reconstructions visibly expose which details CLIP discarded. The proposed un2CLIP reverses this pipeline—freeze the generator and update the encoder so that the generator can reconstruct the input image more accurately—which the authors show amounts to maximizing how much information the embedding carries about the image. Because unCLIP's input space is exactly CLIP's image-text embedding space, the finetuned encoder's outputs remain readable by the original text encoder, preserving language alignment without any projection layer. On the CLIP-blind pairs of the MMVP-VLM benchmark, on training-free open-vocabulary segmentation, and on vision-centric MLLM benchmarks, the finetuned encoder outperforms the original CLIP and previous generative-improvement baselines.

What carries the argument

The load-bearing object is the frozen unCLIP generator $G$: a diffusion-based image generator conditioned on the output-layer image embeddings of the pretrained CLIP encoder, i.e., a learned inverse of $E$. It does two jobs at once. As a training signal it defines the objective $\min_E \mathbb{E}_{\mathbf{x},\boldsymbol{\epsilon},t}[\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_G(\mathbf{x}_t, t, E(\mathbf{x}))\|_2^2]$, so every gradient step asks the encoder to produce embeddings from which the generator can denoise the original image more accurately, transferring the generator's visual knowledge into the encoder. As a constraint, freezing the generator keeps the encoder's outputs close to unCLIP's input domain, which coincides with CLIP's image-text embedding space; this is what makes the design projector-free and what preserves the original text encoder's ability to read the new embeddings. The same diffusion loss also serves as a label-free diagnostic metric whose value correlates with MMVP-VLM performance across training epochs.

What would settle it

Pick a CLIP-blind pair from MMVP-VLM whose distinguishing detail the original encoder maps to nearly the same embedding, and measure three things: (1) whether samples from the frozen generator, conditioned on the original embedding, actually show that detail; (2) whether the un2CLIP-finetuned embeddings separate the pair in cosine distance while staying close to the image's caption under the frozen text encoder; and (3) whether the accuracy gain for that pattern tracks the drop in diffusion loss. If the generator's samples cannot show the detail in (1) yet MMVP accuracy still rises, the proposed detail-transfer mechanism is not what drives the improvement.

Watch

Extended reading notes

Core claim

The central claim is that inverting an unCLIP generator improves the very CLIP encoder that the generator was built from. The paper formalizes detail capture as maximizing the mutual information $I(\mathbf{x}; E(\mathbf{x}))$ between an image and its embedding, subject to keeping the embedding aligned with the image's text embedding; since $I(\mathbf{x}; E(\mathbf{x})) = H(\mathbf{x}) - H(\mathbf{x}|E(\mathbf{x}))$, this reduces to maximizing $\mathbb{E}_{\mathbf{x}}[\log p(\mathbf{x}|E(\mathbf{x}))]$. A pretrained unCLIP diffusion model was trained to maximize exactly this quantity, giving a probability model $p_G(\mathbf{x}|E(\mathbf{x}))$ that lives in the CLIP image-text embedding space, so the paper finetunes the encoder with the same diffusion loss $\min_E \mathbb{E}_{\mathbf{x},\boldsymbol{\epsilon},t}[\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_G(\mathbf{x}_t, t, E(\mathbf{x}))\|_2^2]$ while freezing $G$, using only images. The frozen generator doubles as the constraint that keeps the new embeddings inside the original embedding space. The paper reports that this image-only finetuning raises MMVP-VLM accuracy, improves open-vocabulary segmentation under multiple inference-time protocols, and helps LLaVA-1.5-based MLLMs on vision-centric benchmarks; it also shows that the diffusion loss itself tracks recognition performance across training epochs, but only when the generator is frozen.

Load-bearing premise

The method assumes the frozen generator is an accurate decoder of what the original CLIP embeddings contain: if a fine detail was discarded by the original encoder, the reconstruction gradient contains no trace of that detail, so any observed benchmark gain would come from an uncontrolled change in the embedding distribution rather than from the claimed transfer of detail-capturing ability.

Editorial extensions

If this is right

Substituting any of the tested CLIP image encoders (CLIP ViT-L-14 at 224 and 336, OpenCLIP ViT-H-14, SigLIP ViT-SO-14) with the un2CLIP-finetuned version improves MMVP-VLM accuracy over the original model and over DIVA on the same backbone.
On open-vocabulary semantic segmentation, the finetuned encoder lifts mIoU under the training-free MaskCLIP, SCLIP, and ClearCLIP protocols, adding 3.5 mIoU on average to ClearCLIP, while DIVA gains little and the contemporaneous GenHancer loses ground on this task.
The finetuned encoder transfers its gains to LLaVA-1.5-based MLLMs on vision-centric benchmarks (MMVP, NaturalBench, CV-Bench 2D and 3D, POPE) without changing LLaVA's default training configuration.
The diffusion loss works as a label-free predictor of recognition performance when the generator stays frozen: across training epochs lower loss tracks higher MMVP-VLM accuracy, and the link breaks if the generator is updated or a learnable projection layer is inserted.
The method needs only image data (one epoch of CC3M) and no region-text annotations, so it applies upstream to any CLIP backbone for which a CLIP-embedding-conditioned generator exists or is trained.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension of the paper's mechanism is that benchmark gains should scale with the capacity and coverage of the backend generator, because the objective is an estimate of $p(\mathbf{x}|E(\mathbf{x}))$; a stronger unCLIP model or a larger image corpus should push detail capture further, and a weak generator should cap it.
The paper's appendix shows an explicit trade-off that the main text does not emphasize: the detail-improved encoder loses coarse zero-shot classification accuracy (ImageNet-1K drops from 75.5 to 62.4 for CLIP ViT-L-14@224). A practical consequence the authors leave implicit is that deployments may want both encoders, the original for coarse tasks and the finetuned one for dense or vision-centric t
Because the diagnostic diffusion loss needs no labels, it could serve as a drift monitor for any future finetuning of CLIP-like encoders: practitioners could early-stop or detect alignment loss on unlabeled data without running full benchmark suites.
The framework suggests a general recipe beyond CLIP: any discriminative encoder that has a conditional generative decoder trained in its own latent space could be sharpened by inverting that decoder. Whether the same trick helps other contrastive or self-supervised vision encoders is an open question this paper does not address.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The one thing to know: un2CLIP is a genuinely new way to improve CLIP's detail capture — fine-tune the image encoder against a frozen unCLIP generator, with no projector, in the aligned embedding space. It is clean, image-only, and the gains on MMVP-VLM and open-vocabulary segmentation are consistent across several CLIP backbones. That part is real and worth building on.

The problem is the other half of the claim. The paper says the fine-tuned encoder preserves alignment with the original text embedding space 'simultaneously.' Table 8 tells a different story: ImageNet zero-shot drops from 75.5 to 62.4, a 13-point shift. DIVA, by comparison, keeps 75.5. That is not a small perturbation; it is substantial embedding drift. The authors acknowledge a trade-off, but the abstract and conclusion overclaim preservation. The frozen generator is a reasonable heuristic, but it is not a constraint, and no direct measurement of embedding drift or text alignment is reported. The retrieval numbers are mixed (COCO text-to-image improves, Flickr30K image-to-text drops slightly), which is not enough to support the strong statement.

Other soft spots: MMVP is a small benchmark (150 pairs, 15 per pattern), and there are no error bars or significance tests; some per-category numbers differ by a single pair. The MLLM comparison is incomplete — NaturalBench results are missing for DIVA and GenHancer, so the comparison is not fully apples-to-apples. The diffusion-loss diagnostic in Figure 3 is self-referential (same loss as the training objective), though the main results do not depend on it. That is minor.

The ablations are the best part. The identity-initialized projector and the updating-generator experiments directly support freezing G, and the qualitative encoding-decoding visualizations are convincing. The core mechanism — pushing the encoder to make a generative model that inverts the original CLIP reconstruct better — is plausible and well-motivated. The stress-test worry that the gradient could exploit the generator's blind spots is real, but I don't think it's fatal; the segmentation gains, in particular, are large and consistent.

Who should read this: anyone fine-tuning CLIP for dense prediction or vision-centric MLLMs. It is a solid contribution with a load-bearing but fixable evidence gap. I would send it to peer review, but the reviewers should push for (a) direct alignment/drift measurements (retrieval, distance metrics), (b) error bars or at least variance across seeds, and (c) completed NaturalBench baselines. If the authors can show that the embedding drift is manageable and the gains are not a side effect of distribution shift, this becomes a strong paper. As is, it's conditional: real idea, real gains, overstated claim.

Referee Report

4 major / 4 minor

Summary. The paper proposes un2CLIP, a method to improve a pretrained CLIP image encoder by finetuning it to invert a frozen unCLIP generator. Since unCLIP is trained to generate images from CLIP image embeddings, the authors use the diffusion loss of the generator as a training objective for the encoder, aiming to transfer the generator's visual-detail knowledge while keeping the embeddings aligned with the original CLIP text embedding space. The method is evaluated on the MMVP-VLM benchmark, open-vocabulary segmentation with MaskCLIP/SCLIP/ClearCLIP, and MLLM benchmarks, with additional ablations and zero-shot classification/retrieval results in an appendix.

Significance. If the central claim holds, un2CLIP is a practical and elegant way to improve CLIP's fine-grained visual understanding using only image data and an existing generative model, without requiring region-text annotations. The paper is transparent about its design choices and includes useful ablations (frozen generator, projection layer, updating generator), providing evidence about which components matter. The method outperforms prior work DIVA and is competitive with the contemporaneous GenHancer on several benchmarks. However, the claim that alignment with the original embedding space is preserved 'simultaneously' is not established by the reported experiments, and some benchmark comparisons lack statistical support or complete baselines.

major comments (4)

[Sec. 3.2, Eqs. (3)-(5), and Appendix C, Table 8] The derivation equates the ideal objective in Eq. (4) with the fixed generator's likelihood in Eq. (5), but the generator was trained to invert the original encoder E0, not the evolving encoder E. As E is finetuned, the embeddings can leave the region where G is a faithful model, so minimizing Eq. (5) can reduce loss by exploiting G's off-manifold behavior rather than by encoding more image detail. Freezing G is described only as 'encouraging' closeness, with no explicit constraint or drift measurement. The reported zero-shot ImageNet drop from 75.5 to 62.4 in Table 8 is a direct indication of substantial embedding drift and contradicts the abstract's claim of 'preserving the alignment to its original embedding space simultaneously.' The authors should either add an explicit alignment regularizer or provide alignment metrics (e.g., zero-shot classification, retrieval, text-image similarity) and reframe the claim as a trade-off rather than simultaneous preservation.
[Sec. 4.2, Table 1] The MMVP-VLM benchmark contains only 135 image pairs total (15 per pattern), and many reported differences between methods are small (e.g., 32.6 vs. 31.9 for OpenAI ViT-L-14@224). No error bars, standard deviations, or significance tests are reported, so the claim that un2CLIP 'significantly improves' over baselines is not supported on this benchmark. Please report variance across repeated runs or paired statistical tests, especially since the diffusion-loss evaluation itself samples random noises and timesteps.
[Sec. 4.4, Table 3] The MLLM evaluation is incomplete for comparing against prior methods: the NaturalBench results for DIVA and GenHancer are missing (shown as '-'), so the claim of improved performance over prior methods on vision-centric MLLM tasks cannot be assessed for this benchmark. Additionally, several other benchmarks in Table 3 show nearly identical or mixed results (e.g., Hallusion Avg. 28.4 for both Original and un2CLIP). Please provide the missing baselines or restrict the claim to benchmarks where complete comparisons are available.
[Sec. 4.5, Figure 3, and Appendix A] The 'diffusion loss as a diagnostic tool' is the same objective used for finetuning, so the observed correlation between lower diffusion loss and higher MMVP-VLM performance is partly self-referential: optimizing the training loss naturally decreases it. The ablation with updating the generator partially addresses this, but the paper should validate the diagnostic with an independent reconstruction or likelihood metric, or explicitly discuss the circularity and why the training-loss correlation is still informative.

minor comments (4)

[Table 1] The column headers for the nine MMVP-VLM patterns contain garbled symbols and are very difficult to parse; please use readable names or a legend.
[Throughout] There are several typos and spacing errors, e.g., 'objet detection' in Sec. 2, 'hasbeenapplied' and 'wefind' in the abstract/introduction; a careful proofread is needed.
[Appendix B.1, Table 6] The toy experiment for validating embedding-space alignment between ViT-L-14@224 and ViT-L-14@336 uses only zero-shot classification accuracy; reporting a direct distributional similarity (e.g., mean cosine similarity between corresponding embeddings) would strengthen the justification for sharing the same unCLIP model.
[Table 8] The text in Appendix C attributes the zero-shot classification drop to a coarse-versus-fine granularity trade-off, but the drop is quite large (75.5 to 62.4 on ImageNet). Please quantify how much of this drop is due to the absence of an alignment constraint versus a genuine detail-versus-semantics trade-off.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central finetuning objective is a plug-in conditional-likelihood estimate and all headline results are external benchmarks; only the diffusion-loss diagnostic is mildly self-referential, so the alignment-preservation weakness in Table 8 is a correctness concern, not a circular one.

full rationale

The derivation chain is not circular. The method's objective in Eq. (5) is obtained by substituting the pretrained unCLIP generator G as an estimate of p(x|E(x)) in the conditional-entropy expansion of I(x;E(x)); it is a plug-in likelihood objective, not a definition of the evaluation target. The paper's headline claims are validated on held-out external benchmarks (MMVP-VLM, open-vocabulary segmentation, MLLM tasks), none of which are used to fit E or select hyperparameters: training is on CC3M with the Stable unCLIP configuration. G and all baselines are taken from prior work rather than fitted by the authors, so no result is its own input renamed. The diffusion-loss diagnostic is the only mildly self-referential element, since the same loss is both the training objective and the 'detail capture' metric; however, the paper does not rest the central claim on this diagnostic, it explicitly anchors the diagnostic to the independent MMVP results, and it warns in footnote 2 that the diagnostic conclusion is design-dependent. The observed ImageNet zero-shot drop (75.5 to 62.4 in Table 8) undercuts the claim that alignment is preserved simultaneously, but an unverified assumption or a failed prediction is a correctness risk, not a circularity, because the claim is not made true by construction. No self-citation chain is load-bearing. Score 1 reflects the minor non-load-bearing self-reference in the diagnostic rather than any reduction of the central prediction to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The training objective relies on the diffusion variational bound and on the assumption that a frozen unCLIP generator is a good conditional model. No new entities are introduced. Hand-chosen hyperparameters (epochs, learning rate) follow the DIVA protocol and are not fitted to the target benchmarks.

free parameters (2)

Fine-tuning epochs = 1 epoch (default; 0 to 2 in ablations)
Chosen to match DIVA for fair comparison, not fitted to target benchmarks. Ablations in Figure 3 and Table 4 vary this.
Learning rate = 3e-7
Follows DIVA's setting; not tuned on the target benchmarks.

assumptions (4)

standard math Diffusion training loss is a variational bound on the conditional log-likelihood.
Invoked in Section 3.1 to connect generative modeling to the diffusion objective used in Eq. (2) and Eq. (5).
domain assumption The frozen unCLIP generator G approximates the true conditional distribution p(x|E(x)), so maximizing log p_G(x|E(x)) approximates maximizing mutual information I(x; E(x)).
Section 3.2 says G 'provides a suitable probability model' for Eq. (4), but no evidence is given that the approximation is accurate for embeddings outside G's training distribution.
domain assumption Freezing G keeps the fine-tuned embeddings within the original CLIP embedding space, preserving language alignment.
Section 3.2, 'Reducing Language-shift During Finetuning'; the large zero-shot classification drop in Appendix C suggests this assumption only partially holds.
domain assumption OpenAI CLIP ViT-L-14@224 and @336 have interchangeable embedding spaces, allowing one unCLIP model to serve both.
Appendix B.1 validates via zero-shot encoder/text swaps, but the swap results are only weak evidence across a few datasets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP." pith.science (2026). https://pith.science/paper/XBGMKSBZ

@misc{pith2026250524517,
  author       = {Pith},
  title        = {Pith review of: un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBGMKSBZ}},
  note         = {Machine review of arXiv:2505.24517}
}

abstract

Contrastive Language-Image Pre-training (CLIP) has become a foundation model and has been applied to various vision and multimodal tasks. However, recent works indicate that CLIP falls short in distinguishing detailed differences in images and shows suboptimal performance on dense-prediction and vision-centric multimodal tasks. Therefore, this work focuses on improving existing CLIP models, aiming to capture as many visual details in images as possible. We find that a specific type of generative models, unCLIP, provides a suitable framework for achieving our goal. Specifically, unCLIP trains an image generator conditioned on the CLIP image embedding. In other words, it inverts the CLIP image encoder. Compared to discriminative models like CLIP, generative models are better at capturing image details because they are trained to learn the data distribution of images. Additionally, the conditional input space of unCLIP aligns with CLIP's original image-text embedding space. Therefore, we propose to invert unCLIP (dubbed un$^2$CLIP) to improve the CLIP model. In this way, the improved image encoder can gain unCLIP's visual detail capturing ability while preserving its alignment with the original text encoder simultaneously. We evaluate our improved CLIP across various tasks to which CLIP has been applied, including the challenging MMVP-VLM benchmark, the dense-prediction open-vocabulary segmentation task, and multimodal large language model tasks. Experiments show that un$^2$CLIP significantly improves the original CLIP and previous CLIP improvement methods. Code and models will be available at https://github.com/LiYinqi/un2CLIP.

Figures

Figures reproduced from arXiv: 2505.24517 by the authors.

**Figure 1.** Comparison of unCLIP [20] and un2CLIP pipelines. (a) (b) unCLIP provides an encodingdecoding tool for observing which features of the image are disregarded by CLIP (more examples are shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Open-vocabulary semantic segmentation qualitative comparison. Qualitative Results. In [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Diffusion loss and MMVP-VLM performance with respect to training epochs [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: unCLIP generated images using original and finetuned CLIP models. Visualization Analysis. We give a visualization analysis of our default method and some representative ablations. Specifically, we use the finetuned encoder to perform the encoding-decoding pipeline on…

**Figure 5.** Figure 5: presents qualitative examples of the MMVP-VLM benchmark. For each case, we also apply the CLIP-unCLIP encoding-decoding pipeline to both the original and our improved CLIP models, as done in the visualization analysis paragraph in Section 4.5 of the main paper. These g…

**Figure 6.** Figure 6: presents qualitative examples of MLLM tasks, focusing on vision-centric benchmarks. Question: Is the dog wearing number 2 clearly in the lead? Is there a ladder in the background? Yes. Yes. Yes. No. Is the dog wearing number 2 clearly in the lead? Is there a ladder in …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Contextualized Multimodal Lifelong Person Re-Identification in Hybrid Clothing States
cs.CV 2025-09 reject novelty 3.0 of 10

CMLReID uses dynamic text prompts and dual-path prototypes to improve lifelong person re-identification in hybrid clothing states, reporting gains of about 5 mAP over four baselines.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

Diffusion feedback helps CLIP see better

Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps CLIP see better. InInternational Conference on Learning Representations, 2025

work page 2025
[2]

GenHancer: Imperfect generative models are secretly strong vision-centric enhancers.arXiv preprint arXiv:2503.19480, 2025

Shijie Ma, Yuying Ge, Teng Wang, Yuxin Guo, Yixiao Ge, and Ying Shan. GenHancer: Imperfect generative models are secretly strong vision-centric enhancers.arXiv preprint arXiv:2503.19480, 2025

arXiv 2025
[3]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022

2022
[4]

Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InAnnual Meeting of the Association for Computational Linguistics, 2018

work page 2018
[5]

CLIPScore: A reference-free evaluation metric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, 2021

work page 2021
[6]

ImageNet: Alarge-scalehierarchical imagedatabase

JiaDeng,WeiDong,RichardSocher,Li-JiaLi,KaiLi,andLiFei-Fei. ImageNet: Alarge-scalehierarchical imagedatabase. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009

work page 2009
[7]

Learning multiple layers of features from tiny images

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009

2009
[8]

Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. InComputer Vision and Pattern Recognition Workshop, pages 178–178, 2004

work page 2004

Show all 19 references

[9]

SUN database: Large-scale scene recognition from abbey to zoo

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. InIEEE Conference on Computer Vision and Pattern Recognition, pages 3485–3492, 2010

2010
[10]

Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151, 2013

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151, 2013

2013 arXiv
[11]

3D object representations for fine-grained categorization

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In IEEE International Conference on Computer Vision Workshops, pages 554–561, 2013

2013
[12]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInternational Conf...

2021
[13]

Reproducible scaling laws for contrastive language-image learning

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF Conference on Computer Vision ...

2023
[14]

Kingma and Max Welling

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. InInternational Conference on Learning Representations, 2014

2014
[15]

Sigmoid loss for language image pre-training

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11975–11986, 2023

2023
[16]

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.Transactions of the Association for Computational Linguistics, 2:67–78, 2014

2014
[17]

Microsoft COCO: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. InEuropean Conference on Computer Vision, pages 740–755, 2014

2014
[18]

Maskedautoencoders are scalable vision learners

KaimingHe, XinleiChen, SainingXie, YanghaoLi, PiotrDollár, andRossGirshick. Maskedautoencoders are scalable vision learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022

2022
[19]

An empirical study of training self-supervised vision trans- formers

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision trans- formers. InProceedings of the IEEE/CVF International Conference on Computer Vision,pages9640–9649, 2021. 19

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Diffusion feedback helps CLIP see better

Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps CLIP see better. InInternational Conference on Learning Representations, 2025

work page 2025

[2] [2]

GenHancer: Imperfect generative models are secretly strong vision-centric enhancers.arXiv preprint arXiv:2503.19480, 2025

Shijie Ma, Yuying Ge, Teng Wang, Yuxin Guo, Yixiao Ge, and Ying Shan. GenHancer: Imperfect generative models are secretly strong vision-centric enhancers.arXiv preprint arXiv:2503.19480, 2025

arXiv 2025

[3] [3]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022

2022

[4] [4]

Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InAnnual Meeting of the Association for Computational Linguistics, 2018

work page 2018

[5] [5]

CLIPScore: A reference-free evaluation metric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, 2021

work page 2021

[6] [6]

ImageNet: Alarge-scalehierarchical imagedatabase

JiaDeng,WeiDong,RichardSocher,Li-JiaLi,KaiLi,andLiFei-Fei. ImageNet: Alarge-scalehierarchical imagedatabase. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009

work page 2009

[7] [7]

Learning multiple layers of features from tiny images

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009

2009

[8] [8]

Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. InComputer Vision and Pattern Recognition Workshop, pages 178–178, 2004

work page 2004