Pith. sign in

REVIEW 3 major objections 7 minor 98 references

Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By fine-tuning only CLIP's image encoder on images, guided by DINOv2's pairwise similarity kernel, the paper claims CLIP gains the fine-grained perception it lacks while remaining compatible with its frozen text encoder.

desk verdict A solid, cheap CLIP fine-tuning recipe with a real compatibility gap between the guarantee and the loss. read the letter →

arxiv 2506.02557 v1 pith:3EZMOF5P submitted 2025-06-03 cs.CV

Shizhan Gong , Yankai Jiang , Qi Dou , Farzan Farnia This is my paper
classification cs.CV
keywords kernel-basedalignmentCLIPDINOv2fine-grainedperceptionzero-shotclassificationvision-languagemodelsembeddingregularizationvisualrepresentationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLIP's visual encoder is trained against captions and therefore sees images coarsely: it confuses images that share semantics but differ in color, count, orientation, or spatial layout. The paper tries to fix that without any text data or retraining of the text branch by fine-tuning the image encoder so that its pairwise similarities among images, measured by a polynomial kernel, match the pairwise similarities produced by DINOv2, a self-supervised model known for fine detail. An L2 penalty keeps each new embedding near its original position, and the authors prove a bound on how much cosine similarity to text embeddings can change as a function of that movement. On CLIP-Benchmark, probing, and VQA evaluations they report consistent gains in zero-shot recognition, counting and spatial reasoning, localization, and downstream LLaVA and OpenFlamingo performance. If the claim holds, cheap image-only retuning can upgrade CLIP and compatible MLLMs without the costly realignment most visual-enhancement methods require.

What carries the argument

The central object is the normalized polynomial kernel $k(x,y)=\frac{(\gamma x^\top y + c)^d}{\sqrt{k(x,x)k(y,y)}}$ evaluated on image embeddings, paired with the alignment objective $\min_\theta\, w\,\mathbb{E}[(k_1(f_\theta(I_i),f_\theta(I_j))-k_2(g(I_i),g(I_j)))^2]+\mathbb{E}[\|f_\theta(I_i)-f_{\theta_0}(I_i)\|_2^2]$. The kernel defines what "similar" means in each model's space, so matching the two kernel matrices transfers DINOv2's fine-grained similarity geometry to CLIP, while the L2 term anchors the new embeddings to their old positions to preserve text cosine similarities, per the bound in Proposition 3.2.

What would settle it

On a held-out set, record the per-image L2 distance between original and aligned embeddings and the resulting change in cosine similarity to a fixed set of text embeddings; if a nontrivial fraction of images moves far enough that the Proposition 3.2 bound no longer confines the cosine change, and retrieval or zero-shot accuracy drops on those images, the compatibility claim fails. A simpler version: ablate the regularization by setting its weight to zero, as the authors do, and show that the aligned encoder's text-image retrieval collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that two embedding spaces can be reconciled in kernel space rather than feature space: instead of mapping DINOv2 vectors onto CLIP's vectors with a linear layer, the authors minimize the expected squared difference between the CLIP image-encoder's kernel function evaluated on pairs and DINOv2's kernel on the same pairs. Because the kernel records relative similarity between samples, this reshapes CLIP's neighborhoods to respect fine-grained visual distinctions while leaving the global structure of CLIP's space intact. The added regularization term ties the updated embedding to the original one, and Proposition 3.2 gives a bound on the change in cosine similarity with any text embedding as a function of that L2 deviation; the authors treat this as the mechanism that keeps zero-shot, retrieval, and downstream MLLM compatibility alive. They report that the aligned encoder improves average zero-shot accuracy on 12 datasets for three CLIP variants, improves local and global probing recall, and when swapped into LLaVA and OpenFlamingo improves VQA benchmarks both with and without LLM fine-tuning.

Load-bearing premise

The whole scheme rests on the hope that a modest penalty term keeps the retuned image embeddings close enough to the original ones that their cosine matches to text stay intact; training never enforces a hard cap on that distance.

Editorial extensions

If this is right

  • Zero-shot classification on CLIP-Benchmark improves across three CLIP backbones, with the largest gains on low-resolution and small-object datasets such as CIFAR and EuroSAT.
  • Counting, spatial reasoning, and caption-recognition tasks (SVHN, GTSRB, CLEVR) improve under both zero-shot and linear-probe evaluation, and local and global localization probing recall rises.
  • The aligned visual encoder can replace the original in LLaVA and OpenFlamingo without retraining the LLM, and LoRA fine-tuning of LLaVA amplifies the gains, especially on referring-expression benchmarks.
  • The alignment recipe transfers to other VLM text encoders (SigLIP, DFN, MetaCLIP) and other vision target models (MLCD), so it is not specific to CLIP-DINOv2.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the alignment never touches the text encoder, the same procedure could plausibly be re-run on the aligned model or stacked with other CLIP upgrades; the authors only hint at orthogonality with DIVA.
  • Beyond the paper: a testable extension is to scale the image-only fine-tuning data from ImageNet-1K toward web-scale pools; the authors' own data-size ablation shows gains grow with more data, suggesting further headroom.
  • Beyond the paper: the reliance on pairwise kernel matching suggests a general recipe—any target encoder that excels on a missing perceptual axis can donate its similarity geometry to a frozen multimodal encoder, provided a regularization anchor holds.
  • Beyond the paper: if the cosine-preservation bound is the load-bearing piece, monitoring realized L2 drift during training could serve as a simple diagnostic for when alignment is about to break text compatibility.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes KUEA, a kernel-based unsupervised fine-tuning method that aligns CLIP's visual encoder with DINOv2's embeddings. The training objective (Eq. 6) minimizes the squared difference between a polynomial kernel evaluated on CLIP image embeddings and the same kernel evaluated on DINOv2 embeddings, plus an L2 regularization term that penalizes deviation from the original CLIP embeddings. The method is image-only, keeps the text encoder frozen, and is evaluated on zero-shot classification, image-text retrieval, counting/spatial reasoning, localization probing, and two MLLMs (LLaVA and OpenFlamingo). The authors report average zero-shot accuracy gains of 0.8–1.3 percentage points across CLIP variants, consistent gains in retrieval, and improvements in downstream MLLMs without retuning the LLM. They provide ablations on training epochs, regularization weight, kernel choice, and training-data size, as well as comparisons with DIVA, AM-RADIO, and Additive-MoF. The central claim is that a lightweight, image-only alignment fine-tune improves fine-grained visual perception while retaining compatibility with the frozen text encoder and downstream models.

Significance. If the central claim holds, the method offers a practical recipe for upgrading CLIP-style visual encoders at a fraction of the cost of retraining, with immediate plug-in compatibility for existing MLLMs. The paper's strengths are its breadth of evaluation (12 zero-shot datasets, retrieval, localization probing, two MLLMs), multiple CLIP backbone scales, ablations of key design choices, and the release of code and models. The idea of aligning kernel matrices rather than raw embeddings is simple and plausible, and the empirical gains, though modest, are consistent across most settings. The main weaknesses are that the formal compatibility guarantee is not actually implied by the optimization objective, and the trainable parameters of the CLIP-side kernel introduce a potential confound that is not controlled in the experiments.

major comments (3)
  1. [§3.3, Eq. (6) and Proposition 3.2] Proposition 3.2 bounds the change in cosine similarity to text embeddings under a hard per-sample bound ||fθ(I) − fθ0(I)||₂ ≤ λ. However, Eq. (6) only penalizes the expected squared L2 displacement over the training distribution; nothing in the optimization enforces the pointwise bound. The text after Eq. (6) states that the regularization term 'ensures' that alignment does not cause large deviations, but it only encourages this. Consequently, the abstract's claim that the encoder 'retains compatibility with the frozen text encoder' is not formally supported by Proposition 3.2. Please either (a) verify the premise empirically by reporting the realized distribution of per-sample L2 displacements and the resulting cosine changes against text embeddings (e.g., percentiles on ImageNet or a held-out set), or (b) soften the guarantee language to state that compatibility is empirically preserved. As written, the theory does not establish the central compatibility claim.
  2. [§4.1, Eq. (4) and Table 7] The paper states that for the CLIP branch, the polynomial kernel hyperparameters γ and c are set as trainable, while the DINOv2-side kernel has fixed γ = 1/dim_emb and c = 1. This means the alignment loss in Eq. (4) can be reduced by adapting the kernel function itself, rather than by moving the visual embeddings toward a DINOv2-like geometry. Since all zero-shot evaluations use standard cosine similarity (not the trained kernel), the mechanism behind the reported improvements is ambiguous. Please report the learned values of γ and c, and add an ablation that fixes the CLIP kernel parameters (e.g., to the same values as the DINOv2 side). If the improvements persist with fixed kernel parameters, the attribution of the gain to the visual encoder is much better supported.
  3. [Table 3, ViT-B-16 SVHN row] The ViT-B-16 SVHN zero-shot accuracy drops from 31.31 to 27.40 after alignment, a 3.91-point decrease on a digit-recognition task that falls squarely within the paper's fine-grained perception focus. The text reports only averaged gains (e.g., 'alignment improves the average accuracy by 1.58% for ViT-L-14'), which can mask such degradation. Please report per-dataset results with error bars or standard deviations across multiple runs, and explicitly discuss which benchmarks degrade and why. The current presentation substantially overstates the consistency of the improvement.
minor comments (7)
  1. [§3.3] The sentence 'The regularization term ensures that the alignment process does not cause the aligned embeddings to deviate significantly from the original embeddings' should be reworded to 'encourages', since Eq. (6) is a soft penalty, not a constraint.
  2. [Table 2 and §4.2] The dataset name 'Flicker30K' is misspelled; it should be 'Flickr30K' throughout.
  3. [Appendix A.2] Proposition 3.2 is attributed to Schlarmann et al. (2024) but the proof is said to be 'adapted'; please provide a precise citation to the original statement and verify that the bound is stated identically, or present the result as a new proposition if the adaptation is nontrivial.
  4. [Figures 3 and 5] Figure 3 in the main text and Figure 5 in Appendix C.7 appear to be the same ablation panels; please consolidate them to avoid duplication.
  5. [§1] The phrase 'w/wo alignment' is informal; consider using 'with and without alignment' in the tables and text.
  6. [§3.1] The notation for the polynomial kernel in Eq. (1) uses k_polynomial(γ,c,d)(x,y); in Eq. (2) the normalization denotes the kernel by k̃, but the tilde may be hard to read in print. Consider using a more distinct symbol such as k̂.
  7. [References] The Related Work section contains a large number of self-citations to kernel-method papers; while not inappropriate, the relevance of several of them (e.g., the Vendi-score diversity papers) to CLIP alignment is indirect, and trimming would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central improvements are measured on external benchmarks, and the text-compatibility bound is cited from an external group, not derived from the paper's own fitted inputs.

full rationale

The paper's central claim is an empirical improvement in zero-shot classification, retrieval, fine-grained reasoning, localization probing, and downstream MLLM benchmarks after image-only kernel alignment of CLIP to DINOv2. These benchmarks are external to the training objective and are not constructed from the fitted parameters, so the improvements are independently measured rather than forced by definition. The only formal support for text-compatibility preservation, Proposition 3.2, is cited from Schlarmann et al. (2024), whose authors do not overlap with the present paper, and the proof is reproduced in Appendix A.2 as an external mathematical bound. The kernel hyperparameters for the CLIP branch are trainable and the coefficient w is chosen by ablation, but these are optimization design choices, not the predicted results; the reported gains are evaluated on held-out datasets. The skeptic's concern that Eq. (6) only penalizes expected squared L2 displacement while Proposition 3.2 requires a hard per-sample bound is a correctness or rigor gap, not a circularity: the paper's inference from soft penalty to guaranteed compatibility is unsupported, but it does not reduce the claim to its inputs or to a self-citation chain. The related-work citations to the authors' previous kernel-method papers are contextual and not load-bearing for the alignment derivation or the empirical evaluation. Therefore no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities. It relies on the polynomial kernel feature map, a Lipschitz assumption on the CLIP kernel, the empirical superiority of DINOv2 for fine-grained tasks, and a regularization argument adapted from prior work. The trainable kernel hyperparameters and the regularization weight are the main free parameters.

free parameters (4)
  • Regularization weight w = 0.5 (ViT-B/L), 1.0 (ViT-L-336)
    Chosen by hand and ablation (Fig. 3c); balances kernel alignment and preservation of original embeddings.
  • Polynomial kernel degree d = 3
    Selected via ablation (Fig. 3d); other kernels (Gaussian, cosine) also work but give lower performance.
  • CLIP kernel parameters gamma and c = trainable (values not reported)
    Set as trainable for the CLIP branch, while the DINOv2 branch uses gamma=1/dim_emb and c=1; these are optimized during alignment fine-tuning.
  • DINOv2 kernel gamma and c = gamma=1/dim_emb, c=1
    Chosen as fixed hyperparameters for the target model kernel.
assumptions (4)
  • standard math There exists a feature map for the polynomial kernel (Mercer's theorem)
    Invoked in Section 3.1 to represent kernel values as inner products.
  • domain assumption The CLIP kernel k1 is L-Lipschitz with respect to model parameters theta
    Assumed in Proposition 3.1 to apply the vector Bernstein inequality; the Lipschitz constant is not computed or bounded.
  • domain assumption DINOv2 embeddings provide better supervision for fine-grained visual details
    Underlies the whole method; cited from Tong et al. 2024 and Oquab et al. 2024, but not re-established here.
  • domain assumption The L2 regularization keeps the visual embedding close to the original, preserving text alignment (Proposition 3.2)
    Adapted from Schlarmann et al. 2024; the bound depends on the realized deviation, which is penalized but not hard-constrained in Eq. (6).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models." pith.science (2026). https://pith.science/paper/3EZMOF5P

@misc{pith2026250602557,
  author       = {Pith},
  title        = {Pith review of: Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3EZMOF5P}},
  note         = {Machine review of arXiv:2506.02557}
}
read the original abstract

Vision-language models, such as CLIP, have achieved significant success in aligning visual and textual representations, becoming essential components of many multi-modal large language models (MLLMs) like LLaVA and OpenFlamingo. However, numerous studies have identified CLIP's limited fine-grained perception as a critical drawback, leading to substantial failures in downstream MLLMs. In contrast, vision-centric foundation models like DINOv2 demonstrate remarkable capabilities in capturing fine details from images. In this work, we propose a novel kernel-based method to align CLIP's visual representation with that of DINOv2, ensuring that the resulting embeddings maintain compatibility with text embeddings while enhancing perceptual capabilities. Our alignment objective is designed for efficient stochastic optimization. Following this image-only alignment fine-tuning, the visual encoder retains compatibility with the frozen text encoder and exhibits significant improvements in zero-shot object recognition, fine-grained spatial reasoning, and localization. By integrating the aligned visual encoder, downstream MLLMs also demonstrate enhanced performance.

Figures

Figures reproduced from arXiv: 2506.02557 by the authors.

Figure 1
Figure 1. Main claim of this work. We propose a kernel-based alignment framework, which is able to enhance the visual representation of CLIP via image-only fine-tuning. Moreover, the improvement can be transferred to downstream multi-modal large language models. mining CLIP’s zero-shot performance. Additionally, there are efforts to integrate region-based task loss (Zhong et al., 2022; Wan et al., 2024) or distillation loss f… view at source ↗
Figure 2
Figure 2. Visualization of CLIP encoder’s attention maps. At￾tention maps can show more fine-grained details after alignment. explore replacing DINOv2 with other vision models, such as MLCD (An et al., 2024). The results show improved zero-shot accuracy for all these pairs, demonstrating the generalizability of the proposed framework. Image-to-text and Text-to-image Retrievals. To further demonstrate that the proposed alignme… view at source ↗
Figure 3
Figure 3. Ablation studies. Average zero-shot accuracy across 11 datasets are reported as evaluation metrics: (a) Effects of training epochs; (b) Effects of the regularization term; (c) Effects of the coefficients of alignment; (d) Effects of different kernel functions [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of the alignment loss curve. The alignment loss is stably decreasing during the alignment fine-tuning phase. C.2. Results for VLMs other than CLIP. To evaluate the generalization of our proposed framework, we conducted experiments on three additional mode…
Figure 5
Figure 5. Figure 5: Ablation studies. Average zero-shot accuracy across 11 datasets are reported as evaluation metrics: (a) Effects of training epochs; (b) Effects of the regularization term; (c) Effects of the coefficients of alignment; (d) Effects of different kernel functions. Effects …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

98 extracted references · 42 canonical work pages

  1. [1]

    Tallyqa: Answering complex counting questions

    Acharya, M., Kafle, K., and Kanan, C. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.\ 8076--8084, 2019

    2019

  2. [2]

    and Li, Y

    Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020

    arXiv 2012

  3. [3]

    Multi-label cluster discrimination for visual representation learning

    An, X., Yang, K., Dai, X., Feng, Z., and Deng, J. Multi-label cluster discrimination for visual representation learning. In European Conference on Computer Vision, pp.\ 428--444. Springer, 2024

    2024

  4. [4]

    W., Ilharco, G., Wortsman, M., and Schmidt, L

    Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., and Schmidt, L. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023

    arXiv 2023

  5. [5]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1 0 (2): 0 3, 2023

    arXiv 2023

  6. [6]

    BE it: BERT pre-training of image transformers

    Bao, H., Dong, L., Piao, S., and Wei, F. BE it: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=p-BhZSz59o4

    2022

  7. [7]

    J., Arbel, M., and Gretton, A

    Bi \'n kowski, M., Sutherland, D. J., Arbel, M., and Gretton, A. Demystifying mmd gans. arXiv preprint arXiv:1801.01401, 2018

    arXiv 2018

  8. [8]

    Domain prompt learning with quaternion networks

    Cao, Q., Xu, Z., Chen, Y., Ma, C., and Yang, X. Domain prompt learning with quaternion networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26637--26646, 2024

    2024

Show all 98 references
  1. [9]

    Emerging properties in self-supervised vision transformers

    Caron, M., Touvron, H., Misra, I., J \'e gou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 9650--9660, 2021

    2021

  2. [10]

    Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Doll \'a r, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  3. [11]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  4. [12]

    Remote sensing image scene classification: Benchmark and state of the art

    Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105 0 (10): 0 1865--1883, 2017

    2017

  5. [13]

    Describing textures in the wild

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , and Vedaldi, A. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition ( CVPR ) , 2014

    2014

  6. [14]

    Locality alignment improves vision-language models

    Covert, I., Sun, T., Zou, J., and Hashimoto, T. Locality alignment improves vision-language models. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=qssVptHTPN

    2025

  7. [15]

    Vision transformers need registers

    Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=2dnO3LLiJ1

    2024

  8. [16]

    Dehdashtian, S., Wang, L., and Boddeti, V. N. Fairerclip: Debiasing clip's zero-shot predictions using functions in rkhss. arXiv preprint arXiv:2403.15593, 2024

    2024 arXiv

  9. [17]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  10. [18]

    M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V

    Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023

    2023 arXiv

  11. [19]

    Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories

    Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004

    2004

  12. [20]

    and Dieng, A

    Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.02410, 2022

    2022 arXiv

  13. [21]

    Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al

    Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024

    2024

  14. [22]

    Clip-adapter: Better vision-language models with feature adapters

    Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132 0 (2): 0 581--595, 2024

    2024

  15. [23]

    3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable tumor segmentation

    Gong, S., Zhong, Y., Ma, W., Li, J., Wang, Z., Zhang, J., Heng, P.-A., and Dou, Q. 3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable tumor segmentation. Medical Image Analysis, 98: 0 103324, 2024

    2024

  16. [24]

    Boosting the visual interpretability of clip via adversarial fine-tuning

    Gong, S., Lei, H., Dou, Q., and Farnia, F. Boosting the visual interpretability of clip via adversarial fine-tuning. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=khuIvzxPRp

    2025

  17. [25]

    J., Erhan, D., Carrier, P

    Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. Challenges in representation learning: A report on three machine learning contests. In Neural information processing: 20th international ...

    2013

  18. [26]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 6904--6913, 2017

    2017

  19. [27]

    Recovering low-rank matrices from few coefficients in any basis

    Gross, D. Recovering low-rank matrices from few coefficients in any basis. IEEE Transactions on Information Theory, 57 0 (3): 0 1548--1566, 2011

    2011

  20. [28]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Guo, Z., Xu, R., Yao, Y., Cui, J., Ni, Z., Ge, C., Chua, T.-S., Liu, Z., and Huang, G. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, pp.\ 390--406. Springer, 2024

    2024

  21. [29]

    J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J

    Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3608--3617, 2018

    2018

  22. [30]

    and Ozay, M

    He, B. and Ozay, M. Feature kernel distillation. In International Conference on Learning Representations, 2022

    2022

  23. [31]

    Masked autoencoders are scalable vision learners

    He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022

    2022

  24. [32]

    Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Helber, P., Bischke, B., Dengel, A., and Borth, D. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pp.\ 204--207. IEEE, 2018

    2018

  25. [33]

    Natural adversarial examples

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. CVPR, 2021

    2021

  26. [34]

    Probability inequalities for sums of bounded random variables

    Hoeffding, W. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pp.\ 409--426, 1994

    1994

  27. [35]

    Hofmann, T., Sch \"o lkopf, B., and Smola, A. J. Kernel methods in machine learning. 2008

    2008

  28. [36]

    J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  29. [37]

    T., and Farnia, F

    Jalali, M., Li, C. T., and Farnia, F. An information-theoretic evaluation of generative models in learning multi-modal distributions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  30. [38]

    T., and Farnia, F

    Jalali, M., Li, C. T., and Farnia, F. An information-theoretic evaluation of generative models in learning multi-modal distributions. Advances in Neural Information Processing Systems, 36, 2024

    2024

  31. [39]

    From clip to dino: Visual encoders shout in multi-modal large language models

    Jiang, D., Liu, Y., Liu, S., Zhao, J., Zhang, H., Gao, Z., Zhang, X., Li, J., and Xiong, H. From clip to dino: Visual encoders shout in multi-modal large language models. arXiv preprint arXiv:2310.08825, 2023

    2023 arXiv

  32. [40]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 29...

    2017

  33. [41]

    What's ''up'' with vision-language models? investigating their struggle with spatial reasoning

    Kamath, A., Hessel, J., and Chang, K.-W. What's ''up'' with vision-language models? investigating their struggle with spatial reasoning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=RN5KLywTll

    2023

  34. [42]

    Studiogan: A taxonomy and benchmark of gans for image synthesis

    Kang, M., Shin, J., and Park, J. Studiogan: A taxonomy and benchmark of gans for image synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (12): 0 15725--15742, 2023. doi:10.1109/TPAMI.2023.3306436

    2023

  35. [43]

    Referitgame: Referring to objects in photographs of natural scenes

    Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp.\ 787--798, 2014

    2014

  36. [44]

    A diagram is worth a dozen images

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, pp.\ 235--251. Springer, 2016

    2016

  37. [45]

    The hateful memes challenge: Detecting hate speech in multimodal memes

    Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., and Testuggine, D. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33: 0 2611--2624, 2020

    2020

  38. [46]

    Kohler, J. M. and Lucchi, A. Sub-sampled cubic regularization for non-convex optimization. In International Conference on Machine Learning, pp.\ 1895--1904. PMLR, 2017

    1904

  39. [47]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  40. [48]

    Clip benchmark: Clip-like model evaluation, 2022

    LAION-AI. Clip benchmark: Clip-like model evaluation, 2022. URL https://github.com/LAION-AI/CLIP_benchmark

    2022

  41. [49]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 a

    2023

  42. [50]

    Transfer learning in computer vision tasks: Remember where you come from

    Li, X., Grandvalet, Y., Davoine, F., Cheng, J., Cui, Y., Zhang, H., Belongie, S., Tsai, Y.-H., and Yang, M.-H. Transfer learning in computer vision tasks: Remember where you come from. Image and Vision Computing, 93: 0 103853, 2020

    2020

  43. [51]

    Evaluating object hallucination in large vision-language models

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023 b . URL https://openreview.net/forum?id=xozJw0kZXF

    2023

  44. [52]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp....

    2014

  45. [53]

    Visual spatial reasoning

    Liu, F., Emerson, G., and Collier, N. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11: 0 635--651, 2023

    2023

  46. [54]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  47. [55]

    Decoupled weight decay regularization

    Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  48. [56]

    Understanding zero-shot adversarial robustness for large-scale models

    Mao, C., Geng, S., Yang, J., Wang, X., and Vondrick, C. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:2212.07016, 2022

    2022 arXiv

  49. [57]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp.\ 3195--3204, 2019

    2019

  50. [58]

    Y., et al

    Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp.\ 4. Granada, 2011

    2011

  51. [59]

    Dinov2: Learning robust visual features without supervision

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp.\ 1--31, 2024

    2024

  52. [60]

    and Farnia, F

    Ospanov, A. and Farnia, F. On the statistical complexity of estimating vendi scores from empirical data. arXiv preprint arXiv:2410.21719, 2024

    2024 arXiv

  53. [61]

    Dissecting clip: Decomposition with a schur complement-based approach

    Ospanov, A., Jalali, M., and Farnia, F. Dissecting clip: Decomposition with a schur complement-based approach. arXiv preprint arXiv:2412.18645, 2024 a

    2024 arXiv

  54. [62]

    Towards a scalable reference-free evaluation of generative models

    Ospanov, A., Zhang, J., Jalali, M., Cao, X., Bogdanov, A., and Farnia, F. Towards a scalable reference-free evaluation of generative models. Advances in Neural Information Processing Systems, 37: 0 120892--120927, 2024 b

    2024

  55. [63]

    M., Vedaldi, A., Zisserman, A., and Jawahar, C

    Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012

    2012

  56. [64]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  57. [65]

    Am-radio: Agglomerative vision foundation model reduce all domains into one

    Ranzinger, M., Heinrich, G., Kautz, J., and Molchanov, P. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 12490--12500, 2024

    2024

  58. [66]

    Rezaei, P., Farnia, F., and Li, C. T. Be more diverse than the most diverse: Online selection of diverse mixtures of generative models. arXiv preprint arXiv:2412.17622, 2024

    2024 arXiv

  59. [67]

    Improved zero-shot classification by adapting vlms with text descriptions

    Saha, O., Van Horn, G., and Maji, S. Improved zero-shot classification by adapting vlms with text descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 17542--17552, 2024

    2024

  60. [68]

    Clip meets model zoo experts: Pseudo-supervision for visual enhancement

    Salehi, M., Farajtabar, M., Horton, M., Faghri, F., Pouransari, H., Vemulapalli, R., Tuzel, O., Farhadi, A., Rastegari, M., and Mehta, S. Clip meets model zoo experts: Pseudo-supervision for visual enhancement. arXiv preprint arXiv:2310.14108, 2023

    arXiv

  61. [69]

    D., Croce, F., and Hein, M

    Schlarmann, C., Singh, N. D., Croce, F., and Hein, M. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. arXiv preprint arXiv:2402.12336, 2024

    2024 arXiv

  62. [70]

    Mome: Mixture of multimodal experts for generalist multimodal large language models

    Shen, L., Chen, G., Shao, R., Guan, W., and Nie, L. Mome: Mixture of multimodal experts for generalist multimodal large language models. arXiv preprint arXiv:2407.12709, 2024

    2024 arXiv

  63. [71]

    Finetuning text-to-image diffusion models for fairness

    Shen, X., Du, C., Pang, T., Lin, M., Wong, Y., and Kankanhalli, M. Finetuning text-to-image diffusion models for fairness. arXiv preprint arXiv:2311.07604, 2023

    2023 arXiv

  64. [72]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Shi, M., Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y., Shi, H., et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024

    2024 arXiv

  65. [73]

    Towards vqa models that can read

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8317--8326, 2019

    2019

  66. [74]

    Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32: 0 323--332, 2012

    2012

  67. [75]

    L., Taylor, E., and Loaiza-Ganem, G

    Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Y., Ross, B., Villecroze, V., Liu, Z., Caterini, A. L., Taylor, E., and Loaiza-Ganem, G. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Proces...

    2023

  68. [76]

    Eva-clip: Improved training techniques for clip at scale

    Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  69. [77]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., and Ross, C. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 5238--52...

    2022

  70. [78]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9568--9578, 2024

    2024

  71. [79]

    S., Linmans, J., Winkens, J., Cohen, T., and Welling, M

    Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation equivariant CNNs for digital pathology. June 2018

    2018

  72. [80]

    Clip the gap: A single domain generalization approach for object detection

    Vidit, V., Engilberge, M., and Salzmann, M. Clip the gap: A single domain generalization approach for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 3219--3229, 2023

    2023

  73. [81]

    S., Steiner, A

    Wan, B., Tschannen, M., Xian, Y., Pavetic, F., Alabdulmohsin, I., Wang, X., Pinto, A. S., Steiner, A. P., Beyer, L., and Zhai, X. Locca: Visual pretraining with location-aware captioners. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. UR...

    2024

  74. [82]

    Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019

    2019

  75. [83]

    Diffusion feedback helps clip see better

    Wang, W., Sun, Q., Zhang, F., Tang, Y., Liu, J., and Wang, X. Diffusion feedback helps clip see better. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tLFWU6izoA

    2025

  76. [84]

    Wu, S., Zhang, W., Xu, L., Jin, S., Li, X., Liu, W., and Loy, C. C. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403, 2023

    2023 arXiv

  77. [85]

    Demystifying CLIP data

    Xu, H., Xie, S., Tan, X., Huang, P.-Y., Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feichtenhofer, C. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=5BCFlnfE1g

    2024

  78. [86]

    Explicit inductive bias for transfer learning with convolutional networks

    Xuhong, L., Grandvalet, Y., and Davoine, F. Explicit inductive bias for transfer learning with convolutional networks. In International conference on machine learning, pp.\ 2825--2834. PMLR, 2018

    2018

  79. [87]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2: 0 67--78, 2014

    2014

  80. [88]

    C., and Berg, T

    Yu, L., Poirson, P., Yang, S., Berg, A. C., and Berg, T. L. Modeling context in referring expressions. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp.\ 69--85. Springer, 2016

    2016

  81. [89]

    Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip

    Yu, Q., He, J., Deng, X., Shen, X., and Chen, L.-C. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36: 0 32215--32234, 2023

    2023

  82. [90]

    When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023

    Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=KRLUvxh8uaX

    2023

  83. [91]

    Sigmoid loss for language image pre-training

    Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 11975--11986, 2023

    2023

  84. [92]

    T., and Farnia, F

    Zhang, J., Jalali, M., Li, C. T., and Farnia, F. Identification of novel modes in generative models via fourier-based differential clustering, 2024 a . URL https://arxiv.org/abs/2405.02700

    2024 arXiv

  85. [93]

    T., and Farnia, F

    Zhang, J., Li, C. T., and Farnia, F. An interpretable evaluation of entropy-based novelty of generative models. arXiv preprint arXiv:2402.17287, 2024 b

    2024 arXiv

  86. [94]

    Tip-adapter: Training-free adaption of clip for few-shot classification

    Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., and Li, H. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pp.\ 493--510. Springer, 2022

    2022

  87. [95]

    H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al

    Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16793--16803, 2022

    2022

  88. [96]

    C., and Liu, Z

    Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Journal of Computer Vision, 130 0 (9): 0 2337--2348, 2022

    2022

  89. [97]

    Rethinking centered kernel alignment in knowledge distillation

    Zhou, Z., Shen, Y., Shao, S., Gong, L., and Lin, S. Rethinking centered kernel alignment in knowledge distillation. arXiv preprint arXiv:2401.11824, 2024

    2024 arXiv

  90. [98]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.