{"id":"7a9177ac-6a29-4cfb-8f5d-d0939c2141f8","arxiv_id":"2506.02557","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning CLIP's visual encoder to match DINOv2's kernel-based similarity structure improves its fine-grained visual perception while preserving its alignment to text.","lead":"This paper fine-tunes CLIP's image encoder so that its pairwise similarity patterns match those of the vision model DINOv2, while keeping the text encoder untouched. The result is a lightweight upgrade that modestly improves CLIP's fine-grained perception and carries over to downstream vision-language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The compatibility claim rests on a soft regularization whose premise is never verified: Proposition 3.2 requires a per-sample L2 bound, while Eq. (6) only penalizes expected squared displacement, so the abstract's 'retains compatibility' is not established.","rationale":"The paper's strongest claim has two prongs: the aligned encoder improves fine-grained perception, and it retains compatibility with the frozen text encoder and downstream models. The improvement prong is supported by multiple benchmark tables, albeit with modest gains. The compatibility prong is supposedly guaranteed by Proposition 3.2, but the proof and the training objective do not align: the proposition is conditional on a hard per-sample displacement bound, while Eq. (6) minimizes an expected squared penalty. This is the load-bearing gap because if the displacement tail is large, the method could improve average retrieval or classification while breaking text alignment on a meaningful subset of inputs. The reader's weakest_assumption identifies exactly this issue, and this stress-test agrees. A separate confound—that fine-tuning on ImageNet-1K itself, rather than the DINOv2 kernel, drives the gains—was also considered, but the unverified premise of Proposition 3.2 is the more direct logical gap in the central claim. The paper is already CONDITIONAL, and the requested measurement is a reasonable condition to add; no verdict change is needed unless the measurement reveals a large displacement tail, in which case the compatibility claim would be unsupported.","tokens_in":24497,"tokens_out":10390,"duration_ms":116481,"concrete_test":"On a held-out set (e.g., COCO val or ImageNet val), compute per-image d(I) = ||fθ(I) − fθ0(I)||₂ and the cosine difference |cos(fθ0(I), g(T)) − cos(fθ(I), g(T))| for the paired text, using the exact Eq. (6) model. Report the 95th/99th percentile and maximum of d(I), and the share of images where the Proposition 3.2 bound would require λ > 0.05 to keep the cosine change under 0.1. If the tail is small, the compatibility concern is resolved; if not, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 3.2 is the only formal support for the paper's central compatibility claim, but its premise is never established. Eq. (6) penalizes the expected squared L2 displacement on ImageNet training images, whereas the proposition requires a hard per-sample bound ||fθ(I) − fθ0(I)||₂ ≤ λ for every language-image pair. Nothing in the optimization enforces this bound; a soft penalty can leave a heavy tail of images with large displacement, and those are precisely the fine-grained inputs the method claims to improve. The reported retrieval and MLLM numbers are aggregate and could hide systematic failures on hard examples. Without reporting the realized distribution of L2 displacements and the corresponding cosine changes against text embeddings, the abstract's claim that the encoder 'retains compatibility' is not demonstrated. Section 3.3 says the regularization 'ensures' this, but it only encourages it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KUEA, a kernel-based unsupervised fine-tuning method that aligns CLIP's visual encoder with DINOv2's embeddings. The training objective (Eq. 6) minimizes the squared difference between a polynomial kernel evaluated on CLIP image embeddings and the same kernel evaluated on DINOv2 embeddings, plus an L2 regularization term that penalizes deviation from the original CLIP embeddings. The method is image-only, keeps the text encoder frozen, and is evaluated on zero-shot classification, image-text retrieval, counting/spatial reasoning, localization probing, and two MLLMs (LLaVA and OpenFlamingo). The authors report average zero-shot accuracy gains of 0.8–1.3 percentage points across CLIP variants, consistent gains in retrieval, and improvements in downstream MLLMs without retuning the LLM. They provide ablations on training epochs, regularization weight, kernel choice, and training-data size, as well as comparisons with DIVA, AM-RADIO, and Additive-MoF. The central claim is that a lightweight, image-only alignment fine-tune improves fine-grained visual perception while retaining compatibility with the frozen text encoder and downstream models.","tokens_in":24678,"tokens_out":4901,"duration_ms":50082,"significance":"If the central claim holds, the method offers a practical recipe for upgrading CLIP-style visual encoders at a fraction of the cost of retraining, with immediate plug-in compatibility for existing MLLMs. The paper's strengths are its breadth of evaluation (12 zero-shot datasets, retrieval, localization probing, two MLLMs), multiple CLIP backbone scales, ablations of key design choices, and the release of code and models. The idea of aligning kernel matrices rather than raw embeddings is simple and plausible, and the empirical gains, though modest, are consistent across most settings. The main weaknesses are that the formal compatibility guarantee is not actually implied by the optimization objective, and the trainable parameters of the CLIP-side kernel introduce a potential confound that is not controlled in the experiments.","major_comments":[{"comment":"Proposition 3.2 bounds the change in cosine similarity to text embeddings under a hard per-sample bound ||fθ(I) − fθ0(I)||₂ ≤ λ. However, Eq. (6) only penalizes the expected squared L2 displacement over the training distribution; nothing in the optimization enforces the pointwise bound. The text after Eq. (6) states that the regularization term 'ensures' that alignment does not cause large deviations, but it only encourages this. Consequently, the abstract's claim that the encoder 'retains compatibility with the frozen text encoder' is not formally supported by Proposition 3.2. Please either (a) verify the premise empirically by reporting the realized distribution of per-sample L2 displacements and the resulting cosine changes against text embeddings (e.g., percentiles on ImageNet or a held-out set), or (b) soften the guarantee language to state that compatibility is empirically preserved. As written, the theory does not establish the central compatibility claim.","section":"§3.3, Eq. (6) and Proposition 3.2"},{"comment":"The paper states that for the CLIP branch, the polynomial kernel hyperparameters γ and c are set as trainable, while the DINOv2-side kernel has fixed γ = 1/dim_emb and c = 1. This means the alignment loss in Eq. (4) can be reduced by adapting the kernel function itself, rather than by moving the visual embeddings toward a DINOv2-like geometry. Since all zero-shot evaluations use standard cosine similarity (not the trained kernel), the mechanism behind the reported improvements is ambiguous. Please report the learned values of γ and c, and add an ablation that fixes the CLIP kernel parameters (e.g., to the same values as the DINOv2 side). If the improvements persist with fixed kernel parameters, the attribution of the gain to the visual encoder is much better supported.","section":"§4.1, Eq. (4) and Table 7"},{"comment":"The ViT-B-16 SVHN zero-shot accuracy drops from 31.31 to 27.40 after alignment, a 3.91-point decrease on a digit-recognition task that falls squarely within the paper's fine-grained perception focus. The text reports only averaged gains (e.g., 'alignment improves the average accuracy by 1.58% for ViT-L-14'), which can mask such degradation. Please report per-dataset results with error bars or standard deviations across multiple runs, and explicitly discuss which benchmarks degrade and why. The current presentation substantially overstates the consistency of the improvement.","section":"Table 3, ViT-B-16 SVHN row"}],"minor_comments":[{"comment":"The sentence 'The regularization term ensures that the alignment process does not cause the aligned embeddings to deviate significantly from the original embeddings' should be reworded to 'encourages', since Eq. (6) is a soft penalty, not a constraint.","section":"§3.3"},{"comment":"The dataset name 'Flicker30K' is misspelled; it should be 'Flickr30K' throughout.","section":"Table 2 and §4.2"},{"comment":"Proposition 3.2 is attributed to Schlarmann et al. (2024) but the proof is said to be 'adapted'; please provide a precise citation to the original statement and verify that the bound is stated identically, or present the result as a new proposition if the adaptation is nontrivial.","section":"Appendix A.2"},{"comment":"Figure 3 in the main text and Figure 5 in Appendix C.7 appear to be the same ablation panels; please consolidate them to avoid duplication.","section":"Figures 3 and 5"},{"comment":"The phrase 'w/wo alignment' is informal; consider using 'with and without alignment' in the tables and text.","section":"§1"},{"comment":"The notation for the polynomial kernel in Eq. (1) uses k_polynomial(γ,c,d)(x,y); in Eq. (2) the normalization denotes the kernel by k̃, but the tilde may be hard to read in print. Consider using a more distinct symbol such as k̂.","section":"§3.1"},{"comment":"The Related Work section contains a large number of self-citations to kernel-method papers; while not inappropriate, the relevance of several of them (e.g., the Vendi-score diversity papers) to CLIP alignment is indirect, and trimming would improve readability.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for ICML and the empirical study is extensive. My main concerns are that the theoretical compatibility guarantee is disconnected from the actual optimization objective, and that the trainable CLIP kernel parameters could absorb the alignment signal. Both are addressable with additional experiments (L2 displacement distribution, fixed kernel parameters). I do not see evidence of a load-bearing error that would require rejection, but the central claim in the abstract is currently stronger than the paper's formal and empirical support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious referee, but the 'retains compatibility' claim needs to be walked back or verified. The kernel-alignment idea is new to me: aligning the polynomial kernel matrix of CLIP's visual embeddings with DINOv2's, plus an L2 feature-space regularizer, all on image-only data, with no text encoder update. The experiments back up the modest claims: consistent zero-shot gains of about a point, better gains on fine-grained tasks after linear probing, and nice transfer to LLaVA and OpenFlamingo. I appreciate the ablations showing the regularizer is crucial and the method works across other VLMs (SigLIP, DFN, MetaCLIP) and targets (MLCD). The training cost is sensible. Related-work coverage is solid; the self-citations are to relevant kernel papers, not padding.\n\nThe soft spots are real. First, Proposition 3.2 needs a hard per-sample L2 bound, but the loss in Eq. (6) only penalizes expected squared deviation. Nothing stops a heavy tail of images from moving far in feature space, and those are exactly the fine-grained cases the method claims to help. The retrieval numbers are aggregate and don't rule that out. I'd want the realized distribution of L2 displacements and cosine changes against text embeddings, especially on the hard examples. Second, calling 0.8-1.3 average points 'significant' is a stretch, and ViT-B SVHN zero-shot drops from 31.31 to 27.40, which is a real regression. Third, Prop. 3.1 is introduced via Hoeffding but proved with a vector Bernstein bound, and the 'unbiased estimator' sentence is imprecise. Minor, but clean it up.\n\nOverall, the central idea holds up and the empirical core looks honest. The compatibility guarantee is the weakest link. That's fixable with extra measurements and calmer language. Yes, send it to review.","headline":"A solid, cheap CLIP fine-tuning recipe with a real compatibility gap between the guarantee and the loss.","tokens_in":25227,"tokens_out":4516,"would_cite":true,"duration_ms":37639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By fine-tuning only CLIP's image encoder on images, guided by DINOv2's pairwise similarity kernel, the paper claims CLIP gains the fine-grained perception it lacks while remaining compatible with its frozen text encoder.","keywords":["kernel-based alignment","CLIP","DINOv2","fine-grained perception","zero-shot classification","vision-language models","embedding regularization","visual representation learning"],"falsifier":"On a held-out set, record the per-image L2 distance between original and aligned embeddings and the resulting change in cosine similarity to a fixed set of text embeddings; if a nontrivial fraction of images moves far enough that the Proposition 3.2 bound no longer confines the cosine change, and retrieval or zero-shot accuracy drops on those images, the compatibility claim fails. A simpler version: ablate the regularization by setting its weight to zero, as the authors do, and show that the aligned encoder's text-image retrieval collapses.","tokens_in":24277,"feed_emoji":"👁️","tokens_out":6236,"duration_ms":54689,"temperature":0.7,"pith_summary":"CLIP's visual encoder is trained against captions and therefore sees images coarsely: it confuses images that share semantics but differ in color, count, orientation, or spatial layout. The paper tries to fix that without any text data or retraining of the text branch by fine-tuning the image encoder so that its pairwise similarities among images, measured by a polynomial kernel, match the pairwise similarities produced by DINOv2, a self-supervised model known for fine detail. An L2 penalty keeps each new embedding near its original position, and the authors prove a bound on how much cosine similarity to text embeddings can change as a function of that movement. On CLIP-Benchmark, probing, and VQA evaluations they report consistent gains in zero-shot recognition, counting and spatial reasoning, localization, and downstream LLaVA and OpenFlamingo performance. If the claim holds, cheap image-only retuning can upgrade CLIP and compatible MLLMs without the costly realignment most visual-enhancement methods require.","feed_headline":"Image-only retune sharpens CLIP's eye without breaking its text link","feed_subtitle":"Matching CLIP's pairwise similarities to DINOv2 lifts zero-shot accuracy, spatial reasoning, and downstream VQA.","key_machinery":"The central object is the normalized polynomial kernel $k(x,y)=\\frac{(\\gamma x^\\top y + c)^d}{\\sqrt{k(x,x)k(y,y)}}$ evaluated on image embeddings, paired with the alignment objective $\\min_\\theta\\, w\\,\\mathbb{E}[(k_1(f_\\theta(I_i),f_\\theta(I_j))-k_2(g(I_i),g(I_j)))^2]+\\mathbb{E}[\\|f_\\theta(I_i)-f_{\\theta_0}(I_i)\\|_2^2]$. The kernel defines what \"similar\" means in each model's space, so matching the two kernel matrices transfers DINOv2's fine-grained similarity geometry to CLIP, while the L2 term anchors the new embeddings to their old positions to preserve text cosine similarities, per the bound in Proposition 3.2.","core_discovery":"The paper's central claim is that two embedding spaces can be reconciled in kernel space rather than feature space: instead of mapping DINOv2 vectors onto CLIP's vectors with a linear layer, the authors minimize the expected squared difference between the CLIP image-encoder's kernel function evaluated on pairs and DINOv2's kernel on the same pairs. Because the kernel records relative similarity between samples, this reshapes CLIP's neighborhoods to respect fine-grained visual distinctions while leaving the global structure of CLIP's space intact. The added regularization term ties the updated embedding to the original one, and Proposition 3.2 gives a bound on the change in cosine similarity with any text embedding as a function of that L2 deviation; the authors treat this as the mechanism that keeps zero-shot, retrieval, and downstream MLLM compatibility alive. They report that the aligned encoder improves average zero-shot accuracy on 12 datasets for three CLIP variants, improves local and global probing recall, and when swapped into LLaVA and OpenFlamingo improves VQA benchmarks both with and without LLM fine-tuning.","pith_inferences":["Beyond the paper: because the alignment never touches the text encoder, the same procedure could plausibly be re-run on the aligned model or stacked with other CLIP upgrades; the authors only hint at orthogonality with DIVA.","Beyond the paper: a testable extension is to scale the image-only fine-tuning data from ImageNet-1K toward web-scale pools; the authors' own data-size ablation shows gains grow with more data, suggesting further headroom.","Beyond the paper: the reliance on pairwise kernel matching suggests a general recipe—any target encoder that excels on a missing perceptual axis can donate its similarity geometry to a frozen multimodal encoder, provided a regularization anchor holds.","Beyond the paper: if the cosine-preservation bound is the load-bearing piece, monitoring realized L2 drift during training could serve as a simple diagnostic for when alignment is about to break text compatibility."],"forward_implications":["Zero-shot classification on CLIP-Benchmark improves across three CLIP backbones, with the largest gains on low-resolution and small-object datasets such as CIFAR and EuroSAT.","Counting, spatial reasoning, and caption-recognition tasks (SVHN, GTSRB, CLEVR) improve under both zero-shot and linear-probe evaluation, and local and global localization probing recall rises.","The aligned visual encoder can replace the original in LLaVA and OpenFlamingo without retraining the LLM, and LoRA fine-tuning of LLaVA amplifies the gains, especially on referring-expression benchmarks.","The alignment recipe transfers to other VLM text encoders (SigLIP, DFN, MetaCLIP) and other vision target models (MLCD), so it is not specific to CLIP-DINOv2."],"supporting_citations":[{"why":"Defines the CLIP image and text encoders whose compatibility the paper preserves.","marker":"(Radford et al., 2021)"},{"why":"Provides DINOv2, the vision-centric target model whose kernel matrix supplies fine-grained similarity supervision.","marker":"(Oquab et al., 2024)"},{"why":"Source of the cosine-change bound in Proposition 3.2 that justifies the L2 regularization term.","marker":"(Schlarmann et al., 2024)"},{"why":"Documents CLIP's failures on visual patterns and supplies the MMVP benchmarks used to test fine-grained perception.","marker":"(Tong et al., 2024)"},{"why":"The masked fine-tuning baseline and the local/global probing benchmark used to evaluate localization.","marker":"(Covert et al., 2025)"},{"why":"DIVA, the closest prior image-only CLIP enhancement method, which the kernel alignment compares against.","marker":"(Wang et al., 2025)"},{"why":"Provides LLaVA, the downstream MLLM whose visual encoder is swapped and evaluated.","marker":"(Liu et al., 2024)"},{"why":"Provides OpenFlamingo, the second downstream MLLM used to test transfer without LLM fine-tuning.","marker":"(Awadalla et al., 2023)"}],"fun_headline_variants":["Kernel trick aligns CLIP with DINOv2 for sharper vision","CLIP gets fine-grained eyes via kernel-space matching to DINOv2","Unsupervised kernel fix sharpens CLIP without touching text","Align CLIP to DINOv2 in kernel space to boost perception","No text needed: Kernel alignment gives CLIP a sharper view"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme rests on the hope that a modest penalty term keeps the retuned image embeddings close enough to the original ones that their cosine matches to text stay intact; training never enforces a hard cap on that distance.","fun_headline_variants_meta":{"raw":{"variants":["Kernel trick aligns CLIP with DINOv2 for sharper vision","CLIP gets fine-grained eyes via kernel-space matching to DINOv2","Unsupervised kernel fix sharpens CLIP without touching text","Align CLIP to DINOv2 in kernel space to boost perception","No text needed: Kernel alignment gives CLIP a sharper view"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000526,"raw_usage":{"total_tokens":2536,"prompt_tokens":940,"completion_tokens":1596,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":1503}},"tokens_in":556,"tokens_out":1596,"duration_ms":12202,"temperature":1.0,"reasoning_tokens":1503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:21:09.776591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set, record the per-image L2 distance between original and aligned embeddings and the resulting change in cosine similarity to a fixed set of text embeddings; if a nontrivial fraction of images moves far enough that the Proposition 3.2 bound no longer confines the cosine change, and retrieval or zero-shot accuracy drops on those images, the compatibility claim fails. A simpler version: ablate the regularization by setting its weight to zero, as the authors do, and show that the aligned encoder's text-image retrieval collapses.","supporting_citations":[{"cited_title":"Dinov2: Learning robust visual features without supervision","cited_arxiv_id":null,"evidence_quote":"Provides DINOv2, the vision-centric target model whose kernel matrix supplies fine-grained similarity supervision."},{"cited_title":"Diffusion feedback helps clip see better","cited_arxiv_id":null,"evidence_quote":"DIVA, the closest prior image-only CLIP enhancement method, which the kernel alignment compares against."}],"review_version":1}