{"id":"4ec0891f-413c-4c27-b20e-ed0ce4e06b0e","arxiv_id":"2412.03871","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding nearest-neighbor and cross nearest-neighbor supervision from frozen pretrained unimodal encoders to the CLIP loss improves lightweight vision-language models on zero-shot and retrieval benchmarks.","lead":"CLIP-PING is a training method that adds extra contrastive losses based on nearest neighbors from frozen pretrained image and text encoders to the standard CLIP objective, boosting small vision-language models. On small datasets it reports gains of 5.5% on ImageNet zero-shot and up to 10.7% on image-to-text retrieval over plain CLIP.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ImageNet-family gains may largely reflect the ImageNet-pretrained teacher's label knowledge rather than generic NN/XNN alignment; the missing teacher-pretraining control is the key blocker for the central claim.","rationale":"After reading the paper in good faith, I find the method plausible and the experiments extensive: the ablations on supervision source, support set size, loss weights, and the comparisons against CLIP-D/CLIP-F/CLIP-KD are appropriate controls and show that the proposed losses add signal beyond direct feature distillation. The Flickr30K zero-shot retrieval gains and many linear-eval deltas are on data the teachers were not trained on, which supports the general cross-modal alignment claim. However, the single most load-bearing issue is the ImageNet-family evaluation confound. The default image teacher explicitly has ImageNet-21k and ImageNet-1k label supervision, and the NN loss directly aligns the student's image features to that teacher's feature geometry. Since the headline contribution is zero-shot ImageNet and ImageNet robustness performance, this confound sits exactly at the central claim. The missing control is feasible (swap to an IN-21k-only checkpoint) and would settle whether the gains reflect generic alignment or teacher label knowledge. I therefore do not change the reader's CONDITIONAL verdict; the paper is conditionally acceptable, with the teacher-pretraining control and a held-out hyperparameter procedure required before the headline claims can be taken at face value.","tokens_in":26666,"tokens_out":6226,"duration_ms":62470,"concrete_test":"Train the ViT-XS + MobileBERTTINY student on COCO+CC3M exactly as in Sec. IV, but replace the image teacher with the same ResNet-v2-50 architecture using the ImageNet-21k-only checkpoint (no IN-1k fine-tuning), keeping the text teacher, alpha=0.25, lambda=0.6, and support set size fixed. If the IN-1K / IN-V2 / IN-R / IN-O / IN-S gaps over CLIP shrink by more than about half (e.g., the IN-1K gain drops from 5.5 to below ~2.8), the headline zero-shot gains are substantially teacher-pretraining leakage. As a second discriminator, report the same teacher swap on Flickr30K I2T/T2I; if those retrieval gains are preserved, the generic NN/XNN mechanism is still supported and only the ImageNet-family advertising should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline zero-shot results are ImageNet1K and the ImageNet-V2/R/O/S robustness sets, and the default image teacher is timm/resnetv2_50x1_bit.goog_in21k_ft_in1k, pretrained on ImageNet-21k and fine-tuned on ImageNet-1k (Sec. IV-A.d). The PING losses (Eqs. 4-13) explicitly pull student image features toward teacher-derived nearest-neighbor image features, so the student's visual feature geometry is directly shaped by the teacher's ImageNet-ordained similarity structure. The paper's only teacher swap is to ViT-B/16, also ImageNet-pretrained (Table XIa), so it cannot separate 'a strong unimodal teacher' from 'an ImageNet-label-trained teacher.' Without a control using an image teacher lacking ImageNet label supervision (e.g., the same ResNet-v2-50 pretrained only on ImageNet-21k, without the IN-1k fine-tune, or a self-supervised teacher), the 5.5-point IN-1K gain and larger IN-V2/IN-S gains cannot be attributed to the claimed generic-alignment mechanism. The Flickr30K retrieval gains and non-ImageNet linear-eval improvements are less exposed and provide genuine independent support, but the most emphasized claims are exactly the ones most confounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes CLIP-PING, a training procedure for lightweight vision-language models that augments the standard CLIP contrastive loss with two auxiliary contrastive terms: intra-modal nearest-neighbor (NN) supervision and inter-modal cross nearest-neighbor (XNN) supervision, using features from frozen off-the-shelf unimodal encoders (a BiT ResNet-v2-50 image teacher and BERT-Base text teacher). The final objective is L = (1-λ)L_CLIP + λ[(1-α)L_NN + α L_XNN] (Eqs. 14-15). The authors evaluate on COCO (600K pairs) and COCO+CC3M (3M pairs) with three lightweight image encoders (ViT-XS, ConvNeXt-Pico, MNv4-Hybrid-M) paired with MobileBERT-TINY. They report consistent gains over CLIP and CLIP-KD on cross-modal retrieval (Flickr30K, COCO, CC3M), zero-shot classification (ImageNet1K and four ImageNet robustness sets plus four other datasets), and linear evaluation on 12 datasets. An 'active' variant (A-CLIP-PING) that recomputes teacher features during training yields further gains.","tokens_in":26907,"tokens_out":10134,"duration_ms":93135,"significance":"The proposed mechanism is simple, plausible, and, if the reported gains reflect the mechanism's generic alignment properties, would be a useful addition to the toolkit for training lightweight vision-language models in low-resource settings. The paper is methodologically strong in its breadth: three student architectures, two training sets, multiple ablations (loss weights, support set size, top-k, projection dimension, supervision source, teacher swap), and explicit wall-clock/memory measurements. The method itself is not internally circular: the PING losses add new contrastive terms rather than reducing to a single fitted constant. The central weakness is that the headline ImageNet-family results are obtained with an ImageNet-pretrained teacher, and the most important hyperparameters were selected on the same benchmarks that are later reported as results. These two issues prevent the paper from currently supporting its strongest claims.","major_comments":[{"comment":"The default image teacher is timm/resnetv2_50x1_bit.goog_in21k_ft_in1k, a BiT ResNet-v2-50 pretrained on ImageNet-21k and fine-tuned on ImageNet-1k, and the only teacher swap in Table XIa and Tables XXII-XXIII is to ViT-B/16, which is also ImageNet-pretrained. Because the PING losses in Eqs. (4)-(13) directly shape the student's visual feature geometry toward teacher-derived nearest-neighbor structure, the large gains reported on ImageNet1K and the ImageNet robustness sets in Tables V and VI (5.5% on IN-1K, 5.1% on IN-V2, 5.6% on IN-R, 5.6% on IN-O, 3.9% on IN-S) may substantially reflect transfer of the teacher's ImageNet label knowledge rather than generic cross-modal alignment. Please add a control experiment with an image teacher that has not been trained on ImageNet labels (e.g., a self-supervised DINO teacher or the same BiT model without the ImageNet-1k fine-tune) and report the ImageNet-family zero-shot results with that teacher; if the gains shrink materially, the claims in the abstract should be revised to focus on the non-ImageNet benchmarks.","section":"Sec. IV-A-d, Tables V-VI"},{"comment":"The hyperparameters α, λ, support-set size |Q|, top-k, and projection dimension d are all selected by optimizing Flickr30K retrieval and/or the average zero-shot accuracy across the same four datasets (STL, C10, C100, SA-40) that are later reported as the main results (Tables IXa, IXb, Xa, Xb, and XX; see also the default settings in Sec. IV-A-e). For example, α=0.25 and λ=0.6 are chosen because they achieve the best values in Table IX, and |Q|=32768 is chosen because it is best in Table Xa. This is a test-set selection problem: the headline numbers in Tables III-VII are not an unbiased estimate of the method's generalization ability. Please tune on a held-out validation set (e.g., a split of COCO captions or a separate validation split of CC3M) and report the resulting settings, or present the main results for a range of hyperparameter values to show that the conclusions are not sensitive to the specific choices.","section":"Sec. IV-C and VIII-C (Tables IX, X, XX)"},{"comment":"The comparison with DeCLIP is not on equal footing: DeCLIP is trained at batch size 512 while CLIP-PING uses 1024, and the DeCLIP learning rates are modified because of reported instability. Since DeCLIP's contrastive objective depends on in-batch negatives, this asymmetric setup systematically disadvantages DeCLIP. Given that DeCLIP is the most closely related prior work on neighbor-supervision for CLIP training, please retrain DeCLIP at batch size 1024 with its original hyperparameters, or state explicitly that the reported DeCLIP results are obtained under a different training budget and are not directly comparable.","section":"Table XIII"}],"minor_comments":[{"comment":"The teacher is described as 'ResNet-v2-50', but the timm checkpoint name indicates a BiT model pretrained on ImageNet-21k with fine-tuning on ImageNet-1k; please make this explicit in the text.","section":"Sec. IV-A-d"},{"comment":"The column headers in Table II are incomplete; 'Feat.dim' and 'Size(GB)' are not clearly labeled as columns, and the ViT-B/16 dimension of 21952 requires explanation.","section":"Table II"},{"comment":"Line 9 uses '˜zI_k ← B∗_I (Ik)', which treats the feature bank as a function; please use clearer notation such as 'retrieve the stored feature for Ik from B∗_I'.","section":"Algorithm 1"},{"comment":"The text refers to 'selecting one random neighbor from the top-k NN', but the method section defines NN(·) only as the single nearest neighbor; please clarify the random selection procedure.","section":"Sec. IV-C-f"},{"comment":"The ViT-B/16 ablation table omits the plain CLIP baseline; please include it for completeness.","section":"Table XIa"},{"comment":"The statement that CLIP-PING is 'as efficient as the original CLIP [21]' cites CLIP-CID rather than the original CLIP paper; the citation should be [2].","section":"Sec. IV-B-b"},{"comment":"The caption misspells the text encoder as 'MoblileBERT TINY'; the correct name is MobileBERT-TINY.","section":"Fig. 5 caption"},{"comment":"Reference [1] is a medical-imaging contrastive-learning paper, but it is cited together with [44] for InfoNCE; the InfoNCE citation should point to [44] only.","section":"Sec. III-B-a"},{"comment":"The claim of gains 'without extra computational costs' should be qualified: the method requires one-time feature extraction and disk storage for the feature banks (Table II, up to 22.1 GB for the 3M dataset), and A-CLIP-PING requires active teacher computation.","section":"Abstract / Sec. I"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical study with a clear and honest presentation. The two main concerns are (i) the ImageNet-pretrained teacher confounds the headline ImageNet-family results and (ii) the hyperparameters are selected on the same benchmarks reported as results. Both are addressable with additional experiments. I encourage the editor to ask for a code or data availability statement, as none is currently provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on CLIP-PING (arXiv:2412.03871). The method is a neat, cheap add-on to small CLIP training: freeze features from off-the-shelf unimodal encoders, store them in FIFO support sets, and add two extra contrastive losses that pull each sample toward its nearest neighbor (NN) and cross nearest-neighbor (XNN) in the teacher's feature space. The combination of intra-modal NN and cross-modal XNN supervision from frozen unimodal teachers is not in DeCLIP, NNCLR, or CLIP-F; it's incremental but genuinely new. The experiments are extensive and consistent: three lightweight backbones, zero-shot, retrieval, linear eval, plus ablations on alpha, lambda, support set size, top-k, projection dimension, teacher swap, and supervision source. The efficiency numbers (memory, wall-clock) show the method adds minimal overhead over plain CLIP. That is a solid engineering contribution.\n\nThe soft spots, in order. First, the teacher leakage problem is real and load-bearing for the headline numbers. The default image teacher is ResNet-v2-50 pre-trained on ImageNet-21k and fine-tuned on ImageNet-1k. The PING losses literally pull the student's feature geometry toward that teacher's neighbor structure, so the strong gains on ImageNet1K and the ImageNet robustness sets (IN-V2/R/O/S) may largely reflect transferring ImageNet label knowledge, not generic alignment. The teacher swap to ViT-B/16, also ImageNet-pretrained, does not control for this. The Flickr30K retrieval gains and the non-ImageNet linear eval improvements (Pets, Caltech, Flowers, etc.) are less exposed and give independent support, but the most emphasized claims are exactly the most confounded. The authors should run a control with a teacher not trained on ImageNet labels (e.g., the same ResNet without IN-1k fine-tune, or a self-supervised teacher). Second, alpha, lambda, |Q|, top-k, and projection dimension are all tuned on Flickr30K retrieval and the zero-shot AVG across four datasets, which are also the reported benchmarks. That is a selection effect; a held-out validation split would fix it. Third, no error bars, no code, single seed. Fourth, the DeCLIP comparison is uneven: DeCLIP is trained with half batch size and adjusted LR, then reported as suboptimal. That undersells the baseline.\n\nThese are addressable rather than fatal. The method itself is not circular—the PING losses are genuine additional supervision—and the consistency of gains across architectures suggests the effect is real, just not yet cleanly attributed.\n\nI'd send this to review, with the request that the authors add the teacher-pretraining control and a proper validation split. The paper is useful for anyone working on efficient VLM training or evaluating teacher-student methods. If I were in that area, I'd cite it for the method and for the cautionary tale about teacher leakage.","headline":"A cheap, well-tested NN/XNN supervision trick for small CLIP models, but the headline ImageNet gains are confounded by an ImageNet-pretrained teacher.","tokens_in":27501,"tokens_out":2730,"would_cite":true,"duration_ms":108119,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP-PING shows that lightweight vision-language models gain substantially when their usual image-text contrastive loss is supplemented with nearest-neighbor and cross nearest-neighbor contrastive targets derived from frozen features of…","keywords":["contrastive language-image pre-training","lightweight vision-language models","nearest-neighbor supervision","cross-modal retrieval","zero-shot classification","frozen feature banks","efficient training","knowledge distillation"],"falsifier":"Train CLIP-PING with the same NN/XNN losses but replace the image teacher with a self-supervised vision model that never saw ImageNet labels (or a randomly initialized encoder), and compare ImageNet-1K zero-shot accuracy. If the 5.5-point gain persists, the effect is alignment-driven; if it largely disappears on ImageNet but stays on COCO retrieval, the teacher's ImageNet label knowledge was the driver.","tokens_in":26440,"feed_emoji":"🧲","tokens_out":6202,"duration_ms":56312,"temperature":0.7,"pith_summary":"CLIP-PING claims that a lightweight vision-language model trained with the standard image-text contrastive loss can be substantially improved by adding two extra contrastive losses that pull each sample toward its nearest neighbors in frozen feature banks produced by off-the-shelf unimodal encoders. The paper argues that this \"Proximus Intrinsic Neighbors Guidance\" gives the student richer semantic supervision than the single paired image-text objective, without the architectural constraints or repeated teacher forward passes of explicit distillation. With a small ViT-XS image encoder and MobileBERT text encoder trained on about 3 million COCO+CC3M pairs, the method reports 22.7% zero-shot ImageNet-1K top-1 accuracy and 49.7%/35.4% Flickr30K I2T/T2I Recall@1, beating standard CLIP by 5.5, 10.7, and 5.7 points respectively at nearly the same memory and per-epoch time. A variant that keeps the teachers active during training, A-CLIP-PING, gives further gains at the same cost as CLIP-KD distillation.","feed_headline":"Neighbor guidance lifts lightweight CLIP by up to 10.7 points","feed_subtitle":"A near-free training tweak uses frozen encoder features to add contrastive targets, boosting zero-shot and retrieval at CLIP's cost.","key_machinery":"The mechanism is the Proximus Intrinsic Neighbors (PING) objective built on two frozen feature banks: image features $\\tilde{z}^I_k = F^*_I(I_k)$ and text features $\\tilde{z}^T_k = F^*_T(T_k)$ are extracted once from off-the-shelf encoders and stored, and two FIFO support sets $Q_I$, $Q_T$ (default size 32768) serve for retrieval. Nearest-neighbor supervision applies InfoNCE-style losses between a student embedding and $\\mathrm{NN}(\\tilde{z})$, the closest bank feature in the same modality; cross nearest-neighbor supervision instead uses the partner of the closest feature in the other modality, $\\mathrm{XNN}(\\tilde{z}^I_k) = \\tilde{z}^I_i$ where $\\tilde{z}^T_i = \\mathrm{NN}(\\tilde{z}^T_k)$. The full loss is $L = (1-\\lambda) L_{\\text{CLIP}} + \\lambda [(1-\\alpha)L_{\\text{NN}}^{\\text{PING}} + \\alpha L_{\\text{XNN}}^{\\text{PING}}]$ with $\\alpha=0.25$, $\\lambda=0.6$ chosen by ablation. A linear adapter maps frozen features to the student's projection dimension when the dimensions differ.","core_discovery":"The central discovery, stated on the paper's own terms, is that frozen unimodal encoders can act as \"intrinsic\" teachers without any explicit distillation loss: their precomputed features, stored in auxiliary banks, define nearest-neighbor (NN) and cross nearest-neighbor (XNN) targets that the student learns to align to. Intra-modal NN supervision pulls an image or text embedding toward the feature-bank neighbor the teacher considers most similar within that modality, while inter-modal XNN supervision cross-references the modalities — the image whose paired text is the text neighbor of the query — to encourage indirect alignment of semantically related pairs. Adding these losses with weight λ to the standard CLIP objective consistently improves zero-shot classification, cross-modal retrieval, and linear-transfer accuracy across three lightweight image encoders (a small ViT, a compact ConvNeXt, and a hybrid MobileNet-v4 variant), with reported gains growing on the 3M-pair dataset. The authors interpret the result as evidence that the single image-text pair signal under-supervises small models, and that cheap neighbor-based contrastive supervision from frozen teacher features fills that gap.","pith_inferences":["Because the NN/XNN losses never require the teacher's logits, the recipe transfers to any frozen unimodal encoder, not just CLIP-style ones; a natural test is whether self-supervised visual teachers (no class labels) give the same ImageNet boost, which would separate alignment gains from label leakage.","The same two-stage frozen-bank mechanism could be applied to other paired modalities, such as audio-text or video-audio, where a lightweight student needs cross-modal alignment without running large teachers online.","The teacher here is ResNet-v2-50 pretrained on ImageNet-21k/1k, so part of the reported ImageNet-family improvement may come from the teacher's class knowledge flowing through neighbor targets; evaluating on a non-ImageNet benchmark not seen by the teacher would quantify this.","The support-set sweet spot (gains up to 32768, then decline) suggests neighbor diversity helps only up to the point where retrieved neighbors stay semantically relevant, implying the method's benefit depends on dataset redundancy."],"forward_implications":["Across all three lightweight architectures tested, CLIP-PING beats CLIP and CLIP-KD on COCO and COCO+CC3M pretraining for cross-modal retrieval and zero-shot classification.","On the 3M-pair setting with ViT-XS, ImageNet-1K zero-shot top-1 rises to 22.7%, and ImageNet distribution-shift sets (IN-V2, IN-R, IN-O, IN-S) all improve over CLIP and CLIP-KD.","The method adds only a few MiB of memory and negligible per-epoch time compared with plain CLIP, because teacher features are precomputed once rather than recomputed each batch.","A-CLIP-PING, which runs the teachers actively, yields additional gains (1.4, 2.9, and 1.8 points over CLIP-PING for ViT-XS on ImageNet, Flickr30K I2T, and T2I), confirming that stronger teacher guidance is the source of the signal.","Ablations show both NN and XNN supervision matter, the optimal support size is 32768, top-1 neighbor selection beats larger k, and a projection dimension of 256 balances cost and accuracy."],"supporting_citations":[{"why":"Defines the standard image-text contrastive objective that CLIP-PING builds on and serves as the primary baseline.","marker":"[2]"},{"why":"Introduces cross-modal nearest-neighbor supervision that motivates the XNN inter-modal guidance.","marker":"[9]"},{"why":"Provides the COCO training pairs used to pretrain the evaluated lightweight models.","marker":"[13]"},{"why":"Provides the Conceptual Captions 3M pairs combined with COCO for the 3M-pair setting.","marker":"[14]"},{"why":"Supplies the CLIP-KD distillation baseline that CLIP-PING is compared against and that A-CLIP-PING matches in cost.","marker":"[18]"},{"why":"Supplies the nearest-neighbor contrastive learning mechanism that the intra-modal NN loss adapts.","marker":"[25]"},{"why":"Provides the ImageNet-1K evaluation benchmark where the 22.7% zero-shot top-1 result is reported.","marker":"[27]"},{"why":"Supplies the Big Transfer pretraining that initializes the ResNet-v2-50 image feature extractor used as teacher.","marker":"[50]"},{"why":"Supplies the BERT-Base text encoder used as frozen text feature extractor.","marker":"[52]"}],"fun_headline_variants":["Neighbor targets from frozen encoders boost lightweight CLIP","CLIP-PING lifts small VLMs up to 10.7 points","Frozen neighbor supervision sharpens CLIP for free","Intrinsic neighbor guidance pre-trains lighter CLIP","Boost lightweight CLIP with neighbor contrastive loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ImageNet-pretrained teacher's features do not secretly carry ImageNet class knowledge into the student through the neighbor losses — in other words, the gains come from better multimodal alignment, not from the teacher already knowing the test classes.","fun_headline_variants_meta":{"raw":{"variants":["Neighbor targets from frozen encoders boost lightweight CLIP","CLIP-PING lifts small VLMs up to 10.7 points","Frozen neighbor supervision sharpens CLIP for free","Intrinsic neighbor guidance pre-trains lighter CLIP","Boost lightweight CLIP with neighbor contrastive loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1534,"prompt_tokens":1076,"completion_tokens":458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":387}},"tokens_in":692,"tokens_out":458,"duration_ms":4723,"temperature":1.0,"reasoning_tokens":387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:59:11.573516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CLIP-PING with the same NN/XNN losses but replace the image teacher with a self-supervised vision model that never saw ImageNet labels (or a randomly initialized encoder), and compare ImageNet-1K zero-shot accuracy. If the 5.5-point gain persists, the effect is alignment-driven; if it largely disappears on ImageNet but stays on COCO retrieval, the teacher's ImageNet label knowledge was the driver.","supporting_citations":[{"cited_title":"Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm,","cited_arxiv_id":null,"evidence_quote":"Introduces cross-modal nearest-neighbor supervision that motivates the XNN inter-modal guidance."},{"cited_title":"Big transfer (bit): General visual representation learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the Big Transfer pretraining that initializes the ResNet-v2-50 image feature extractor used as teacher."}],"review_version":1}