{"id":"a1f2582c-74b6-486c-8883-9331a95ed850","arxiv_id":"2411.19346","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NoLA combines LLM class descriptions, DINO feature alignment, and visual prompt tuning to improve CLIP zero-shot classification without labels, averaging 3.6% over LaFTer on 11 datasets.","lead":"NoLA is a label-free method that tunes CLIP's classifier using unlabeled images, LLM-generated class descriptions, and DINO's visual features. It reports a 3.6% average accuracy gain over the prior label-free method LaFTer across 11 image classification datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported label-free gains may be inflated: the supplementary selects the k hyperparameter using test-set accuracy, which leaks target labels into model selection.","rationale":"I read the paper in good faith. The method is a coherent three-stage pipeline, the ablations are internally consistent, and the code and model links are a useful resource. The reader's weakest_assumption is about whether the alignment module h can map DINO features into CLIP's joint embedding space well enough to serve as a reliable teacher. That is a plausible concern, but the framework could still work even if the mapping is imperfect, because the final objective only needs h(gs(X0)) to be a better-than-CLIP soft label for prompt tuning. The more load-bearing issue is the selection of the hyperparameter k in the supplementary. The paper's label-free claim requires that no test information be used at any stage, including hyperparameter selection. The supplementary's Figure 4 plots DL-network top-1 accuracy against different k percentages; if those accuracies are computed on the test split, then the 20% rule is tuned to the test sets of the same datasets used in Table 1. That is a direct threat to the central 'label-free' claim and to the magnitude of the reported gain, because a test-tuned k will systematically favor the method in the reported comparison. This concern also interacts with the absence of error bars and multi-seed results, but the k-leakage is the primary structural issue. A validation-based or fixed-k rerun would settle whether the claim survives without test information. Since the reader already issued CONDITIONAL, this concern reinforces rather than changes the verdict; it should remain CONDITIONAL pending a label-free k-selection experiment.","tokens_in":17757,"tokens_out":3400,"duration_ms":31692,"concrete_test":"Re-run NoLA on all 11 datasets with k chosen without access to test labels: fix k=16 following prior work, or select k using a held-out portion of the unlabeled training set with a CDE-confidence criterion, then report final test accuracy in the same manner as Table 1. If the average gain over LaFTer drops below the reported 3.6%, or NoLA no longer achieves state-of-the-art on 9 of 11 datasets, the headline claim is not established as label-free.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a 3.6% average absolute gain over LaFTer in a label-free setting depends on the choice of k, the number of confident pseudo-labels per class used to train the alignment module h. The main text states that k is set to 20% of the average number of images per class, and that this 20% choice is 'justified by empirical analysis, which we present in the supplementary material.' The supplementary section 'Analysis on k selection' reports 'Top-1 Accuracy of trained DL network with different values for k' on ImageNet, CIFAR-100, UCF101, and Caltech101. This appears to be test-set accuracy of the DL network, and the 20% rule is selected from those curves. If that reading is correct, test labels are being used to select a hyperparameter of a supposedly label-free pipeline, and the final Table 1 numbers and the 9/11 state-of-the-art count are produced with this test-selected k. No validation-based or unsupervised criterion for choosing k is provided, and no error bars or multi-seed results are reported. Selecting k on test accuracy can only improve the reported numbers; it makes the 3.6% average gain unverifiable as a label-free result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NoLA, a label-free prompt-tuning method for CLIP that leverages LLM-generated class descriptions and DINO's self-supervised visual features. The pipeline has three stages: (i) a CDE classifier built from averaged CLIP text embeddings of LLM-generated class descriptions; (ii) a DINO-based labelling (DL) network, consisting of a frozen DINO backbone and a learned alignment module h, trained on top-k confident pseudo-labels from the CDE classifier; (iii) prompt-tuning of CLIP's vision encoder by training visual prompts to minimize a smoothed cross-entropy loss against the DL network's predictions, using FixMatch-style weak/strong augmentation. The paper reports an average absolute gain of 3.6% over the state-of-the-art label-free method LaFTer across 11 datasets, and an average 11.91% improvement over zero-shot CLIP, claiming state-of-the-art label-free performance on 9 of 11 datasets. The paper includes ablations showing each stage contributes, and an appendix describing the k-selection analysis.","tokens_in":17977,"tokens_out":4607,"duration_ms":40736,"significance":"The paper addresses a relevant problem: adapting CLIP to target datasets without labels. The idea of combining DINO's strong visual features with CLIP's text-aligned embedding space in a label-free manner is interesting, and the stage-wise ablations (Tables 3 and 4) provide some evidence that each component contributes. If the label-free claim were airtight, the method would be a practical contribution to test-time adaptation and label-efficient learning. However, the central claim is compromised by the choice of the k hyperparameter, as detailed below, and the self-training loop deserves careful consideration. The paper also provides a public code link, which is a positive for reproducibility.","major_comments":[{"comment":"The choice of k = 20% of the average number of images per class is selected by inspecting the 'Top-1 Accuracy of trained DL network' curves in Figure 4 of the supplementary, which report accuracy on the test sets of ImageNet, CIFAR-100, UCF101, and Caltech101. This means test labels are used to select a hyperparameter of a supposedly label-free pipeline, leaking label information into the reported results. Because the final numbers in Table 1 depend on this choice, the average 3.6% gain over LaFTer is not a verifiable label-free result as presented. The authors must either (a) choose k using only unlabeled data (e.g., a held-out portion of the training set with pseudo-label confidence) and rerun all experiments, or (b) clearly report the results for a fixed rule without test-set tuning. This is load-bearing for the central claim.","section":"Supplementary, Analysis on k selection; Main text, 'DINO-based Labelling (DL) Network'"},{"comment":"The final prompt-learning objective trains CLIP to match the DL network's predictions, but the DL network was trained on pseudo-labels produced by the CDE classifier, which is built directly from CLIP's own text encoder and LLM descriptions. This is a self-training loop in which the student (CLIP with learnable prompts) is taught by a teacher whose supervision originates from the same CLIP text encoder. While the ablation in Table 4 (replacing DINO with CLIP in the DL network) shows that DINO features contribute beyond CLIP, the paper should more explicitly analyze and discuss the risk of confirmation bias: the alignment module h may simply map DINO features into the CLIP text space without adding genuinely new visual information. To strengthen the claim, the authors could demonstrate that the DL network's pseudo-labels on a held-out unlabeled set are more accurate than the CDE classifier's own predictions, or otherwise quantify the added information from DINO.","section":"Eq. (4) and 'DINO-based Labelling (DL) Network'"},{"comment":"The reported results lack error bars or multiple seeds. The claimed gains over LaFTer are small on some datasets (e.g., +1.2% on ImageNet, +0.1% on UCF101, +1.0% on CIFAR100), and without variance estimates it is unclear whether these differences are statistically meaningful. The paper should report at least three random seeds and provide standard deviations for the main results, especially given that the k-selection rule itself adds a source of variance.","section":"Table 1 and Results"}],"minor_comments":[{"comment":"There are numerous typographical errors and inconsistent terminologies, e.g., 'LaFter' vs 'LaFTer', 'unlabelled' vs 'unlabeled', 'performace', 'Morover', 'Inn order', 'propotional'. A thorough proofread is needed.","section":"Throughout"},{"comment":"The claim of 'state-of-the-art performance in 9 out of 11 datasets' is not accurate by the numbers in the table: EuroSAT (Ours 73.5 vs LaFTer 73.9) and CIFAR10 (Ours 94.9 vs LaFTer 95.8) are not the best, and Caltech101 is tied with WaffleCLIP at 94.8. Please clarify how the 9/11 count is computed.","section":"Table 1"},{"comment":"The sentence 'Since our method is entirely label-free, we do not use information about the number of samples per class' is misleading: the average number of images per class is computed from the total number of training images and the number of classes, which is information about the dataset split but not per-class labels. Please rephrase to avoid confusion.","section":"Methodology, 'DINO-based Labelling (DL) Network'"},{"comment":"The paper does not specify the architecture of the alignment module h (e.g., number of layers, hidden dimension), nor the exact smoothing parameter used in the smoothed cross-entropy loss. These details should be provided for reproducibility.","section":"Implementation Details"},{"comment":"In Eq. (2), the class index subscript is inconsistent: 'ϕC' is used in the first line and 'ϕC' in the second line, but the summation index is 'i'; the notation should be cleaned up.","section":"Equation (2)"}],"recommendation":"major_revision","confidential_remarks":"The k-selection issue is the most serious concern: using test accuracy to choose a hyperparameter undermines the paper's central label-free claim. This is not a fixable typing error; it requires re-running the experiments with a validation-based or fully unsupervised selection rule. If the authors can show that a fixed rule (e.g., k=16 or a confidence-based dynamic rule) yields similar gains, the paper would be substantially stronger. The self-training loop is a conceptual risk that should be addressed, but the provided ablation suggests DINO does contribute. Given the manuscript's potential and the clear path to remediation, major revision is appropriate rather than outright rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: NoLA is a genuinely new pipeline for label-free prompt tuning, but its headline 3.6% gain over LaFTer is not currently verifiable as a label-free result. The supplementary selects the k hyperparameter using test-set accuracy, which leaks target labels into model selection.\n\nWhat's new and good: NoLA combines three stages that haven't been put together before. It builds a classifier from LLM descriptions, uses its pseudo-labels to align a frozen DINO encoder to CLIP's embedding space, and then uses that DINO-based labeler as the teacher for visual prompt tuning. The ablations are well-structured and each stage contributes something. The gains on fine-grained datasets like Flowers102 (82.7 vs. LaFTer's 71.0) and RESISC45 (75.4 vs. 68.3) are large and, if real, practically useful. Code and models are linked.\n\nThe soft spots are real, and one is load-bearing. The supplementary's \"Analysis on k selection\" plots top-1 accuracy of the trained DL network on the test set for different k values, and the 20% rule is chosen from those curves. That means the final Table 1 numbers were produced with a hyperparameter selected on the test set. For a method whose entire premise is label-free, this is a serious evaluation flaw. A validation-based or unsupervised criterion for k is needed, or at least a sensitivity analysis that doesn't touch test labels. Without that, the 3.6% average gain could be partly an artifact of test-set overfitting.\n\nThere's also no error bars or multiple seeds. Given the gains are a few points on average, that's not a minor omission; it's necessary context. The circularity the reader flagged is real but less damning: the final teacher is built from pseudo-labels produced by CLIP's own text encoder plus LLM descriptions. That's the standard self-training structure, and LaFTer has the same property, but the paper should state it explicitly and discuss why the DINO alignment breaks the loop.\n\nThe method itself is coherent and the ablations are internally consistent. I don't think there's a load-bearing error in the method; the issue is the evaluation protocol. This is fixable.\n\nWho should read it: people working on label-free CLIP adaptation or using SSL encoders in prompt tuning. It's a solid paper for that audience, but the current version's headline numbers shouldn't be cited.\n\nFor peer review: I'd send it to reviewers. The core idea is novel enough and the ablations strong enough to warrant referee time. But I'd make clear that the evaluation needs to be redone with proper validation-based k selection and variance reporting before the claims are accepted.","headline":"NoLA is a novel pipeline for label-free prompt tuning, but the headline 3.6% gain is unverified because k is selected on test labels.","tokens_in":18549,"tokens_out":3635,"would_cite":false,"duration_ms":30153,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NoLA tunes CLIP with no labels, beating LaFTer by 3.6%","keywords":["label-free prompt tuning","zero-shot classification","CLIP","DINO","large language models","self-supervised learning","vision-language models","pseudo-labeling"],"falsifier":"Take a benchmark dataset with ground-truth labels (e.g., CIFAR-100), train the alignment head $h$ on the exact same top-$k$ samples but with true labels instead of CDE pseudo-labels, and then run the full DINO-assisted prompt learning. If the oracle-aligned version does not outperform NoLA by a clear margin, the pseudo-label quality is not the bottleneck; if it improves substantially, the method's success depends on the CDE pseudo-labels being accurate, which would make the approach fragile on datasets where LLM descriptions are weak.","tokens_in":17549,"feed_emoji":"🏷️","tokens_out":9653,"duration_ms":69295,"temperature":0.7,"pith_summary":"The paper claims that CLIP can be tuned for a fixed set of classes without any labeled images by combining two existing sources of knowledge: LLM-generated class descriptions and the visual features of a self-supervised model like DINO. The proposed NoLA method first builds a text classifier from LLM descriptions, then uses that classifier to pseudo-label the most confident training images and train a small alignment head on frozen DINO features, and finally uses the resulting DINO-based labelling network to supervise learnable visual prompts on CLIP's vision encoder. On 11 image classification datasets, the paper reports an average absolute gain of 3.6% over the previous label-free method LaFTer and an average 11.91% improvement over zero-shot CLIP, reaching state-of-the-art label-free accuracy on 9 of the 11 datasets. This matters because it suggests that human labels are not a hard requirement for adapting vision-language models to downstream classification tasks, and that the complementary strengths of SSL backbones and LLM text knowledge can be combined in a label-free pipeline.","feed_headline":"Label-free CLIP tuning beats prior method by 3.6% on 11 datasets","feed_subtitle":"LLM-written descriptions plus DINO's self-supervised features replace human labels for prompt tuning.","key_machinery":"The carrying mechanism is the DINO-based Labelling (DL) network: a frozen DINO ViT-B/16 backbone $g_s$ plus a small trainable alignment head $h$ that maps DINO's visual features into the CLIP joint embedding space. This head is trained with smoothed cross-entropy on pseudo-labels produced by the class-description embedding (CDE) classifier, which averages the CLIP text embeddings of multiple LLM-generated descriptions per class. Once trained, the DL network serves as the teacher in the final prompt-learning stage, where learnable visual prompts are prepended to CLIP's vision encoder and optimized, together with the CDE classifier, so that CLIP's prediction on a strongly augmented view matches the DL network's prediction on the weakly augmented view (Eq. 4). The alignment head thus converts a self-supervised feature space into a vision-language embedding space without any human labels, which is what makes the transfer of DINO's visual quality to CLIP possible.","core_discovery":"The central claim is that a zero-shot CLIP classifier can be substantially improved for a closed set of classes using only unlabeled images, if one first enriches the text side with LLM-generated descriptions and then uses a self-supervised visual backbone as a teacher. The paper argues that CLIP's visual features are suboptimal for fine-grained discrimination, while DINO features are richer; the missing link is a mapping from DINO's feature space to CLIP's joint embedding space. NoLA obtains that mapping by training a lightweight alignment head $h$ on pseudo-labels produced by the class-description embedding (CDE) classifier, with a smoothed cross-entropy loss and a top-$k$ confident-sample selection. The aligned DINO network then acts as a pseudo-labeler for prompt-tuning CLIP's vision encoder using a FixMatch-style weak/strong augmentation consistency objective, jointly updating learnable visual prompts and the CDE classifier. The paper reports an average absolute gain of 3.6% over LaFTer across 11 datasets, an average 11.91% improvement over zero-shot CLIP, and state-of-the-art label-free accuracy on 9 of the 11 datasets, with ablations showing each of the three stages contributes to the final accuracy.","pith_inferences":["It is natural to expect that the same alignment-and-teach recipe would transfer to other SSL backbones (e.g., SimCLR, MAE) and other vision-language models (e.g., SigLIP, ALIGN), because the alignment head only needs pseudo-labels to map the SSL space into the VLM space; this is not tested in the paper.","The top-$k$ confidence selection rule (20% of the estimated per-class mean, clamped to 16–512) is a heuristic; a confidence-threshold-based selection or an adaptive schedule could improve robustness on heavily long-tailed datasets, which the paper does not explore.","Since the CDE classifier is built from LLM descriptions, the method inherits the LLM's biases and may degrade on domains where the LLM has weak or inaccurate knowledge (e.g., specialist medical or remote-sensing categories); the paper's 11 datasets are all common object/scene/action benchmarks.","The paper reports gains on public benchmarks; a stress test on a heavily imbalanced or out-of-distribution dataset (e.g., iNaturalist or a medical image set) would clarify whether the average 3.6% gain over LaFTer is robust beyond the evaluated domains."],"forward_implications":["NoLA's label-free accuracy exceeds CoOp with 1-shot or 5-shot labels on several of the 11 datasets, showing that unlabeled adaptation can partially close the gap to few-shot prompt learning.","The stage-wise ablation shows each component earns its keep: the CDE classifier raises the six-dataset average from 67.9% to 72.0%, the DL network to 73.9%, and the full DINO-assisted prompt learning to 80.5%.","The method requires no human labels, no caption or metadata for the target images, and no per-dataset prompt engineering beyond the LLM description prompts, which are generic per domain.","The trained DL network, with frozen DINO and a light alignment head, is the teacher for prompt tuning, so the final CLIP model retains its original weights apart from the learned prompts and the updated CDE classifier."],"supporting_citations":[{"why":"Provides CLIP, the vision-language model whose zero-shot classifier is tuned by NoLA.","marker":"(Radford et al. 2021)"},{"why":"Provides DINO, the self-supervised visual encoder that supplies rich features via the DL network.","marker":"(Caron et al. 2021)"},{"why":"Supplies the LLM-generated class description dataset and the CDE construction method adopted for the text classifier.","marker":"(Pratt et al. 2023)"},{"why":"LaFTer, the previous state-of-the-art label-free method that NoLA aims to surpass on the 11 datasets.","marker":"(Mirza et al. 2024b)"},{"why":"FixMatch, whose weak/strong augmentation consistency objective is used for the prompt-learning stage.","marker":"(Sohn et al. 2020)"},{"why":"Visual prompt tuning, the method for injecting learnable prompts into the frozen CLIP vision encoder.","marker":"(Jia et al. 2022)"},{"why":"CoOp, the few-shot prompt-learning baseline used to contextualize NoLA's label-free gains.","marker":"(Zhou et al. 2022b)"}],"fun_headline_variants":["Label-free CLIP tuning with DINO and LLM beats SOTA by 3.6%","NoLA: Use unlabeled data to upgrade CLIP's zero-shot accuracy","Unlabeled images improve CLIP zero-shot: 3.6% gain over LaFTer","CLIP + DINO + LLM: label-free tuning beats zero-shot by 11.9%","Label-free prompt tuning: LLM text + DINO vision lifts CLIP by 3.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a small alignment head, trained only on pseudo-labels from an LLM-built text classifier, can map DINO's visual features into CLIP's embedding space accurately enough that the resulting teacher's pseudo-labels are reliable for prompt tuning.","fun_headline_variants_meta":{"raw":{"variants":["Label-free CLIP tuning with DINO and LLM beats SOTA by 3.6%","NoLA: Use unlabeled data to upgrade CLIP's zero-shot accuracy","Unlabeled images improve CLIP zero-shot: 3.6% gain over LaFTer","CLIP + DINO + LLM: label-free tuning beats zero-shot by 11.9%","Label-free prompt tuning: LLM text + DINO vision lifts CLIP by 3.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000924,"raw_usage":{"total_tokens":4059,"prompt_tokens":1143,"completion_tokens":2916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":759,"completion_tokens_details":{"reasoning_tokens":2794}},"tokens_in":759,"tokens_out":2916,"duration_ms":17337,"temperature":1.0,"reasoning_tokens":2794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:16:03.534480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark dataset with ground-truth labels (e.g., CIFAR-100), train the alignment head $h$ on the exact same top-$k$ samples but with true labels instead of CDE pseudo-labels, and then run the full DINO-assisted prompt learning. If the oracle-aligned version does not outperform NoLA by a clear margin, the pseudo-label quality is not the bottleneck; if it improves substantially, the method's success depends on the CDE pseudo-labels being accurate, which would make the approach fragile on datasets where LLM descriptions are weak.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Visual prompt tuning, the method for injecting learnable prompts into the frozen CLIP vision encoder."}],"review_version":1}