{"id":"dadde540-f546-4118-bec8-d07bd55e0ae2","arxiv_id":"2505.02056","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A concept-alignment and confusion-aware margin framework improves CLIP fine-tuning with imbalanced pseudolabels, but the headline 6.29% gain is limited to the unsupervised setting.","lead":"This paper proposes a way to fine-tune vision-language models without labeled data by fixing two causes of imbalanced pseudolabels: mismatched class names and confusion between similar classes. The method uses ChatGPT to rewrite failing class descriptions and a margin loss to reduce inter-class confusion, reporting consistent gains in the unsupervised setting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's mismatch detection is unvalidated and can remove the very classes it should retain; on EuroSAT it detects zero mismatch classes while CAP still gains, so the concept-alignment contribution is not established.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: Algorithm 1 assumes KMeans clusters align with true class identities, with only qualitative support. I agree this is the most load-bearing issue because the paper's novel mechanism, concept alignment, depends on correctly detecting which classes are concept-mismatched. If the detector is unreliable, the empirical UL gains could come from the confusion-aware margin, the two-adapter design, or the top-k selection, rather than from the proposed causal diagnosis. The EuroSAT zero-detection result strengthens this concern: with the stated t=ceil(C/10)=1, Algorithm 1 removes all classes, so no mismatch class is corrected, yet CAP improves substantially; the paper does not explain how the claimed mechanism operates there. I do not elevate the headline-metric issue to the primary attack because it is a qualification problem that could be fixed by reporting the UL-only aggregate explicitly, whereas the detector concern requires new evidence. The remedy is a concrete detection-evaluation check: compare YMM to ground-truth low-accuracy classes and test whether random subsets of the same size reproduce the gains. If the detector fails that check, the concept-alignment mechanism should be downgraded to a heuristic whose benefit is empirically observed but not attributable to correcting concept mismatch. This does not change the overall conditional verdict: the UL results may still be valid, but the broader causal and state-of-the-art claims need re-analysis.","tokens_in":17240,"tokens_out":11613,"duration_ms":118681,"concrete_test":"Run Algorithm 1 on all six datasets with multiple KMeans seeds and compare YMM against a ground-truth reference defined as the bottom t=ceil(C/10) (or bottom 10%) classes by zero-shot CLIP per-class accuracy; report precision, recall, and Jaccard overlap with the reference. Also ablate by replacing YMM with random class subsets of the same size and measuring final UL accuracy. If precision/recall are near chance or unstable across seeds, or if random subsets match the full method's accuracy, the claim that concept alignment specifically targets concept-mismatched classes is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 is the load-bearing step for the concept-alignment contribution, and its core assumption is unverified. Each iteration removes the text feature/class whose feature is most confidently matched to a KMeans centroid, treating that pair as a well-matched class. For a genuinely concept-mismatched class, its text feature may still be the least-bad match to some centroid, especially after well-matched classes are removed, so it can be removed before Yfinal is formed; conversely, on fine-grained data KMeans can split or merge classes, so the retained centroids need not correspond to true classes. The paper validates detection only with a qualitative RESISC45 example and per-dataset counts in Figure 14, never with precision/recall against ground-truth low-accuracy classes. The EuroSAT case is a sharp internal symptom: with C=10 and t=ceil(C/10)=1, the while loop in Algorithm 1 runs until Y is empty, so YMM is empty, yet CAP improves UL accuracy by 7.75 over CPL (Table 1). The contribution of concept alignment on that dataset is therefore unidentified. Since the central claim attributes the gains to correcting concept mismatch, the unvalidated detector is the weakest point of the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies imbalanced pseudolabels produced by vision-language models (CLIP) during unsupervised adaptation. It attributes the imbalance to two causes: concept mismatch (text features of a class are misaligned with its visual concept) and concept confusion (similar classes' text features fail to capture distinguishing visual cues). The proposed CAP framework first detects concept-mismatched classes via an iterative clustering algorithm (Algorithm 1) and augments their text descriptions with an LLM, then trains the model with a confusion-aware calibrated margin that up-weights uncertain and similar-class confusions, using separate adapters for confidently pseudolabeled and remaining unlabeled data. Experiments on six datasets under UL, SSL, and TRZSL paradigms compare with FPL, GRIP, and CPL; the abstract reports a 6.29% relative improvement over the state of the art.","tokens_in":17477,"tokens_out":4229,"duration_ms":42156,"significance":"If the central claims are valid, the paper offers a useful conceptual decomposition of pseudolabel imbalance in VLMs and a practical method that improves UL accuracy consistently across six benchmarks while being substantially faster than iterative baselines (about 3.5x over GRIP). The ablation study (Table 2) indicates both proposed components contribute, and the method's code is publicly available for reproducibility. However, the main novelty of the paper is concept alignment, and its detection mechanism is not quantitatively validated; the reported headline improvement is not representative across all three learning paradigms. These issues limit the current significance but do not invalidate the core UL result.","major_comments":[{"comment":"The mismatch-detection algorithm is load-bearing for the concept-alignment contribution, yet it is never validated against ground truth. The paper provides only a qualitative example (Figure 1) and the number of detected classes (Figure 14); there is no precision/recall analysis comparing the detected YMM with classes that actually have low zero-shot or post-fine-tuning accuracy. The algorithm's assumption that K-Means clusters on raw image features correspond to true class identities is especially questionable on fine-grained datasets such as CUB and FGVC-Aircraft, where clusters can split or merge classes, so the retained centroids need not represent concept-mismatched classes. I ask the authors to add quantitative detection evaluation (e.g., overlap of YMM with the lowest-accuracy classes, or per-class accuracy improvement for detected vs. non-detected classes) and to discuss the failure modes of the clustering assumption.","section":"§3.1, Algorithm 1"},{"comment":"The EuroSAT configuration creates an internal inconsistency that undermines the attribution of gains to concept alignment. With C=10 and t=ceil(C/10)=1, the while loop in Algorithm 1 terminates only when |Y|<1, i.e., all classes are removed, so YMM is empty and no LLM-based text augmentation is performed. Nevertheless, Table 1 shows CAP improves UL accuracy on EuroSAT by 7.75 points over CPL (75.01 vs. 67.26). Thus on this dataset the observed improvement cannot be explained by concept alignment, yet the paper does not discuss this case. Please explain why the method still improves substantially on EuroSAT, and clarify which components drive the gain when YMM is empty (e.g., CACM or the two-adapter training).","section":"§3.1, Algorithm 1 vs. Table 1"},{"comment":"The headline claim of a 6.29% relative improvement over the state of the art is not representative of the full set of results. In SSL, CAP underperforms CPL on CUB (58.04 vs. 58.53) and FGVC-Aircraft (21.79 vs. 22.48); in TRZSL, it underperforms CPL on CUB (61.35 vs. 66.20) and FGVC-Aircraft (29.03 vs. 30.86). The statement that CAP 'consistently surpasses existing methods' is true only under the UL setting. Please specify the exact setting and computation used for the 6.29% figure, and qualify the consistency claim to match the actual table.","section":"Abstract, Conclusion, and Table 1"},{"comment":"The sensitivity analyses for the margin scale m (Figure 8), the number of pseudolabels k (Table 5), and the confidence threshold tau (Table 6) are all evaluated on the test set, and the default values are selected based on test accuracy. This constitutes test-set tuning and can inflate reported performance. Please use a held-out validation split for selection, or report results across the full range without cherry-picking the test-best value, and state the selection procedure explicitly in the main text.","section":"§4.4, Figures 8 and Appendix D.6, D.7"}],"minor_comments":[{"comment":"There are several typos and inconsistencies, e.g., 'avaliable' in the abstract, 'foundamental' in the introduction, 'vairous' in §4.2, 'pesudolabels' in §3.1, 'concpet' near the end of §3.2, 'diveded' in §4.1, 'fuction' in Appendix C.2, and 'RESICS45' in the caption of Figure 14. A thorough proofread is needed.","section":"Throughout"},{"comment":"The zero-shot CLIP rows report standard deviations of 0.00 for all entries. Please clarify whether these are single-run numbers or a typo; if they are intended to be zero due to deterministic inference, say so explicitly.","section":"Table 1"},{"comment":"The paper does not specify how the 6.29% relative improvement is computed (e.g., average of per-dataset relative gains or a single aggregated accuracy). Please include the definition or a reference to the results table so the claim is reproducible.","section":"Abstract, Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper's central UL result is plausible and the method is clearly described. However, the concept-alignment detection is the main novelty and is currently validated only qualitatively, with an internal inconsistency on EuroSAT that prevents the reader from attributing the observed gains to the proposed mechanism. In addition, the headline performance claim is overstated relative to the full table. I believe these issues are addressable with additional experiments and a more careful presentation, so I recommend major revision rather than rejection. I would also encourage the editor to ensure the '6.29%' claim is clarified and that the test-set sensitivity analyses are repositioned as ablations, not model selection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tom,\n\nQuick take: this paper is worth a serious look if you work on pseudolabeling for VLMs, but don't swallow the headline. The re-framing of imbalance into concept mismatch versus concept confusion is useful, and the unsupervised results are consistently better than CPL on six benchmarks. What's new is the iterative clustering detector for mismatched classes and the confusion-aware margin that combines class similarity with the model's prediction tendency. The two-adapter design—one for the reliable pseudolabeled set, one for the unlabeled set—is also a clean way to avoid confirmation bias.\n\nThe UL column in Table 1 is the strong part. The gains are several points on Flowers102, RESISC45, and EuroSAT, with standard deviations, and the ablation in Table 2 shows both components contribute on the three datasets tested. The sensitivity analyses for m and tau show the method is not knife-edge. That's real evidence.\n\nThe soft spots are real too, and they're in the causal story. Algorithm 1 is the load-bearing piece for concept alignment, but it's validated only through a qualitative example and per-dataset counts. There is no precision/recall against known low-accuracy classes, and on fine-grained data the KMeans assumption that clusters correspond to classes is shaky. The EuroSAT case is the sharpest symptom: with C=10 and t=1, the loop removes all classes, so no mismatch classes are detected, yet CAP still gains 7.75 points over CPL. That gain has to come from the margin and the training scheme, not concept alignment. It's not fatal—the ablation does show CA helps elsewhere—but it means the paper's central explanation for the gains is not uniformly supported.\n\nThe headline 6.29% relative improvement is averaged only over the UL setting; in SSL and TRZSL the method loses to CPL on CUB and FGVC-Aircraft. Hyperparameters m, k, and tau are selected from test-set sensitivity curves, which is a mild selection-bias issue. And the LLM-generated descriptions (ChatGPT 4o-mini) are not reproducible and are never compared against simpler LLM-description baselines like CuPL, so the value added by that component is not isolated.\n\nNet: send to peer review, but ask for a validation of the detector, per-setting averages, and a CuPL comparison. The UL result is solid enough to justify referee time; the broader claims need revision.\n\n— [your name]","headline":"Worth a serious referee: real UL gains and a useful diagnosis, but the causal story hinges on an unvalidated detector and the 6.29% headline only holds in one setting.","tokens_in":18062,"tokens_out":3538,"would_cite":false,"duration_ms":34671,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-language models generate imbalanced pseudolabels because class names can mismatch visual concepts and can blur similar concepts; this paper proposes a framework that detects and repairs both failures and reports a 6.29% relative…","keywords":["vision-language models","pseudolabels","class imbalance","concept mismatch","concept confusion","CLIP fine-tuning","confidence calibration","prompt tuning"],"falsifier":"Compare the classes Algorithm 1 flags as concept-mismatched against known per-class zero-shot accuracies and cluster purities on a label-available benchmark; if flagged classes are not consistently among the worst zero-shot classes, the detector's cluster-to-class premise fails. A sharper check is EuroSAT: with $t=\\lceil C/10\\rceil=1$ the loop removes all ten classes, so concept alignment should detect nothing; if CAP still outperforms its no-concept-alignment ablation there, the reported gains cannot be attributed to mismatch detection.","tokens_in":17002,"feed_emoji":"🏷️","tokens_out":9715,"duration_ms":82621,"temperature":0.7,"pith_summary":"The paper claims that the imbalanced pseudolabels CLIP produces when adapting to downstream image classification come from two identifiable flaws, concept mismatch and concept confusion. Concept mismatch is the failure of a class name's text feature to line up with the visual features of its images; concept confusion is the failure of text features to separate similar classes, biasing predictions toward one of them. The paper proposes a concept-adaptive pseudolabeling (CAP) framework that detects mismatched classes by iterative clustering, repairs their text descriptions with a large language model, and trains with a confusion-aware calibrated margin that penalizes overconfident and imbalanced predictions. On six benchmarks across unsupervised, semi-supervised, and transductive zero-shot settings, it reports consistent gains and a relative improvement of 6.29% over the prior state of the art. If correct, the result matters because it makes task adaptation of vision-language models viable without annotated data.","feed_headline":"6.29% better: fix CLIP pseudo-labels by aligning concepts","feed_subtitle":"A two-part fix balances pseudo-labels across six image datasets and three learning setups.","key_machinery":"The load-bearing machinery is the pairing of an iterative mismatch detector with a confusion-aware calibrated margin. The detector repeatedly runs K-Means on image features, scores text features against cluster centroids, and removes the best-matched text-class and its cluster, leaving a small set of suspected concept-mismatched classes; a large language model then writes candidate descriptions and the one most similar to remaining centroids is chosen. The margin uses a logit-adjustment-style loss, $L_m(y,z) = -\\log\\frac{e^{z_y}}{e^{z_y} + \\sum_{c \\neq y} e^{z_c + M_{yc}}}$, where the margin matrix $M = S \\odot m$ combines inter-class similarity $S$ with class-wise scales $m_c = m\\,\\Delta\\,\\delta_c$ derived from the model's prediction-frequency imbalance. Recomputing $M$ each epoch makes the calibration adapt as confusion is resolved. Two separate adapters, one for clean pseudolabels and one for dynamically labeled unlabeled data, prevent errors made on the unlabeled stream from contaminating the high-confidence stream.","core_discovery":"On the paper's own terms, the central discovery is that the imbalance in vision-language pseudolabels has two distinct, addressable causes. In concept mismatch, the text embedding of a class is far from that class's image embeddings even though the images cluster tightly, so confident pseudolabeling assigns few or no correct labels to the class; the paper's cluster analysis on RESISC45 shows low-accuracy classes still form coherent clusters, locating the problem on the text side. In concept confusion, text embeddings of similar classes fail to encode the distinguishing visual cues, so zero-shot predictions and pseudolabels become biased toward one class. The CAP framework treats mismatch by detecting the affected classes and enhancing their text descriptions, and treats confusion by adding a similarity- and tendency-aware margin to the training loss, forcing more distinguishable and balanced logits. The paper reports that this combination raises accuracy and balances predictions across classes, with the largest gains on previously underperforming classes.","pith_inferences":["The same detect-then-enhance pattern could apply to other vision-language tasks where class names misalign with visuals, such as open-vocabulary detection and segmentation, by treating each output category as a class whose name may need repair.","On fine-grained datasets, the detector's reliance on K-Means is the natural place to probe: hierarchical or soft clustering might identify mismatched classes without splitting or merging true classes, and a comparison against per-class zero-shot accuracy would show which detector is more faithful.","The margin's class-wise scales could be read each epoch as a live measure of which confusions persist, potentially serving as a diagnostic tool for class granularity rather than only a training loss."],"forward_implications":["Underperforming classes receive more correct pseudolabels through concept alignment, so per-class accuracy on the tail of the zero-shot distribution rises rather than being averaged away.","The calibrated margin makes confidence scores better calibrated within confused groups, so a fixed confidence threshold yields more reliable pseudolabels and less confirmation bias.","Because pseudolabels become more balanced and accurate, the method improves results in all three learning paradigms it was tested on, including transductive zero-shot learning where only seen classes are labeled.","Separating main and pseudo adapters lets the model keep learning from all unlabeled data while insulating the high-confidence pseudolabel stream from the lower-quality dynamic pseudolabels.","The method trains once rather than iteratively, making it about 3.5 times faster than the iterative GRIP baseline."],"supporting_citations":[{"why":"Supplies CLIP, the vision-language model whose pseudolabels and encoders the paper adapts.","marker":"(Radford et al., 2021)"},{"why":"Introduces UPL and the observation of imbalanced pseudolabels and confirmation bias that motivates the paper.","marker":"(Huang et al., 2022)"},{"why":"Provides the FPL and GRIP baselines, including the top-k pseudolabel selection that concept alignment replaces for mismatched classes.","marker":"(Menghini et al., 2023)"},{"why":"CPL is the strongest baseline whose results define the state-of-the-art comparison and the reported 6.29% relative improvement.","marker":"(Zhang et al., 2024b)"},{"why":"Logit adjustment is the calibration idea the confusion-aware calibrated margin adapts.","marker":"(Menon et al., 2021)"},{"why":"MaPLe is the prompt-tuning backbone used in the fine-tuning framework.","marker":"(Khattak et al., 2023)"},{"why":"Supplies the confidence-thresholded pseudo-labeling used to generate dynamic pseudolabels for unlabeled data.","marker":"(Sohn et al., 2020)"},{"why":"RESISC45 is the dataset whose cluster analysis motivates the concept-mismatch diagnosis.","marker":"(Cheng et al., 2017)"}],"fun_headline_variants":["Fix CLIP pseudolabels: align concepts, balance margins","Two causes of imbalanced pseudo-labels, one fix","CAP: Align concepts, adjust margins to balance pseudo-labels","Align concepts, tune margins: rebalancing CLIP pseudo-labels","Two-part fix for imbalanced CLIP pseudo-labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest assumption is that K-Means clusters of raw image features correspond to the true classes, so that repeatedly removing the best-matched text-cluster pair leaves exactly the classes whose names are misaligned.","fun_headline_variants_meta":{"raw":{"variants":["Fix CLIP pseudolabels: align concepts, balance margins","Two causes of imbalanced pseudo-labels, one fix","CAP: Align concepts, adjust margins to balance pseudo-labels","Align concepts, tune margins: rebalancing CLIP pseudo-labels","Two-part fix for imbalanced CLIP pseudo-labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3195,"prompt_tokens":911,"completion_tokens":2284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2197}},"tokens_in":527,"tokens_out":2284,"duration_ms":14394,"temperature":1.0,"reasoning_tokens":2197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:03:18.681006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the classes Algorithm 1 flags as concept-mismatched against known per-class zero-shot accuracies and cluster purities on a label-available benchmark; if flagged classes are not consistently among the worst zero-shot classes, the detector's cluster-to-class premise fails. A sharper check is EuroSAT: with $t=\\lceil C/10\\rceil=1$ the loop removes all ten classes, so concept alignment should detect nothing; if CAP still outperforms its no-concept-alignment ablation there, the reported gains cannot be attributed to mismatch detection.","supporting_citations":[{"cited_title":"U., Rasheed, H","cited_arxiv_id":null,"evidence_quote":"MaPLe is the prompt-tuning backbone used in the fine-tuning framework."}],"review_version":1}