REVIEW 4 major objections 3 minor 19 references
Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin
T0 review · 4 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Vision-language models generate imbalanced pseudolabels because class names can mismatch visual concepts and can blur similar concepts; this paper proposes a framework that detects and repairs both failures and reports a 6.29% relative…
desk verdict Worth a serious referee: real UL gains and a useful diagnosis, but the causal story hinges on an unvalidated detector and the 6.29% headline only holds in one setting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the pairing of an iterative mismatch detector with a confusion-aware calibrated margin. The detector repeatedly runs K-Means on image features, scores text features against cluster centroids, and removes the best-matched text-class and its cluster, leaving a small set of suspected concept-mismatched classes; a large language model then writes candidate descriptions and the one most similar to remaining centroids is chosen. The margin uses a logit-adjustment-style loss, $L_m(y,z) = -\log\frac{e^{z_y}}{e^{z_y} + \sum_{c \neq y} e^{z_c + M_{yc}}}$, where the margin matrix $M = S \odot m$ combines inter-class similarity $S$ with class-wise scales $m_c = m\,\Delta\,\delta_c$ derived from the model's prediction-frequency imbalance. Recomputing $M$ each epoch makes the calibration adapt as confusion is resolved. Two separate adapters, one for clean pseudolabels and one for dynamically labeled unlabeled data, prevent errors made on the unlabeled stream from contaminating the high-confidence stream.
What would settle it
Compare the classes Algorithm 1 flags as concept-mismatched against known per-class zero-shot accuracies and cluster purities on a label-available benchmark; if flagged classes are not consistently among the worst zero-shot classes, the detector's cluster-to-class premise fails. A sharper check is EuroSAT: with $t=\lceil C/10\rceil=1$ the loop removes all ten classes, so concept alignment should detect nothing; if CAP still outperforms its no-concept-alignment ablation there, the reported gains cannot be attributed to mismatch detection.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the imbalance in vision-language pseudolabels has two distinct, addressable causes. In concept mismatch, the text embedding of a class is far from that class's image embeddings even though the images cluster tightly, so confident pseudolabeling assigns few or no correct labels to the class; the paper's cluster analysis on RESISC45 shows low-accuracy classes still form coherent clusters, locating the problem on the text side. In concept confusion, text embeddings of similar classes fail to encode the distinguishing visual cues, so zero-shot predictions and pseudolabels become biased toward one class. The CAP framework treats mismatch by detecting the affected classes and enhancing their text descriptions, and treats confusion by adding a similarity- and tendency-aware margin to the training loss, forcing more distinguishable and balanced logits. The paper reports that this combination raises accuracy and balances predictions across classes, with the largest gains on previously underperforming classes.
Load-bearing premise
The weakest assumption is that K-Means clusters of raw image features correspond to the true classes, so that repeatedly removing the best-matched text-cluster pair leaves exactly the classes whose names are misaligned.
Editorial extensions
If this is right
- Underperforming classes receive more correct pseudolabels through concept alignment, so per-class accuracy on the tail of the zero-shot distribution rises rather than being averaged away.
- The calibrated margin makes confidence scores better calibrated within confused groups, so a fixed confidence threshold yields more reliable pseudolabels and less confirmation bias.
- Because pseudolabels become more balanced and accurate, the method improves results in all three learning paradigms it was tested on, including transductive zero-shot learning where only seen classes are labeled.
- Separating main and pseudo adapters lets the model keep learning from all unlabeled data while insulating the high-confidence pseudolabel stream from the lower-quality dynamic pseudolabels.
- The method trains once rather than iteratively, making it about 3.5 times faster than the iterative GRIP baseline.
Reading between the lines
- The same detect-then-enhance pattern could apply to other vision-language tasks where class names misalign with visuals, such as open-vocabulary detection and segmentation, by treating each output category as a class whose name may need repair.
- On fine-grained datasets, the detector's reliance on K-Means is the natural place to probe: hierarchical or soft clustering might identify mismatched classes without splitting or merging true classes, and a comparison against per-class zero-shot accuracy would show which detector is more faithful.
- The margin's class-wise scales could be read each epoch as a live measure of which confusions persist, potentially serving as a diagnostic tool for class granularity rather than only a training loss.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies imbalanced pseudolabels produced by vision-language models (CLIP) during unsupervised adaptation. It attributes the imbalance to two causes: concept mismatch (text features of a class are misaligned with its visual concept) and concept confusion (similar classes' text features fail to capture distinguishing visual cues). The proposed CAP framework first detects concept-mismatched classes via an iterative clustering algorithm (Algorithm 1) and augments their text descriptions with an LLM, then trains the model with a confusion-aware calibrated margin that up-weights uncertain and similar-class confusions, using separate adapters for confidently pseudolabeled and remaining unlabeled data. Experiments on six datasets under UL, SSL, and TRZSL paradigms compare with FPL, GRIP, and CPL; the abstract reports a 6.29% relative improvement over the state of the art.
Significance. If the central claims are valid, the paper offers a useful conceptual decomposition of pseudolabel imbalance in VLMs and a practical method that improves UL accuracy consistently across six benchmarks while being substantially faster than iterative baselines (about 3.5x over GRIP). The ablation study (Table 2) indicates both proposed components contribute, and the method's code is publicly available for reproducibility. However, the main novelty of the paper is concept alignment, and its detection mechanism is not quantitatively validated; the reported headline improvement is not representative across all three learning paradigms. These issues limit the current significance but do not invalidate the core UL result.
major comments (4)
- [§3.1, Algorithm 1] The mismatch-detection algorithm is load-bearing for the concept-alignment contribution, yet it is never validated against ground truth. The paper provides only a qualitative example (Figure 1) and the number of detected classes (Figure 14); there is no precision/recall analysis comparing the detected YMM with classes that actually have low zero-shot or post-fine-tuning accuracy. The algorithm's assumption that K-Means clusters on raw image features correspond to true class identities is especially questionable on fine-grained datasets such as CUB and FGVC-Aircraft, where clusters can split or merge classes, so the retained centroids need not represent concept-mismatched classes. I ask the authors to add quantitative detection evaluation (e.g., overlap of YMM with the lowest-accuracy classes, or per-class accuracy improvement for detected vs. non-detected classes) and to discuss the failure modes of the clustering assumption.
- [§3.1, Algorithm 1 vs. Table 1] The EuroSAT configuration creates an internal inconsistency that undermines the attribution of gains to concept alignment. With C=10 and t=ceil(C/10)=1, the while loop in Algorithm 1 terminates only when |Y|<1, i.e., all classes are removed, so YMM is empty and no LLM-based text augmentation is performed. Nevertheless, Table 1 shows CAP improves UL accuracy on EuroSAT by 7.75 points over CPL (75.01 vs. 67.26). Thus on this dataset the observed improvement cannot be explained by concept alignment, yet the paper does not discuss this case. Please explain why the method still improves substantially on EuroSAT, and clarify which components drive the gain when YMM is empty (e.g., CACM or the two-adapter training).
- [Abstract, Conclusion, and Table 1] The headline claim of a 6.29% relative improvement over the state of the art is not representative of the full set of results. In SSL, CAP underperforms CPL on CUB (58.04 vs. 58.53) and FGVC-Aircraft (21.79 vs. 22.48); in TRZSL, it underperforms CPL on CUB (61.35 vs. 66.20) and FGVC-Aircraft (29.03 vs. 30.86). The statement that CAP 'consistently surpasses existing methods' is true only under the UL setting. Please specify the exact setting and computation used for the 6.29% figure, and qualify the consistency claim to match the actual table.
- [§4.4, Figures 8 and Appendix D.6, D.7] The sensitivity analyses for the margin scale m (Figure 8), the number of pseudolabels k (Table 5), and the confidence threshold tau (Table 6) are all evaluated on the test set, and the default values are selected based on test accuracy. This constitutes test-set tuning and can inflate reported performance. Please use a held-out validation split for selection, or report results across the full range without cherry-picking the test-best value, and state the selection procedure explicitly in the main text.
minor comments (3)
- [Throughout] There are several typos and inconsistencies, e.g., 'avaliable' in the abstract, 'foundamental' in the introduction, 'vairous' in §4.2, 'pesudolabels' in §3.1, 'concpet' near the end of §3.2, 'diveded' in §4.1, 'fuction' in Appendix C.2, and 'RESICS45' in the caption of Figure 14. A thorough proofread is needed.
- [Table 1] The zero-shot CLIP rows report standard deviations of 0.00 for all entries. Please clarify whether these are single-run numbers or a typo; if they are intended to be zero due to deterministic inference, say so explicitly.
- [Abstract, Conclusion] The paper does not specify how the 6.29% relative improvement is computed (e.g., average of per-dataset relative gains or a single aggregated accuracy). Please include the definition or a reference to the results table so the claim is reproducible.
Circularity Check
No significant circularity: the central claims are evaluated on held-out test sets, and the proposed components are self-training heuristics rather than derived predictions.
full rationale
The paper's central claim is an empirical one: CAP improves test accuracy and balance over baselines on six benchmarks. All headline numbers (including the 6.29% relative improvement) are post-hoc summaries of Table 1, not quantities predicted from fitted parameters. The two components are not circular by construction. Concept alignment (Algorithm 1) detects 'concept-mismatched' classes using KMeans clustering of image features and text-feature similarity, then intersects the retained set with Ylow-t (classes with fewest predicted samples). This is a heuristic operationalization of 'mismatch', not a definition that assumes the conclusion; the subsequent evaluation of corrected pseudolabel accuracy (Figure 6) is against true labels. The confusion-aware calibrated margin (Eqs. 1-7) is computed from the model's own predictions on unlabeled data and from prototype similarities, which creates a self-training feedback loop, but the margin is not fitted to test labels and the final accuracy is independently measured. The EuroSAT case (C=10, t=1 makes Yfinal empty while CAP still gains 7.75% over CPL in UL) weakens the attribution of the gain to concept alignment on that dataset, but it does not make the derivation circular because the method still contains the margin and adapter components. There are no load-bearing self-citations; the citation to Menon et al. (2021) is for the general logit-adjustment idea, which the paper extends with a similarity- and tendency-based margin rather than renaming. No equation reduces to its own input, and no fitted parameter is relabeled as a prediction. The only concerns are about validation of the mismatch detector and attribution of gains, which are correctness/robustness issues, not circularity.
Assumptions & free parameters
free parameters (5)
- t (mismatch detection threshold) =
ceil(C/10)
- n (LLM query count per class) =
5
- k (pseudolabels per class) =
16
- m (margin scale) =
12
- tau (confidence threshold) =
0.85 (UL), 0.5 (SSL/TRZSL)
assumptions (6)
- domain assumption K-Means clustering of image features produces clusters that correspond to true classes.
- domain assumption The semantic gap in VLMs manifests as the two phenomena of concept mismatch and concept confusion.
- domain assumption LLM-generated descriptions, when selected by similarity to cluster centroids, are better aligned with visual concepts than the original class names.
- domain assumption The margin matrix built from class similarities and prediction tendencies improves calibration and reduces confusion.
- domain assumption The model's confidence-based counts sigma(c) are a reliable measure of class-wise prediction tendency for setting margins.
- domain assumption Using separate adapters for DPL and DUL avoids confirmation bias.
invented entities (2)
-
Concept mismatch and concept confusion taxonomy
-
Confusion-aware calibrated margin matrix M
Cite this review
Pith. "Pith review of Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin." pith.science (2026). https://pith.science/paper/GTFPJATD
@misc{pith2026250502056,
author = {Pith},
title = {Pith review of: Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin},
year = {2026},
howpublished = {\url{https://pith.science/paper/GTFPJATD}},
note = {Machine review of arXiv:2505.02056}
}
read the original abstract
Adapting vision-language models (VLMs) to downstream tasks with pseudolabels has gained increasing attention. A major obstacle is that the pseudolabels generated by VLMs tend to be imbalanced, leading to inferior performance. While existing methods have explored various strategies to address this, the underlying causes of imbalance remain insufficiently investigated. To fill this gap, we delve into imbalanced pseudolabels and identify two primary contributing factors: concept mismatch and concept confusion. To mitigate these two issues, we propose a novel framework incorporating concept alignment and confusion-aware calibrated margin mechanisms. The core of our approach lies in enhancing underperforming classes and promoting balanced predictions across categories, thus mitigating imbalance. Extensive experiments on six benchmark datasets with three learning paradigms demonstrate that the proposed method effectively enhances the accuracy and balance of pseudolabels, achieving a relative improvement of 6.29% over the SoTA method. Our code is avaliable at https://anonymous.4open.science/r/CAP-C642/
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Addepalli, S., Asokan, A. R., Sharma, L., and Babu, R. V . Leveraging vision-language models for improving do- main generalization in image classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pp. 23922–23932. IEEE,
work page 2024
-
[3]
Detailed settings for experiments. Flowers102 RESISC45 DTD CUB EuroSAT FGVCAircraft Statistic data Class number 102 45 47 200 10 100 Training set size 2040 6300 3760 5594 27000 6667 Testing set size 6149 25200 1880 5794 5000 3333 Training Setting Prompt LayersL 8 Prompt per Layer 2 Image Augmentation random resized crop Confidence Thresholdτ 0.85 0.5 k in...
-
[5]
Gu, X., Lin, T., Kuo, W., and Cui, Y
doi: 10.1007/S11263-023-01891-X. Gu, X., Lin, T., Kuo, W., and Cui, Y . Open-vocabulary object detection via vision and language knowledge dis- tillation. In Proc. of ICLR. OpenReview.net,
-
[9]
Fine-grained visual classification of aircraft
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151,
-
[10]
Kim, J., Ku, Y ., Kim, J., Cha, J., and Baek, S
1109/CVPR52729.2023.01832. Kim, J., Ku, Y ., Kim, J., Cha, J., and Baek, S. Vlm-pl: Advanced pseudo labeling approach for class incremen- tal object detection via vision-language model. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4170–4181,
arXiv 2023
-
[12]
doi: 10.1007/978-3-031-72643-9\
-
[13]
Florence: A new foundation model for computer vision
Yuan, L., Chen, D., Chen, Y .-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. Florence: A new foundation model for computer vision. ArXiv preprint, abs/2111.11432,
-
[15]
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. ArXiv preprint, abs/2109.01134,
Show all 19 references
-
[16]
C., and Liu, Z
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Condi- tional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 16795–16804. IEEE,
2022
-
[17]
Handling Imbalanced Pseudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
doi: 10.1109/CVPR52688.2022.01631. 10 Handling Imbalanced Pseudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin Appendix for “Handling Imbalanced Pseudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin” A. Examples of Mis...
2022
-
[19]
Comparison Methods We briefly introduce the baselines in this section
C.2. Comparison Methods We briefly introduce the baselines in this section. Few-pseudolabels (FPL) (Menghini et al., 2023): FPL is the same as UPL (Huang et al., 2022), which generates offline pseudolabels by selecting the top-k confident samples per class in zero-shot predict...
2023
-
[2011]
Marvelovd: Marrying object recognition and vision-language models for robust open-vocabulary object detection
Wang, K., Cheng, L., Chen, W., Zhang, P., Lin, L., Zhou, F., and Li, G. Marvelovd: Marrying object recognition and vision-language models for robust open-vocabulary object detection. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Com...
2024
-
[2014]
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y ., Li, H., and Qiao, Y
doi: 10.1109/CVPR.2014.461. Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y ., Li, H., and Qiao, Y . Clip-adapter: Better vision- language models with feature adapters. ArXiv preprint, abs/2110.04544,
2014 arXiv
-
[2017]
Describing textures in the wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp. 3606–3613. IEEE Computer Society,
2014
-
[2018]
The caltech-ucsd birds-200-2011 dataset
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Tech- nology,
2011
-
[2019]
Unsupervised prompt learning for vision-language models
Huang, T., Chu, J., and Wei, F. Unsupervised prompt learning for vision-language models. ArXiv preprint, abs/2204.03649,
-
[2021]
Zang, Y ., Li, W., Zhou, K., Huang, C., and Loy, C. C. Uni- fied vision and language prompt learning. ArXiv preprint, abs/2210.07225,
-
[2022]
U., Rasheed, H
Khattak, M. U., Rasheed, H. A., Maaz, M., Khan, S. H., and Khan, F. S. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 19113–19122. IEEE,
2023
-
[2024]
2024.02258
doi: 10.1109/CVPR52733. 2024.02258. Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Has- son, Y ., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock,...
2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.