REVIEW 4 major objections 5 minor 33 references
Improving Multi-Label Contrastive Learning by Leveraging Label Distribution
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read A multi-label contrastive learner that recovers label distributions from logical labels and uses them as loss weights outperforms state-of-the-art methods on six metrics across nine datasets.
desk verdict Promising combination of label distribution learning and multi-label contrastive learning, but the recovery mechanism lacks graded supervision, one core loss term has a sign error, and the empirical claims are not sufficiently supported; worth a careful review but not acceptable as is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the recovered label distribution $d_i$, a per-instance probability vector over the label set that encodes graded relevance. It is produced by a fully connected layer with parameters $W$, and is constrained by three terms: $G^{(k)}_i = \sum_{p \in P(i)} C^{(k)}_{i,p} \|d_i - d_p\|_2^2$ asks contrastive positives to agree on label distributions; $H_i = \|Q^{ll}_p - Q^{ld}_p\|_2^2$ keeps the stored distributions close to the stored logical labels; and $\beta\|W\|_F^2$ controls model complexity. The two variants differ only in the similarity function $C^{(1)}$ (RBF) versus $C^{(2)}$ (contrastive log-probability). These definitions feed directly into the final reweighted losses of Equations (12) and (13), so the learned distribution is the mechanism by which graded label importance enters contrastive learning.
What would settle it
Train MulSupConCLD on a synthetic multi-label dataset whose true label distributions are known (for example, drawn from a Dirichlet distribution over labels that are marked present when the weight exceeds a threshold), then compare each recovered distribution to the true one with KL divergence or rank correlation. If the recovered distributions are indistinguishable from one-hot logical vectors, the reweighting in Equations (12) and (13) reduces to binary weighting, and replacing them with logical-label weights should reproduce the same performance.
Extended reading notes
Core claim
The central claim is that positive-sample selection is not the main bottleneck in multi-label supervised contrastive learning; the way label information is weighted inside the loss is. MulSupConLD keeps the simple ANY criterion for positives — samples sharing at least one label — and replaces binary indicator weights with a recovered label distribution $d_i$ over the label set, normalized so that $\sum_j d^j_i = 1$. Two estimators are trained jointly with the contrastive loss: one uses an RBF kernel $C^{(1)}_{i,p} = \exp(-\|z_i - z_p\|_2^2 / 2\sigma^2)$ and one uses the contrastive log-probability $C^{(2)}_{i,p} = \log \frac{\exp(\mathrm{sim}(z_i, z_p)/\tau)}{\sum_{a \in A(i)} \exp(\mathrm{sim}(z_i, z_a)/\tau)}$. Both enforce that positive samples share similar label distributions, while a constraint term keeps the recovered distribution near the original logical labels. The final loss multiplies each positive pair's contribution for label $j$ by $d^j_p \cdot y^j_p$, so labels estimated as more important exert more influence; the paper shows that this reweighting improves performance over the binary-weighted MulSupCon baseline on image and vector datasets.
Load-bearing premise
The method assumes that the label distribution recovered while being penalized for deviating from the logical labels still carries genuinely graded importance information, rather than collapsing back into the binary labels it was trained to approximate.
Editorial extensions
If this is right
- Any supervised contrastive learner for multi-label tasks can adopt the recovered distribution as drop-in per-label weights, bypassing hand-designed positive-sample selection rules.
- The consistent edge of MulSupConCLD over MulSupConRLD suggests that the contrastive similarity itself is a more informative signal for distribution recovery than a fixed RBF kernel with a hand-set bandwidth.
- Because the same fully connected layer generates both the label distribution used in pretraining and the final classifier, the pretraining and downstream classification stages share a graded-label representation rather than a binary one.
- The gains span image datasets (trained with a convolutional encoder) and vector datasets (trained with a multilayer perceptron), indicating the mechanism is not tied to one data modality or encoder family.
Reading between the lines
- Because Equation (8) actively penalizes deviation from the binary logical labels, the recovered distribution is likely better interpreted as a relative importance ranking than as a calibrated probability; a direct check would compare it to ground-truth soft labels on synthetic data.
- The same distribution-reweighting idea transfers to single-label contrastive learning whenever labels carry confidence, such as noisy labels or teacher-soft targets, where the binary mask currently discards confidence information.
- The long-tail direction suggested in the conclusion could be tested immediately by inspecting whether the recovered distributions assign higher weights to rare labels; if not, the reweighting may be amplifying head-class dominance rather than correcting it.
- The ANY positive criterion combined with distribution weights leaves open a softer variant where positive weight is proportional to distribution overlap, which could improve recall on ambiguous multi-label instances without hurting precision.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes MulSupConLD, a multi-label supervised contrastive learning method that replaces binary logical-label weights with an estimated label distribution d. Two recovery variants are introduced: MulSupConRLD uses RBF-kernel feature similarities (Eqs. 6-9), and MulSupConCLD uses the contrastive log-probability (Eqs. 10-11). The recovered distribution is used to reweight the contrastive loss in Eqs. (12)-(13). The framework follows an ANY-style positive sample selection with MoCo-style encoders, is pretrained, and is then fine-tuned with BCE. Experiments on four image datasets and five vector datasets report six metrics, with ablations and parameter sensitivity on Scene and PASCAL. The authors claim that the method outperforms state-of-the-art methods and that label distributions capture varying label importance.
Significance. The intended contribution is timely: if the recovered label distributions genuinely encoded per-label graded importance, this would be a lightweight improvement over MulSupCon and a novel bridge between label distribution learning and contrastive learning. The evaluation is broad in scope (nine datasets, six metrics), and the ablation and sensitivity studies are a positive feature. However, the paper provides no code, no direct analysis of the recovered distributions, and no control experiment that separates label-distribution reweighting from the extra model components. The mathematical form of the recovery objective, especially Eq. (8), indicates that d collapses to normalized logical labels rather than graded importance. Because this mechanism is the paper's only claimed novelty, the significance is not established by the current evidence.
major comments (4)
- [Section 3.2, Eq. (8)] The central claim is that the recovered distribution d encodes graded label importance. This is contradicted by the recovery objective: d is constrained to the probability simplex (sum_j d_i^j = 1), while the logical label y_i is a binary vector whose entries sum to the number of positive labels. Minimizing H_i = ||Q_ll - Q_ld||^2 therefore drives d_i toward the L2 projection of y_i onto the simplex, which for k positive labels is approximately d_i^j = 1/k for positive labels and 0 otherwise. That is exactly normalized logical labels, not graded importance. The G terms in Eqs. (7) and (11) only require positive pairs to have similar d, which is also satisfied by any vector that is constant on co-occurring label groups. The paper reports no statistics of the recovered d, no calibration check, and no comparison against the baseline d = y/|y|. Without such evidence, the reweighting in Eqs. (12)-(13) cannot be claimed to leverage label distribution rather than binary labels.
- [Section 3.2, Eqs. (10)-(11)] C^(2)_{i,p} in Eq. (10) is the logarithm of a softmax probability, so it is non-positive (typically negative). Equation (11) adds G^(2)_i = sum_p C^(2)_{i,p} ||d_i - d_p||^2 to the loss; because the coefficient is negative, minimizing this term maximizes the squared distance between d_i and d_p for every positive pair. This is the opposite of the stated goal in Section 3.2 that label distributions of similar samples should be similar. The same C^(2) appears in Eqs. (12)-(13) with a leading minus sign, where the sign is conventional; the inconsistency is specific to the distribution-recovery loss of the CLD variant. The sign can be corrected, but the correction does not by itself supply graded supervision.
- [Section 3.2 and Section 4.1] The proposed training loop is circular with respect to the claimed mechanism. d is produced by a fully connected layer on the same features z whose contrastive similarities define both C^(2) in the recovery loss and the final reweighted contrastive loss. Since no dataset provides ground-truth label distributions, the recovery objective is the only training signal for d, and it contains no external importance information. The reported end-task accuracy is an external benchmark, but it cannot distinguish the label-distribution hypothesis from the alternative that the extra fully connected parameters, the RBF/log-probability terms, and the regularizer beta||W||^2_F improve optimization independently of any graded label signal. A minimal control would replace d_p^j with y_p^j / |y_p| (or a random distribution) while keeping all other components, and the paper should report the divergence between recovered and logical-label-derived distributions; neither is present.
- [Section 4.1, Tables 2-4] The abstract states that the method outperforms state-of-the-art methods on six metrics across nine datasets, but the tables do not support an unqualified superiority claim. For example, Table 3 shows Scene mAP of MulSupConCLD at 0.821 versus 0.859 for MulSupCon; Table 2 shows Scene maF1 of MulSupConCLD at 0.766 versus 0.787 for MulSupCon; Table 4 shows MS-COCO miF1 of MulSupConRLD at 0.5865 versus 0.6245 for MulSupCon. Moreover, no standard deviations or significance tests are reported, so even the favorable differences, often a few tenths of a point, cannot be assessed for statistical reliability. The empirical claim needs to be qualified and supported with variance estimates or repeated-run comparisons.
minor comments (5)
- [Section 1 vs Section 3.2 and Algorithm 1] The positive-sample definition is inconsistent: the introduction says 'we adopt the ANY strategy', but Section 3.2 says 'We use ALL to define positive samples' and Algorithm 1 uses mask <- y[i] intersect Qll, which is ANY. Please reconcile this, as it is needed for reproduction.
- [References and Related Work] MulSupCon is attributed to 'Zeng et al. (2024)' in the Introduction and to 'Zhang and Wu (2024)' in Section 2; the reference list contains both, and the correct source for MulSupCon appears to be Zhang and Wu.
- [Figure 2] Figure 2 appears to be an unrendered placeholder: the caption refers to bar plots, but the figure content is a plain list of metric names and values. The ablation results should be presented as actual plots or tables with clear group labels.
- [Section 4, Settings] The relationship between the recovered distribution D and the final classification head is underspecified. The text says the fully connected layer generating label distributions is later trained with BCE, but it does not state whether D is used at inference or only as a pretraining auxiliary target.
- [Section 4.3, Tables 5-8] The sensitivity tables would be easier to interpret if the default values of alpha and beta used for the main results were stated; the tables fix one parameter at a time, but the main configuration is never given.
Circularity Check
No significant circularity: the central claim is benchmarked on external test labels, and the label-distribution recovery is a soft regularized component rather than a definitional identity.
full rationale
The paper's headline claim is an empirical comparison on nine datasets against external test-set labels, so the evaluation is not circular. The intermediate label distribution d is not set equal to logical labels by construction: Eq. (8) is one regularizer (alpha * H_i) in a multi-term loss that also includes G terms enforcing distributional similarity among positive pairs and a Frobenius-norm penalty, so d has freedom to deviate from y. Lack of calibration checks or statistics on the recovered d is an evidence gap about whether the label-distribution mechanism works as advertised, not a circular derivation. The only self-citation is a passing reference to CCL-SC (Wu et al., 2024), which shares an author but is not load-bearing for any claim in this paper. Accordingly, the derivation chain is self-contained against external benchmarks and no step reduces to its own input by construction.
Assumptions & free parameters
free parameters (4)
- alpha =
not reported for main runs; grid {0.001, 0.01, 0.1, 1}
- beta =
not reported for main runs; grid {0.001, 0.01, 0.1, 1}
- sigma =
0.01
- temperature tau =
0.1
assumptions (4)
- domain assumption The label distribution D is defined on the simplex (sum=1) and can be recovered from binary labels by minimizing a contrastive-similarity-based objective.
- ad hoc to paper Contrastive feature similarity (or its RBF version) is a valid proxy for label importance.
- ad hoc to paper The logical-label alignment term H in Equation (8) does not erase the graded information in D.
- domain assumption MoCo-style momentum encoder and SimCLR augmentations transfer from single-label to multi-label settings.
invented entities (1)
-
Recovered label distribution D
Cite this review
Pith. "Pith review of Improving Multi-Label Contrastive Learning by Leveraging Label Distribution." pith.science (2026). https://pith.science/paper/OCOJAL54
@misc{pith2026250119145,
author = {Pith},
title = {Pith review of: Improving Multi-Label Contrastive Learning by Leveraging Label Distribution},
year = {2026},
howpublished = {\url{https://pith.science/paper/OCOJAL54}},
note = {Machine review of arXiv:2501.19145}
}
read the original abstract
In multi-label learning, leveraging contrastive learning to learn better representations faces a key challenge: selecting positive and negative samples and effectively utilizing label information. Previous studies selected positive and negative samples based on the overlap between labels and used them for label-wise loss balancing. However, these methods suffer from a complex selection process and fail to account for the varying importance of different labels. To address these problems, we propose a novel method that improves multi-label contrastive learning through label distribution. Specifically, when selecting positive and negative samples, we only need to consider whether there is an intersection between labels. To model the relationships between labels, we introduce two methods to recover label distributions from logical labels, based on Radial Basis Function (RBF) and contrastive loss, respectively. We evaluate our method on nine widely used multi-label datasets, including image and vector datasets. The results demonstrate that our method outperforms state-of-the-art methods in six evaluation metrics.
Figures
Reference graph
Works this paper leans on
-
[1]
J. Bai, S. Kong, and C. Gomes. Disentangled variational autoencoder based multi-label classification with covariance-aware multivariate probit model. In Proceedings of the 29th International Joint Conference on Artificial Intelligence, pages 4313--4321, 2021
work page 2021
-
[2]
J. Bai, S. Kong, and C. P. Gomes. Gaussian mixture variational autoencoder with contrastive learning for multi-label classification. In Proceedings of the 39th International Conference on Machine Learning, pages 1383--1398, 2022
work page 2022
-
[3]
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pages 1597--1607, 2020
work page 2020
-
[4]
T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng. Nus-wide: A real-world web image database from national university of singapore. In Proceedings of the 2nd ACM International Conference on Image and Video Retrieval, pages 1--9, 2009
work page 2009
-
[5]
M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 0 303--338, 2010
work page 2010
-
[6]
X. Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 28 0 (7): 0 1734--1748, 2016
work page 2016
- [7]
-
[8]
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the 33rd IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9726--9735, 2020
work page 2020
Show all 33 references
-
[9]
Huang, Y
G. Huang, Y. Long, C. Luo, and S. Liu. Similarity-dissimilarity loss with hierachical contrastive learning for multi-label classification. ArXiv, abs/2410.13439, 2024
2024
-
[10]
M. J. Huiskes and M. S. Lew. The MIR flickr retrieval evaluation. In Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, pages 39--43, 2008
2008
-
[11]
X. Jia, Z. Li, X. Zheng, W. Li, and S.-J. Huang. Label distribution learning with label correlations on local samples. IEEE Transactions on Knowledge and Data Engineering, 33 0 (4): 0 1619--1631, 2021
2021
-
[12]
X. Jia, X. Shen, W. Li, Y. Lu, and J. Zhu. Label distribution learning by maintaining label ranking relation. IEEE Transactions on Knowledge and Data Engineering, 35 0 (2): 0 1695--1707, 2023 a
2023
-
[13]
Y. Jia, J. Tang, and J. Jiang. Label distribution learning from logical label. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence, pages 4228--4236, 2023 b
2023
-
[14]
Khosla, P
P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 18661--18673, 2020
2020
-
[15]
Lanchantin, A
J. Lanchantin, A. Sekhon, and Y. Qi. Neural message passing for multi-label classification. In Proceedings of the 20th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, pages 138--163, 2020
2020
-
[16]
T.-Y. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll \'a r, and C. L. Zitnick. Microsoft coco: C ommon objects in context. In Proceedings of the 13th European Conference on Computer Vision, pages 740--755, 2014
2014
-
[17]
Markatopoulou, V
F. Markatopoulou, V. Mezaris, and I. Patras. Implicit and explicit concept relations in deep neural networks for multi-label video/image annotation. IEEE Transactions on Circuits and Systems for Video Technology, 29 0 (6): 0 1631--1644, 2019
2019
-
[18]
T. Ren, X. Jia, W. Li, L. Chen, and Z. Li. Label distribution learning with label-specific features. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3318--3324, 2019
2019
-
[19]
Ridnik, E
T. Ridnik, E. B. Baruch, N. Zamir, A. Noy, I. Friedman, M. Protter, and L. Zelnik - Manor. Asymmetric loss for multi-label classification. In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision, pages 82--91, 2021
2021
-
[20]
R. Shao, N. Xu, and X. Geng. Multi-label learning with label enhancement. In Proceedings of the 18th IEEE International Conference on Data Mining, pages 437--446, 2018
2018
-
[21]
Wang and X
J. Wang and X. Geng. Label distribution learning by exploiting label distribution manifold. IEEE Transactions on Neural Networks and Learning Systems, 34 0 (2): 0 839--852, 2023
2023
-
[22]
Wu, S.-H
Y.-C. Wu, S.-H. Lyu, H. Shang, X. Wang, and C. Qian. Confidence-aware contrastive learning for selective classification. In Proceedings of the 41st International Conference on Machine Learning, pages 53706--53729, 2024
2024
-
[23]
Xu and Z.-H
M. Xu and Z.-H. Zhou. Incomplete label distribution learning. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3175--3181, 2017
2017
-
[24]
N. Xu, Y. Liu, and X. Geng. Label enhancement for label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 33 0 (4): 0 1632--1643, 2021
2021
-
[25]
J. Yuan, S. Chen, Y. Zhang, Z. Shi, X. Geng, J. Fan, and Y. Rui. Graph attention transformer network for multi-label image classification. ACM Transactions on Multimedia Computing, Communications and Applications, 19 0 (4): 0 1--16, 2023
2023
-
[26]
D. Zeng, E. Zha, J. Kuang, and Y. Shen. Multi-label text classification based on semantic-sensitive graph convolutional network. Knowledge-Based Systems, 284: 0 111303, 2024
2024
-
[27]
Zhang and Z.-H
M.-L. Zhang and Z.-H. Zhou. Multilabel neural networks with applications to functional genomics and text categorization. IEEE Transactions on Knowledge and Data Engineering, 18 0 (10): 0 1338--1351, 2006
2006
-
[28]
Zhang and Z.-H
M.-L. Zhang and Z.-H. Zhou. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26 0 (8): 0 1819--1837, 2014
2014
-
[29]
Zhang, Q.-W
M.-L. Zhang, Q.-W. Zhang, J.-P. Fang, Y.-K. Li, and X. Geng. Leveraging implicit relative labeling-importance information for effective multi-label learning. IEEE Transactions on Knowledge and Data Engineering, 33 0 (5): 0 2057--2070, 2021
2021
-
[30]
Zhang and M
P. Zhang and M. Wu. Multi-label supervised contrastive learning. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pages 16786--16793, 2024
2024
-
[31]
Zhang, R
S. Zhang, R. Xu, C. Xiong, and C. Ramaiah. Use all the labels: A hierarchical multi-label contrastive learning framework. In Proceedings of the 35th IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16639--16648, 2022
2022
-
[32]
Zheng, J
Q. Zheng, J. Zhu, H. Tang, X. Liu, Z. Li, and H. Lu. Generalized label enhancement with sample correlations. IEEE Transactions on Knowledge and Data Engineering, 35 0 (1): 0 482--495, 2023
2023
-
[33]
Zhong, M
Z. Zhong, M. Hirano, K. Shimada, K. Tateishi, S. Takahashi, and Y. Mitsufuji. An attention-based approach to hierarchical multi-label music instrument classification. In Proceedings of the 31st IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1--5, 2023
2023
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.