REVIEW 4 major objections 5 minor 44 references
Enhancing Text-Based Hierarchical Multilabel Classification for Mobile Applications via Contrastive Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Level-wise contrastive pretraining of a hierarchical multilabel classifier produces more separable app embeddings, the best app-dataset F1 among the compared methods, and a 10.70% KS gain in a deployed credit-risk model.
desk verdict HMCL's level-wise negative sampling is a real but incremental idea; the headline numbers are undercut by test-set-based strategy selection and an unquantified downstream KS gain. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is HMCL, a contrastive pretraining scheme for the text encoder, coupled with HMCN, the hierarchical multilabel classifier it initializes. HMCL's load-bearing piece is negative sampling: at each level of the taxonomy, for every active anchor label it samples a same-level negative label and then an instance carrying that negative label but not the anchor label; the Level strategy keeps the negative pool inside the anchor's own level, avoiding the leaf bias of All and the narrowness of Sibling. The contrastive loss scores each pair with a sigmoid rather than a softmax, because one pair can be both positive and negative under different anchor labels. HMCN then combines a global branch over all labels with a local branch that predicts level by level and transfers information downward through cross-attention, trained with focal loss plus a path-regularization term that penalizes a child being more probable than its parent.
What would settle it
Re-run the app and RCV1 comparisons with the sampling strategy selected on a separate validation fold and then evaluated on untouched test data; if Level's advantage over All and Sibling shrinks or disappears, the claimed superiority of Level is an artifact of test-set selection rather than a property of the method.
Extended reading notes
Core claim
The paper's central claim is that a hierarchical multilabel contrastive pretraining scheme (HMCL), run before training a hierarchical multilabel classification network (HMCN), produces app representations whose cosine similarities reflect label structure instead of generic text similarity. For each active label of an anchor app at each level, positives are drawn from other apps sharing that label, while negatives are drawn from labels at the same level; of the three studied strategies (All, Level, Sibling), Level is the strongest. After pretraining, HMCN—which merges a global prediction over all labels with a local level-by-level prediction that passes parent-level information downward through cross-attention—attains the best reported micro-F1 (80.75) and macro-F1 (48.62) on the app dataset, outperforming the HGCLR and HILL contrastive baselines, and remains competitive on RCV1 and WOS. The same pretraining improves both uniformity and alignment of the embeddings, and the production labels were associated with a 10.70% improvement in the KS value of a downstream fraud-risk model over a year of deployment.
Load-bearing premise
The reported gains assume that choosing the Level negative-sampling strategy after seeing its results on the same test sets used for the final tables does not inflate the numbers; no separate holdout or nested validation is reported for that choice.
Editorial extensions
If this is right
- Because HMCL runs at pretraining time and is decoupled from HMCN, the same pretrained encoder can be attached to other hierarchical classifiers and is not tied to this specific network.
- All three negative-sampling strategies improve over plain HMCN, so the benefit of hierarchical contrastive pretraining does not depend on the exact Level choice; Level is simply the strongest of the three tested.
- The sharper embedding geometry that HMCL produces (better uniformity and alignment) supports the paper's claim that the resulting labels are more useful as user-interest features in downstream profiling.
- Because app stores generally publish names, descriptions, and editorial summaries, the same approach should transfer to other app-store taxonomies.
- For the downstream business, the 10.70% KS improvement means the risk model separates likely fraud victims from other users more sharply, allowing earlier protective action.
Reading between the lines
- An obvious testable extension is to apply HMCL's Level strategy to other deep text taxonomies (product catalogs, legal codes, scientific topics) to see whether same-depth negative sampling is the part that transfers, since the paper demonstrates it on only three taxonomies.
- The 10.70% KS gain is a deployment comparison rather than an ablation that isolates the HMCL contribution; a cleaner measurement would feed the same downstream model the old and new label features and compare KS on a fixed held-out set.
- The near-zero or negative cosine similarities between unrelated apps after HMCL suggest the embeddings could also serve tasks like near-duplicate app detection or recommendation, not just classification.
- Because the same pair can be positive at one level and negative at another, the sigmoid-based loss is a mild assumption; a comparison against a softmax formulation with level-aware normalization would show whether the loss design or the sampling strategy carries the empirical gain.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HMCN, a hierarchical multilabel classification network with global and local prediction manners, cross-attention-based information passing, path regularization, and focal loss, together with HMCL, a contrastive pretraining scheme with three negative sampling strategies (All, Level, Sibling). The approach is evaluated on a Tencent App Store dataset, RCV1, and WOS. The authors report that HMCL with the Level strategy improves HMCN and outperforms SOTA baselines on the app data, gives competitive results on RCV1 and WOS, and that a deployed version improved a downstream credit-risk metric (KS) by 10.70%.
Significance. If the claims hold, the paper offers a practical recipe for hierarchical multilabel classification with text inputs: a reproducible training scheme with a contrastive pretraining stage, validated on one industrial and two public datasets, and backed by a real deployment. The paper is honest about positioning relative to large language models and includes implementation details that facilitate reproducibility. The external-dataset evaluation and comparisons with multiple existing methods are strengths. However, the evaluation methodology has a load-bearing test-set selection issue and some controlled-comparison gaps, so the significance of the empirical claims is currently tempered.
major comments (4)
- [Section 5.2, Tables 3–5] The negative sampling strategy is selected by comparing All, Level, and Sibling on the test sets of the app and RCV1 datasets, and then the chosen Level strategy is used in all remaining experiments. This is test-set selection: the final reported numbers are not independent of the selection step, and the reported improvements over HMCN and SOTA baselines are optimistic. A nested validation or a separate holdout for strategy selection is needed; because Table 2 reports validation splits, such a procedure should be feasible.
- [Section 5.2, Table 3] The text states that 'all three strategies outperform the independent HMCN significantly,' but Table 3 shows that on RCV1 micro-F1, All (87.13±0.36) is below HMCN (87.52±0.18). The improvement is therefore not robust across strategies; it depends on the particular choice of Level. The claim that HMCL enhances HMCN in general is not supported by the data as presented.
- [Section 5.3, Table 4] The app-data comparison uses different base encoders for the proposed method and the contrastive baselines: HMCN and HMCN&HMCL use RoFormer, while HGCLR and HILL are implemented with BERT, as stated in Section 5.3. The reported gains over HGCLR and HILL may reflect differences in the base encoder rather than the proposed method itself. A controlled comparison with a shared base encoder is required to attribute the improvement.
- [Section 5.5.2] The downstream result of 10.70% improvement in KS value is reported without any uncertainty measure, without a description of the baseline 'previously deployed approach' against which the improvement is measured, and without the test-set size or statistical significance. As a headline claim in the abstract and introduction, this result needs more evaluation detail to be interpretable.
minor comments (5)
- [Section 5.5.1] There is a typo in 'As illustrated in illustrated in Fig. 4'.
- [Section 5.2] The sentence 'This coincides with our analysis in Section 4.1' appears to refer to the comparison of sampling strategies, but the relevant analysis is in Section 4.2.4, not Section 4.1.
- [Section 5.4] The word 'mutlilabel' is misspelled in 'the HMCN is designed for a mutlilabel scenario.'
- [Section 4.2.3] The phrase 'The siblings of u have already excluded u by definition' is awkward; it should read 'The set of siblings of u has already excluded u by definition.'
- [Appendix A] The contrastive sampling is repeated 10, 20, and 50 times for levels 1, 2, and 3 on the app and RCV1 datasets, and 5 and 20 times for WOS; this is a free hyperparameter that is not justified or sensitivity-tested, and should at least be discussed.
Circularity Check
No circularity: the pipeline is measured against external datasets and baselines; the test-set strategy selection concern is data snooping, not derivation-by-construction.
full rationale
The paper's derivation chain is empirical rather than definitional. HMCL defines a batch-wise contrastive loss (Eq. 13) from positive and negative samples constructed by the All/Level/Sibling strategies; HMCN optimizes focal loss plus a path-regularization term (Eq. 10); the combined method is then evaluated on held-out test portions of the app dataset, RCV1, and WOS, and compared with external baselines. No parameter is fitted to a test-set quantity and then renamed as a prediction, and no result is asserted solely because the authors previously defined it so. The HMCN architecture is explicitly adapted from Wehrmann et al. [35], and the positive-sampling procedure is adopted from Zhang et al. [40]; the one self-citation in the introduction ([17], by two of the authors) is a motivational reference for downstream applications and is not load-bearing for the main claim. The nearest concern is Section 5.2, where the Level negative-sampling strategy is selected after comparing All, Level, and Sibling on the same test sets used in the final tables, and the text states that "all three strategies outperform the independent HMCN significantly" while Table 3 shows RCV1 All micro-F1 (87.13) below HMCN (87.52). This is a test-set reuse / statistical validity issue, not circularity: the reported micro-F1 and macro-F1 values are measured outcomes, not quantities forced by construction, and the downstream 10.70% KS improvement is evaluated by a separate business team on a dedicated held test set. Therefore no circular step is present.
Assumptions & free parameters
free parameters (7)
- Contrastive scaling alpha =
0.1
- Path regularization lambda =
1
- Focal loss alpha =
0.25
- Focal loss gamma =
2
- Level-dependent negative sampling repeats =
10/20/50 for app and RCV1 levels 1/2/3; 5/20 for WOS levels 1/2
- Label activation threshold =
0.5
- Training hyperparameters =
HMCL lr 1e-5, batch 8; HMCN lr 5e-3 (app) or 1e-4 (public), batch 8; decay 0.8
assumptions (5)
- domain assumption Each child label has exactly one parent, and a label can be active only when its parent is active.
- domain assumption The token at position 0 after RoFormer encodes the meaning of each text field.
- domain assumption Text fields such as name, description, and editorial comments are sufficient to determine app labels.
- domain assumption The held-out test sets used to choose the Level strategy still estimate generalization.
- ad hoc to paper The downstream 10.70% KS improvement is caused by the HMCN and HMCL labels.
Cite this review
Pith. "Pith review of Enhancing Text-Based Hierarchical Multilabel Classification for Mobile Applications via Contrastive Learning." pith.science (2026). https://pith.science/paper/NXYQXAH5
@misc{pith2026250704413,
author = {Pith},
title = {Pith review of: Enhancing Text-Based Hierarchical Multilabel Classification for Mobile Applications via Contrastive Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/NXYQXAH5}},
note = {Machine review of arXiv:2507.04413}
}
read the original abstract
A hierarchical labeling system for mobile applications (apps) benefits a wide range of downstream businesses that integrate the labeling with their proprietary user data, to improve user modeling. Such a label hierarchy can define more granular labels that capture detailed app features beyond the limitations of traditional broad app categories. In this paper, we address the problem of hierarchical multilabel classification for apps by using their textual information such as names and descriptions. We present: 1) HMCN (Hierarchical Multilabel Classification Network) for handling the classification from two perspectives: the first focuses on a multilabel classification without hierarchical constraints, while the second predicts labels sequentially at each hierarchical level considering such constraints; 2) HMCL (Hierarchical Multilabel Contrastive Learning), a scheme that is capable of learning more distinguishable app representations to enhance the performance of HMCN. Empirical results on our Tencent App Store dataset and two public datasets demonstrate that our approach performs well compared with state-of-the-art methods. The approach has been deployed at Tencent and the multilabel classification outputs for apps have helped a downstream task--credit risk management of user--improve its performance by 10.70% with regard to the Kolmogorov-Smirnov metric, for over one year.
Figures
Reference graph
Works this paper leans on
-
[1]
Rami Aly, Steffen Remus, and Chris Biemann. 2019. Hierarchical multi-label classification of text with capsule networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Work- shop. 323–330
work page 2019
-
[2]
Siddhartha Banerjee, Cem Akkaya, Francisco Perez-Sorrosal, and Kostas Tsiout- siouliklis. 2019. Hierarchical transfer learning for multi-label text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 6295–6300
work page 2019
-
[3]
Ali Cevahir and Koji Murakami. 2016. Large-scale Multi-class and Hierarchical Product Categorization for an E-commerce Giant. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers . 525–535
work page 2016
-
[4]
Haibin Chen, Qianli Ma, Zhenxi Lin, and Jiangyue Yan. 2021. Hierarchy-aware la- bel semantics matching network for hierarchical text classification. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 4370–4379
work page 2021
-
[5]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Interna- tional conference on machine learning . PMLR, 1597–1607
2020
-
[6]
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05) , Vol. 1. IEEE, 539–546
work page 2005
-
[7]
Son D Dao, Ethan Zhao, Dinh Phung, and Jianfei Cai. 2021. Multi-label image classification with contrastive learning. arXiv preprint arXiv:2107.11626 (2021)
work page Pith review arXiv 2021
-
[8]
Zhongfen Deng, Hao Peng, Dongxiao He, Jianxin Li, and Philip S Yu. 2021. HTCInfoMax: A global model for hierarchical text classification via information maximization. arXiv preprint arXiv:2104.05220 (2021)
work page Pith review arXiv 2021
Show all 44 references
-
[9]
Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
2018 arXiv
-
[10]
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pas- cal Vincent, and Samy Bengio. 2010. Why Does Unsupervised Pre-training Help Deep Learning? Journal of Machine Learning Research 11, 19 (2010), 625–660. http://jmlr.org/papers/v11/erhan10a.html
2010
-
[11]
Rafael S Gonçalves, Matthew Horridge, Rui Li, Yu Liu, Mark A Musen, Csongor I Nyulas, Evelyn Obamos, Dhananjay Shrouty, and David Temple. 2019. Use of owl and semantic web technologies at pinterest. InThe Semantic Web–ISWC 2019: 18th International Semantic Web Conference, Auck...
2019
-
[12]
Siddharth Gopal and Yiming Yang. 2013. Recursive regularization for large-scale classification with hierarchical and graphical dependencies. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. 257–265
2013
-
[13]
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in ne...
2020
-
[14]
Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research...
2010
-
[15]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 9729–9738
2020
-
[16]
Wei Huang, Enhong Chen, Qi Liu, Yuying Chen, Zai Huang, Yang Liu, Zhou Zhao, Dan Zhang, and Shijin Wang. 2019. Hierarchical multi-label text classification: An attention-based recurrent network approach. In Proceedings of the 28th ACM international conference on information an...
2019
-
[17]
Weipeng Huang, Guangyuan Piao, Raul Moreno, and Neil J. Hurley. 2020. Partially Observable Markov Decision Process Modelling for Assessing Hierarchies. In Asian Conference on Machine Learning . PMLR, 641–656
2020
-
[18]
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Advances in neural information processing systems 33 (2020), 18661– 18673
2020
-
[19]
Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S Gerber, and Laura E Barnes. 2017. Hdltex: Hierarchical deep learning for text classification. In 2017 16th IEEE international conference on machine learning and applications (ICMLA) . IEEE, 364–371
2017
-
[20]
David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research 5, Apr (2004), 361–397
2004
-
[21]
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. 2017. Focal Loss for Dense Object Detection. In 2017 IEEE International Conference on Computer Vision (ICCV) . IEEE Computer Society, Los Alamitos, CA, USA, 2999–3007. doi:10.1109/ICCV.2017.324
2017 doi
-
[22]
MA Liu, Jennifer Lewis Priestley Ph D, et al . 2018. A comparison of machine learning algorithms for prediction of past due service in commercial credit. (2018)
2018
-
[23]
Yuning Mao, Jingjing Tian, Jiawei Han, and Xiang Ren. 2019. Hierarchical text classification with reinforced label assignment. arXiv preprint arXiv:1908.10419 (2019)
2019 arXiv
-
[24]
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024)
2024 arXiv
-
[25]
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
2018 arXiv
-
[26]
Kervy Rivas Rojas, Gina Bustamante, Arturo Oncevay, and Marco A Sobrevilla Cabezudo. 2020. Efficient strategies for hierarchical text classification: External knowledge and auxiliary tasks. arXiv preprint arXiv:2005.02473 (2020)
2020 arXiv
-
[27]
Ruslan Salakhutdinov and Geoff Hinton. 2007. Learning a nonlinear embedding by preserving class neighbourhood structure. InArtificial intelligence and statistics. PMLR, 412–419
2007
-
[28]
Kazuya Shimura, Jiyi Li, and Fumiyo Fukumoto. 2018. HFT-CNN: Learning hierarchical category structure for multi-label short text categorization. In Pro- ceedings of the 2018 conference on empirical methods in natural language processing. 811–816
2018
-
[29]
Carlos N Silla and Alex A Freitas. 2011. A survey of hierarchical classification across different application domains. Data mining and knowledge discovery 22 (2011), 31–72
2011
-
[30]
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063
2024
-
[31]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)
2017
-
[32]
Ran Wang, Xinyu Dai, et al. 2022. Contrastive learning-enhanced nearest neigh- bor mechanism for multi-label text classification. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) . 672–679
2022
-
[33]
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning . PMLR, 9929–9939
2020
-
[34]
Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. 2022. Incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classification. arXiv preprint arXiv:2203.03825 (2022)
2022 arXiv
-
[35]
Jonatas Wehrmann, Ricardo Cerri, and Rodrigo Barros. 2018. Hierarchical multi- label classification networks. In International conference on machine learning . PMLR, 5075–5084
2018
-
[36]
Linli Xu, Sijie Teng, Ruoyu Zhao, Junliang Guo, Chi Xiao, Deqiang Jiang, and Bo Ren. 2021. Hierarchical multi-label text classification with horizontal and vertical category correlations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing...
2021
-
[37]
Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. 2019. Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. Advances in neural information processing systems 32 (2019)
2019
-
[38]
Fattane Zarrinkalam, Stefano Faralli, Guangyuan Piao, Ebrahim Bagheri, et al
-
[39]
Pingyue Zhang and Mengyue Wu. 2024. Multi-Label Supervised Contrastive Learning. Proceedings of the AAAI Conference on Artificial Intelligence 38, 15 (Mar. 2024), 16786–16793. doi:10.1609/aaai.v38i15.29619
2024 doi
-
[40]
Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. 2022. Use all the labels: A hierarchical multi-label contrastive learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 16660–16669
2022
-
[41]
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)
2023 arXiv
-
[42]
Jie Zhou, Chunping Ma, Dingkun Long, Guangwei Xu, Ning Ding, Haoyu Zhang, Pengjun Xie, and Gongshen Liu. 2020. Hierarchy-aware global model for hi- erarchical text classification. In Proceedings of the 58th annual meeting of the association for computational linguistics . 1106–1117
2020
-
[43]
He Zhu, Junran Wu, Ruomei Liu, Yue Hou, Ze Yuan, Shangzhe Li, Yicheng Pan, and Ke Xu. 2024. HILL: Hierarchy-aware Information Lossless Contrastive Learning for Hierarchical Text Classification. arXiv preprint arXiv:2403.17307 (2024). A Implementation Details We now specify the...
2024 arXiv
-
[2020]
Foun- dations and Trends® in Information Retrieval 14, 5 (2020), 445–617
Extracting, mining and predicting users’ interests from social media. Foun- dations and Trends® in Information Retrieval 14, 5 (2020), 445–617
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.