REVIEW 3 major objections 6 minor 36 references
Harnessing Superclasses for Learning from Hierarchical Databases
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A weighted superclass loss is a proper scoring rule exactly when the tree is balanced.
desk verdict Solid, moderate-impact paper: correct math, careful experiments, and the main caveat is an untested but clearly stated assumption about hierarchy quality. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a balanced weighted tree, a rooted class tree with positive weights on every node except the root such that each leaf-to-root path has total weight $1/2$. The loss is built from these weights as a weighted sum of log-probabilities over all ancestor superclass masses. The paper's exponential weighting scheme, $w_j = \left(\frac12 - \sum_{k\in a(p(j))} w_k\right)\frac{1-q}{1-q^{h(j)+1}}$, lets one interpolate from coarse-focused ($q<1$) to leaf-focused ($q>1$) training. The proof that balanced weights are necessary and sufficient for properness is the load-bearing identity: $\sum_{j\in a(k)} w_j$ is independent of the leaf $k$ if and only if the weighting is balanced.
What would settle it
Construct a training set with a class tree whose internal nodes group visually dissimilar classes (for example, by random assignment), train with the hierarchical loss, and compare hierarchical distance and accuracy to flat cross-entropy. If the loss still helps, balanced weighting alone explains the gains; if it hurts or shows no benefit, the hierarchy-quality assumption is essential to the paper's experimental claims.
Extended reading notes
Core claim
On a weighted class tree $T$, the paper defines the loss $L_T(f,y) = -\sum_{j\in a(y)} w_j \log(\sum_{k\in v_j} f_k)$, where $a(y)$ lists the true class and its superclass ancestors and $v_j$ is the set of leaf classes under node $j$. Proposition 3 states that this loss is a proper scoring rule if and only if the tree weighting is balanced, meaning the weights along the path from the root to any leaf sum to the same constant, here $1/2$. The proof evaluates the gradient of the expected loss at the true posterior $\pi$ and shows it is constant across classes precisely under balanced weights. The paper further shows that the hierarchical cross-entropy loss used in prior work is a special case of this family, and it introduces two hierarchy-aware evaluation measures: a coarsening accuracy curve and a Wasserstein distance computed on the tree.
Load-bearing premise
The load-bearing premise is that the class hierarchy given with the data reflects real semantic or visual similarity between classes, so that rewarding correct superclass predictions teaches the model useful structure rather than arbitrary categories.
Editorial extensions
If this is right
- Any softmax classifier can become hierarchy-aware by swapping the loss function; no new outputs, branches, or post-processing are needed.
- On small training sets, the loss achieves lower hierarchical distance and fewer coarse errors than cross-entropy while matching its flat accuracy.
- Because all granularities are scored by one proper scoring rule, there is no need to tune a trade-off between coarse and fine objectives.
- The family includes the hierarchical cross-entropy loss of prior work, which therefore inherits the properness guarantee.
- The extra computation over cross-entropy is negligible, so the method is essentially free at training time.
Reading between the lines
- A direct test of the domain assumption would be to shuffle the class tree or use a contrived hierarchy; the paper's improvement should shrink or vanish in that setting.
- The loss's ancestor-scoring structure suggests a natural extension to partially labeled data where only a superclass is known for some examples.
- The coarsening accuracy curve and the tree-Wasserstein distance could become standard evaluation metrics for any hierarchical classifier, not just this loss.
- The balanced-weighting condition could be used as a design principle when building or pruning hierarchies for new datasets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a family of hierarchical classification losses for softmax classifiers. For a weighted class tree T, the loss L_T(f,y) = -Σ_{j∈a(y)} w_j log(Σ_{k∈v_j} f_k) is shown to be a proper scoring rule if and only if the tree weighting is balanced, i.e., every leaf-to-root path has the same total weight. An exponential weighting scheme with one hyperparameter q is introduced, and the hierarchical cross-entropy (HXE) of Bertinetto et al. is shown to be a special balanced case. Experiments on iNaturalist19, TinyImageNet, and ImageNet with ResNet50 and MobileNetV3 compare the method with cross-entropy and HXE across training-set sizes, using hierarchical distance, a Wasserstein softmax metric, and a coarsening-accuracy curve. The main empirical finding is that, for small training sets, the loss reduces hierarchical distance and coarse errors without degrading fine-grained accuracy.
Significance. If the result holds, the paper makes a useful conceptual contribution: it unifies a family of hierarchical losses under a proper-scoring-rule condition and provides a simple drop-in loss for any softmax architecture. The mathematical core is sound: Proposition 3 is correct, the balanced-weighting condition is derived without fitted constants, and the rewriting of HXE as a balanced weighted loss is elegant. The source code is provided, and the additional computational cost is negligible. The main weakness is that the empirical benefit depends on the hierarchy matching visual similarity, which is stated as a condition in Section 2 but never stress-tested; all benchmarks use taxonomies known to be visually meaningful, and the same tree defines both the training loss and the evaluation metrics.
major comments (3)
- [Section 2 and Section 5.1] The paper's central empirical claim — that the loss reduces hierarchical distance and coarse errors without accuracy loss — is only demonstrated on three datasets whose taxonomies (iNaturalist taxonomy, WordNet-based trees) are known to align with visual similarity. The paper explicitly conditions on this in Section 2 ('can help improve performance when class similarity in the hierarchy matches visual similarity'), but it never tests a mismatched, shuffled, or corrupted hierarchy. Because the same tree defines both the training loss and the evaluation metrics, the reported gains could partly reflect the loss optimizing the evaluation criterion rather than a genuine improvement in visual structure. I request an experiment with a perturbed or arbitrary hierarchy, or, failing that, a clear statment that all empirical claims are conditional on hierarchy quality.
- [Section 3.3 and Abstract] The claim that properness 'eliminates the need for a performance trade-off between different granularities' is stronger than what is established. Properness guarantees that, in the population, the true posterior minimizes every balanced loss. However, Section 5.3 and Figure 4 show that the hyperparameter q induces a clear finite-sample trade-off between fine-grained accuracy and hierarchical distance. The paper should qualify this claim to the population level or discuss the finite-sample trade-off explicitly.
- [Abstract and Section 5.1] The abstract states that the method 'improves accuracy and reduces the number of coarse errors,' but Section 5.1 reports that the improvement in hierarchical distance is 'not paid by a loss in standard accuracy since we obtain equivalent standard accuracies.' The evidence supports improved hierarchical/coarsened accuracy, not improved fine-grained accuracy. The abstract should be revised to specify which accuracy is meant, to avoid overclaiming.
minor comments (6)
- [Equation (3)] Equation (3) is undefined at q=1 because the factor (1-q)/(1-q^{h(j)+1}) becomes 0/0; the authors discuss q=1 as a meaningful case ('all weights are roughly equal'). Please define the formula by continuity or explicitly restrict to q≠1 and treat q=1 as a limit.
- [Section 3.2] The weighting scheme is introduced for q≥0, but for q=0 all nodes at depth greater than one receive zero weight, contradicting the statement that weights are 'strictly positive elsewhere' except the root. Please restrict to q>0 or discuss the degenerate q=0 case separately.
- [Appendix A, proof of Proposition 3] The proof shows that the gradient of the expected loss vanishes at the true posterior π, but it does not explicitly state that L_T(f,y) is convex in f. Since this convexity (a positively weighted sum of -log of linear forms) is what makes the vanishing-gradient condition sufficient for a global minimum on the simplex, please add a sentence making it explicit.
- [Section 5.3, Figure 4] The claim that 'for any parameter α, there is a model with our weighting scheme that is strictly better on both metrics' goes beyond the finite grid of tested hyperparameters. Please soften the statement to refer to the tested range or provide a formal argument for the dominance.
- [Section 4.1] The definition of the subtree Tτ used in the coarsening accuracy curve is terse; please clarify that pruning removes all descendants of the selected nodes and illustrate the threshold inequalities with a small example.
- [Section 4.2] For the ImageNet experiment, the networks are pre-trained on ImageNet-1K and then fine-tuned on a downsampled version of the same label set. Although this is a standard transfer setting, please state explicitly that the model has already seen the fine-grained classes and discuss any potential label leakage when interpreting the low-data regime results.
Circularity Check
No significant circularity: the proper-scoring-rule theorem and the HXE equivalence are self-contained, and the single self-citation in related work is not load-bearing.
full rationale
The central derivation (Section 3, Propositions 2-4) is mathematically self-contained. L_T is defined from a weighted tree, and Proposition 3 is proven by computing the gradient of the expected loss at the true posterior; the gradient equals the path weight sum, which is constant exactly when the tree is balanced. No fitted parameter is needed for the theorem to hold; q is a user-chosen hyperparameter and the proof covers all balanced weightings. Proposition 4 rewrites the HXE loss of Bertinetto et al. and verifies balancedness by an explicit telescoping computation (the ancestor-weight sum equals exp(-alpha)), so the claim that HXE is a proper scoring rule is derived from the paper's own equations rather than imported from the cited work. The experimental evaluation uses the provided class hierarchy both for training and for hierarchical metrics, but this is a semantic choice for defining 'coarse error', not a construction in which the prediction is equal to the input. The paper explicitly states the external validity condition ('can help improve performance when class similarity in the hierarchy matches visual similarity') rather than hiding it. The only self-citation is reference [27] (Tanzi et al.), used in Related Work as an example of cascade classifiers; it plays no role in the theoretical or empirical claims and therefore is not load-bearing. The score is 1 rather than 0 only to note this harmless self-citation; there are no circular steps.
Assumptions & free parameters
free parameters (1)
- q (exponential weighting growth rate) =
0.9 and 1.2, with sensitivity range 0.8 to 1.4 in Figure 4
assumptions (3)
- domain assumption The class hierarchy T is known and correctly encodes the semantic/visual relationships among classes.
- standard math The model's softmax output f(x; θ) is a probability vector over fine-grained classes, and the data is generated from a well-defined conditional distribution P(Y|X).
- standard math The expected loss as a function of f is convex, ensuring that the zero-gradient condition in the proof of Proposition 3 characterizes a global minimum.
Cite this review
Pith. "Pith review of Harnessing Superclasses for Learning from Hierarchical Databases." pith.science (2026). https://pith.science/paper/5SWBIWTH
@misc{pith2026241116438,
author = {Pith},
title = {Pith review of: Harnessing Superclasses for Learning from Hierarchical Databases},
year = {2026},
howpublished = {\url{https://pith.science/paper/5SWBIWTH}},
note = {Machine review of arXiv:2411.16438}
}
read the original abstract
In many large-scale classification problems, classes are organized in a known hierarchy, typically represented as a tree expressing the inclusion of classes in superclasses. We introduce a loss for this type of supervised hierarchical classification. It utilizes the knowledge of the hierarchy to assign each example not only to a class but also to all encompassing superclasses. Applicable to any feedforward architecture with a softmax output layer, this loss is a proper scoring rule, in that its expectation is minimized by the true posterior class probabilities. This property allows us to simultaneously pursue consistent classification objectives between superclasses and fine-grained classes, and eliminates the need for a performance trade-off between different granularities. We conduct an experimental study on three reference benchmarks, in which we vary the size of the training sets to cover a diverse set of learning scenarios. Our approach does not entail any significant additional computational cost compared with the loss of cross-entropy. It improves accuracy and reduces the number of coarse errors, with predicted labels that are distant from ground-truth labels in the tree.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Babbar, R., Partalas, I., Gaussier, E., Amini, M.R., Amblard, C.: Learning tax- onomy adaptation in large-scale classification. J. Mach. Learn. Res.17(98), 1–37 (2016)
work page 2016
- [2]
-
[3]
In: Workshop Notes of the KDD’02 Workshop on Multi- Relational Data Mining
Blockeel, H., Bruynooghe, M., Džeroski, S., Ramon, J., Struyf, J.: Hierarchi- cal multi-classification. In: Workshop Notes of the KDD’02 Workshop on Multi- Relational Data Mining. pp. 21–35 (2002)
work page 2002
- [4]
-
[5]
arXiv preprintabs/1707.08819 (2017)
Chrabaszcz, P., Loshchilov, I., Hutter, F.: A downsampled variant of imagenet as an alternative to the CIFAR datasets. arXiv preprintabs/1707.08819 (2017)
arXiv 2017
- [6]
-
[7]
Deng, J., Berg, A.C., Li, K., Fei-Fei, L.: What does classifying more than 10,000 image categories tell us? In: Eur. Conf. Comput. Vis. (ECCV). pp. 71–84. Springer (2010)
work page 2010
-
[8]
Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H., Adam, H.: Large-scale object classification using label relation graphs. In: Eur. Conf. Comput. Vis. (ECCV). pp. 48–64 (2014)
work page 2014
Show all 36 references
-
[9]
In: Conf
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 248–255 (2009)
2009
-
[10]
In: Conf
Deselaers, T., Ferrari, V.: Visual and semantic similarity in ImageNet. In: Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 1777–1784 (2011)
2011
-
[11]
Journal of the Royal Statistical Society Series B: Statistical Methodology 74(3), 569–592 (2012)
Evans, S.N., Matsen, F.A.: The phylogenetic Kantorovich–Rubinstein metric for environmental sequence samples. Journal of the Royal Statistical Society Series B: Statistical Methodology 74(3), 569–592 (2012)
2012
-
[12]
In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T
Garg, A., Sani, D., Anand, S.: Learning hierarchy aware features for reducing mistake severity. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Eur. Conf. Comput. Vis. (ECCV). pp. 252–267. Springer (2022)
2022
-
[13]
Journal of the American statistical Association102(477), 359–378 (2007)
Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estima- tion. Journal of the American statistical Association102(477), 359–378 (2007)
2007
-
[14]
Goo, W., Kim, J., Kim, G., Hwang, S.J.: Taxonomy-regularized semantic deep convolutional neural networks. In: Eur. Conf. Comput. Vis. (ECCV). pp. 86–101 (2016). https://doi.org/10.1007/978-3-319-46475-6_6
2016 doi
-
[15]
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Int. Conf. Mach. Learn. (ICML). pp. 1321–1330 (2017)
2017
-
[16]
In: Conf
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 770–778 (2016)
2016
-
[17]
Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for MobileNetV3. In: Int. Conf. Comput. Vis. (ICCV). pp. 1314–1324 (2019)
2019
-
[18]
Karthik, S., Prabhu, A., Dokania, P.K., Gandhi, V.: No cost likelihood manipula- tion at test time for making better mistakes in deep networks. In: Int. Conf. Learn. Represent. (ICLR) (2021)
2021
-
[19]
Kingma, D.P., Ba, J.L.: Adam: Amethod for stochastic optimization. In: Int. Conf. Learn. Represent. (ICLR) (2014)
2014
-
[20]
Koo, J., Klabjan, D., Utke, J.: Combined convolutional and recurrent neural net- works for hierarchical classification of images. In: Int. Conf. on Big Data (BigData). pp. 1354–1361 (2020)
2020
-
[21]
Le, T., Yamada, M., Fukumizu, K., Cuturi, M.: Tree-sliced variants of Wasserstein distances. Adv. Neural Inform. Process. Syst. (NeurIPS)32 (2019) 16 N. Urbani et al
2019
-
[22]
In: Conf
Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., Yu, S.X.: Large-scale long-tailed recognition in an open world. In: Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 2537–2546 (2019). https://doi.org/10.1109/CVPR.2019.00264
2019
-
[23]
In: Conf
Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. In: Conf. Comput. Vis. Pattern Recog. (CVPR). vol. 2, pp. 2161–2168 (2006)
2006
-
[24]
Foundations and Trends in Machine Learning11(5-6), 355–607 (2019)
Peyré, G., Cuturi, M.: Computational optimal transport. Foundations and Trends in Machine Learning11(5-6), 355–607 (2019)
2019
-
[25]
In: Conf
Salakhutdinov, R., Torralba, A., Tenenbaum, J.B.: Learning to share visual ap- pearance for multiclass object detection. In: Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 1481–1488 (2011).https://doi.org/10.1109/CVPR.2011.5995720
2011
-
[26]
Srivastava, N., Salakhutdinov, R.: Discriminative transfer learning with tree-based priors. In: Adv. Neural Inform. Process. Syst. (NeurIPS). pp. 2094–2102 (2013)
2013
-
[27]
European Journal of Radiology133, 109373 (2020)
Tanzi, L., Vezzetti, E., Moreno, R., Aprato, A., Audisio, A., Massè, A.: Hi- erarchical fracture classification of proximal femur x-ray images using a multi- stage deep learning approach. European Journal of Radiology133, 109373 (2020). https://doi.org/https://doi.org/10.1016/...
2020
-
[28]
In: Conf
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The iNaturalist species classification and detection dataset. In: Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 8769–8778 (2018)
2018
-
[29]
arXiv preprint abs/2112.02353 (2021), https://arxiv.org/abs/2112.02353
Wang, R., cai, D., Xiao, K., Jia, X., Han, X., Meng, D.: Label hierarchy tran- sition: Modeling class hierarchies to enhance deep classifiers. arXiv preprint abs/2112.02353 (2021), https://arxiv.org/abs/2112.02353
2021 arXiv
-
[30]
PLOS ONE 14(12), 1–17 (12 2019)
Wu, C., Tygert, M., LeCun, Y.: A hierarchical loss and its problems when classifying non-hierarchically. PLOS ONE 14(12), 1–17 (12 2019). https:// doi.org/10.1371/journal.pone.0226222, https://doi.org/10.1371/journal. pone.0226222
2019 doi
-
[31]
Wu,T.,Morgado,P.,Wang,P.,Ho,C.,Vasconcelos,N.:Solvinglong-tailedrecogni- tion with deep realistic taxonomic classifier. In: Eur. Conf. Comput. Vis. (ECCV). pp. 171–189 (2020).https://doi.org/10.1007/978-3-030-58598-3_11
2020 doi
-
[32]
arXiv preprint abs/1705.11105 (2017)
Wu, Z., Saito, S.: HiNet: Hierarchical classification with neural network. arXiv preprint abs/1705.11105 (2017)
2017 arXiv
-
[33]
Xie, S., Yang, T., Wang, X., Lin, Y.: Hyper-class augmented and regularized deep learningforfine-grainedimageclassification.In:Conf.Comput.Vis.PatternRecog. (CVPR). pp. 2645–2654 (2015).https://doi.org/10.1109/CVPR.2015.7298880
2015
-
[34]
Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., Yu, Y.: HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. In: Int. Conf. Comput. Vis. (ICCV). pp. 2740–2748 (2015)
2015
-
[35]
Zhao, B., Fei-Fei, L., Xing, E.P.: Large-scale category structure aware image cate- gorization. In: Adv. Neural Inform. Process. Syst. (NeurIPS). pp. 1251–1259 (2011)
2011
-
[36]
Zhu, X., Bain, M.: B-CNN: branch convolutional neural network for hierarchical classification. arXiv preprintabs/1709.09890 (2017) A Proofs Proof of Proposition 1.For the first part of the proposition, we state that, for any leaf vk, we have: X j∈a(k) wj = wk + 1 2 − 1 2 −...
2017 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.