Pith. sign in

REVIEW 3 major objections 5 minor 19 references

SHREWD: Semantic Hierarchy-based Relational Embeddings for Weakly-supervised Deep Hashing

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Weakly supervised binary hash codes can match hierarchical retrieval quality without class labels.

desk verdict A useful incremental combination of semantic-hierarchy distances and a binarization loss, but the central binarization mechanism is underspecified and needs a cleaner derivation before the headline claim is fully established. read the letter →

arxiv 1908.05602 v1 pith:XWCMD42P submitted 2019-08-12 cs.IR cs.AIcs.CVcs.ITcs.LGmath.IT

Heikki Arponen , Tom E Bishop This is my paper · ORCID
classification cs.IRcs.AIcs.CVcs.ITcs.LGmath.IT
keywords deephashingsemantichierarchyweaklysupervisedlearningbinaryhashcodesKullback-Leiblerdivergenceimageretrievalembeddinghierarchical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a deep hashing system needs class labels at all, or whether the semantic distances between labels suffice. It proposes SHREWD, a loss that matches the Manhattan distances of neural embeddings to distances derived from a semantic hierarchy, plus an empirical KL-divergence term that pushes embeddings toward a beta(0.1,0.1) target so they can be binarized without losing retrieval quality. On CIFAR-100 and ILSVRC-2012 the method reaches mAHP scores of 0.8690 and 0.8676 with 64-bit binary codes, improving on a continuous-embedding hierarchy baseline. The practical point is that binary code retrieval can be trained from relational similarity alone, without explicit class membership, and that the right regularizer makes the binarization step nearly lossless.

What carries the argument

The load-bearing object is the two-term loss in Eq. 1. Lsim (Eq. 2) is a weighted absolute difference between normalized Manhattan distances in embedding space and normalized semantic distances d(yb,yb') from the label hierarchy, with weights wbb' that favor close pairs such as cat-dog over distant pairs such as cat-moon. LKL (Eq. 3) estimates KL divergence from the batch embedding distribution to a beta(0.1,0.1) target using nearest-neighbor distances, and thereby serves as a trainable binarization-and-spread regularizer. Together they let gradients flow continuously while producing embeddings that, after quantization, have Hamming distances aligned with semantic distances.

What would settle it

Sample vectors from a known distribution in dimensions 8, 32, and 128, compute Eq. 3 for increasing batch sizes and compare it to the exact KL divergence to the beta(0.1,0.1) target; if the loss falls while the true KL rises, or if the estimate is strongly biased, the binarization mechanism claimed for LKL is not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that in weakly supervised hashing, explicit class labels are unnecessary: a network trained only to reproduce hierarchical distances between labels, and regularized toward binary values, learns hash codes whose Hamming distances preserve those semantics. The mechanism is the combined loss L = Lsim + λ1LKL + λ2Lcls, where Lsim compares normalized Manhattan distances of embeddings with normalized semantic distances, and LKL is an empirical KL-divergence loss based on the Kozachenko-Leonenko entropy estimator targeting a beta(0.1,0.1) distribution. Ablations show that LKL is what keeps binary codes competitive: without it, hierarchical precision drops sharply after binarization; with it, 64-bit SHREWD achieves mAHP 0.8690 on CIFAR-100 and 0.8676 on ILSVRC-2012.

Load-bearing premise

The load-bearing premise is that the nearest-neighbor ratio in Eq. 3 is a reliable estimate of KL divergence to the beta(0.1,0.1) target in the embedding dimension, despite the fact that the paper does not derive the estimator or show it is unbiased in high dimensions.

Editorial extensions

If this is right

  • If the central claim is right, hierarchical retrieval metrics like mAHP can be optimized directly with binary codes, so a deployed system can use Hamming-distance search instead of floating-point nearest neighbors.
  • Weak supervision works: any target similarity metric d(y,y') — tag co-occurrence, user preference ranks, word-embedding distances — can replace the class hierarchy, since the loss never needs the class label itself.
  • Binarization need not be a post-hoc penalty: the KL term makes quantization part of training, which is what allows binary codes to keep the hierarchy structure.
  • The paper's ablations show that adding the KL term to existing hashing losses should reduce the gap between continuous and binary retrieval quality, even without semantic hierarchies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Eq. 3 is described as an empirical KL divergence, but the paper omits the constants and dimension-dependent terms of the Kozachenko-Leonenko estimator; in high dimensions the quantity being minimized may be a biased surrogate. A controlled low-dimensional validation could settle whether the binarization effect comes from true entropy matching or from a simpler variance-shrinki
  • Editorial inference: Because only pairwise distances enter Lsim, the method should transfer directly to tag- or caption-based supervision, where the hierarchy is replaced by any learned metric on labels; this is a testable extension the paper mentions but does not run.
  • Editorial inference: The sharp drop between 16-bit and 64-bit SHREWD (mAHP 0.7478 vs 0.8690) suggests there is a minimum code length below which a hierarchy of 100 classes cannot be embedded; finding that scale as a function of hierarchy depth would inform practical code-length choice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SHREWD (and its supervised variant SHRED), a deep hashing method for image retrieval that replaces the usual binary class-similarity supervision with a semantic distance derived from a label hierarchy (e.g. WordNet). The loss is L = Lsim + λ1 LKL + λ2 Lcls, where Lsim matches normalized Manhattan distances between image embeddings to semantic label distances, LKL is an empirical KL-divergence term meant to push embeddings toward a beta(0.1, 0.1) distribution and thereby make binarization lossless, and Lcls is an optional cross-entropy classifier term. The paper evaluates on CIFAR-100 and ILSVRC-2012, reporting mAHP scores at 64 and 128 bits, and shows an ablation in which removing LKL sharply degrades binary-code retrieval precision.

Significance. If the results hold, the paper would make a useful contribution to semantic hashing: it shows that a weakly supervised objective based on label similarities rather than class labels can produce compact binary codes with good hierarchical retrieval, and the ablation in Table 1 and Figure 1 is informative about the role of the binarization term. The core idea of combining a semantic distance-matching loss with a distribution-matching binarization loss is reasonable, and the reported improvements over continuous-embedding baselines at 64-bit Hamming retrieval are interesting. However, the central derivation of the KL term is not supplied, the experimental comparisons mix incompatible settings, and no uncertainty estimates are reported, so the strength of the empirical claims is not yet established.

major comments (3)
  1. [Section 3, Table 1] The LKL term is presented as an empirical KL divergence based on the Kozachenko-Leonenko estimator, but Eq. (3) is not a valid estimator as written. First, the KL divergence in d dimensions contains a dimension-dependent entropy term, and the Kozachenko-Leonenko plug-in has a multiplicative d factor; Eq. (3) omits this, so the relative scale of LKL to Lsim changes with hash length unless λ1 is retuned per dimension. Second, KL(p||q) is finite only if the support of p is contained in the support of q; q = beta(0.1,0.1) is supported on [0,1]^d, but no activation or rescaling of the hashing layer is specified, so if outputs are unbounded or tanh-valued the target KL is infinite and the finite-sample objective is dominated by boundary effects. Third, the term log(ν(ẑ_b; z)) must use a leave-one-out estimate excluding ẑ_b itself, otherwise it is identically zero; this convention is not stated. Because Tables 1 and 3 and Figure 1 attribute most of the binary-code improvement to LKL, the mechanism behind the headline result is not established. These issues are fixable (derive the estimator, specify the activation, state the leave-one-out convention), but they are load-bearing for the paper's explanation of why binarization works.
  2. [Section 3, Table 1] The headline comparison is not apples-to-apples. Rows marked † use non-quantized continuous embeddings, while SHREWD/SHRED report retrieval with binary Hamming codes, and the Zhe et al. row is measured with mAHP@2500 rather than the mAHP@250 used elsewhere, as the footnote acknowledges. The paper nevertheless presents a single mAHP column and concludes 'new state of the art' results. This conclusion is not supported by the table as presented. Please either evaluate all methods with binarized codes under the same retrieval protocol, or clearly separate continuous and binary results and exclude the incompatible metric.
  3. [Section 3, Tables 1-3] No error bars, confidence intervals, or multiple-run statistics are reported for any of the experiments. Since training is stochastic and some differences are small (for example SHREWD at 64 vs 128 bits in Table 2, and the ILSVRC mAHP@250 values in Table 3), the reader cannot assess whether the reported improvements are statistically meaningful. Reporting means and standard deviations over at least three runs would substantiate the central empirical claim.
minor comments (5)
  1. [Section 2.2] The target distribution beta(0.1,0.1) is continuous, yet Eq. (3) uses a finite sample z 'of e.g. size B' of vectors from the target distribution; the paper should state how many target samples are drawn per batch and whether they are resampled every step.
  2. [Section 2.1, Eq. (2)] The notation d(y_b, y_b') is used for ground-truth semantic distance, but the weight w_bb' is written as γ^ρ/(γ + d(y_b,y_b'))^ρ, which suggests d is a distance; the paper should clarify whether d is a distance or a similarity, since the normalization in the loss treats it as a distance.
  3. [Section 3] No training hyperparameters are reported (optimizer, learning rate, batch size, epochs, weight decay, data augmentation), which makes reproduction difficult even though the architecture is stated.
  4. [References] The in-text citation 'Jin (2018)' does not match the reference list entry 'Sheng Jin. Unsupervised Semantic Deep Hashing. March 2018'; the reference should include the publication venue.
  5. [Table 1 caption] The use of † and ‡ symbols in Table 1 is confusing: † is defined as non-quantized codes, but ‡ is defined only indirectly via the footnote for Zhe et al.; consider using separate columns for continuous and binary results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the method is trained on fixed semantic similarities and evaluated on external held-out benchmark images; no fitted constant is fed back into the claimed result.

full rationale

The derivation chain is self-contained and externally grounded. The semantic similarity loss (Eq. 2) is computed from a fixed, pre-supplied hierarchy distance d(y_b, y'_b), and the KL loss (Eq. 3) is regularized toward a fixed beta(0.1,0.1) target distribution; neither term is fit to the retrieval scores that are later reported. The headline results (Tables 1 and 3) are measured on held-out images from CIFAR-100 and ILSVRC-2012 using Hamming distances of binarized codes, so the predictions do not reduce by construction to a fitted input. The only shared resource between training and evaluation is the WordNet hierarchy, but that is the intended definition of the retrieval ground truth for the mAHP metric, not a fitted parameter smuggled into the result. There is no load-bearing self-citation: Kozachenko-Leonenko is cited as an external entropy estimator, and the paper's own ablations vary the loss terms rather than importing an unverified uniqueness claim. Concerns about whether Eq. 3 is a correctly specified KL estimator are correctness or empirical-validation issues, not circularity, because the loss term is proposed and tested against external benchmarks rather than being defined in terms of the result it predicts.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The paper contributes a loss combination rather than a new physical entity. Its central empirical claim rests on several hand-set or unreported hyperparameters and on the assumption that the provided semantic distances and the approximate KL loss are reliable. No code or full training configuration is supplied.

free parameters (7)
  • lambda1 (KL loss weight) = not reported
    Appears in Eq. 1 as weight on LKL; value is not given in the paper, yet the ablation shows LKL is necessary for the result. The optimal balance is tuned but undisclosed.
  • lambda2 (classification loss weight) = not reported
    Appears in Eq. 1 as weight on Lcls for SHRED; no value or selection procedure reported.
  • gamma (similarity weighting parameter) = 0.1
    Set by hand in wbb' = gamma^rho / (gamma + d)^rho; no sensitivity analysis.
  • rho (similarity weighting exponent) = 2
    Set by hand in the same weight; no derivation or sensitivity analysis.
  • alpha = beta (beta target distribution parameters) = 0.1
    Chosen so the beta target is concentrated near binary values; no sensitivity analysis.
  • tau_z and tau_y (distance normalization scales) = batch-estimated, method unspecified
    Appear in Lsim to normalize Manhattan and semantic distances; the estimation procedure is not described, and batch dependence affects gradient scale.
  • Unreported optimization hyperparameters (learning rate, optimizer, epochs, batch size, weight decay, augmentation) = not reported
    No training configuration is given, so the reported metrics depend on undisclosed settings.
assumptions (6)
  • domain assumption A meaningful semantic distance d(y,y') is available for every training pair and is a reliable proxy for perceptual retrieval similarity.
    Introduced in Section 2: 'we assume that there exists an estimate of similarity between targets d(y,y')'. The entire Lsim term matches this supplied metric, so noisy or missing distances would corrupt the hash codes.
  • domain assumption The nearest-neighbor distance ratio in Eq. 3 is a valid and useful empirical estimate of KL divergence to the beta(0.1,0.1) target distribution.
    Section 2.2 cites Kozachenko-Leonenko (1987) for entropy but omits constants and dimension factors, and no proof or simulation is provided that this gradient matches the target distribution.
  • domain assumption Binarization by sign thresholding of the continuous embeddings preserves retrieval quality when the KL loss is active.
    Claimed in Section 2.2 and demonstrated only empirically in Figure 1; no quantization error analysis is given.
  • domain assumption The WordNet hierarchy used for CIFAR-100 and ILSVRC-2012 is an appropriate semantic distance for the retrieval task.
    Inherited from Barz and Denzler (2018); the paper does not validate the hierarchy against alternative similarity measures or human judgments.
  • domain assumption Minimizing Manhattan distance between continuous codes transfers to Hamming distance between binarized codes.
    Section 2.1 chooses Manhattan distance because final retrieval uses Hamming distance, but no argument is given that L1 closeness before thresholding implies Hamming closeness after thresholding.
  • standard math Kozachenko-Leonenko entropy estimator properties hold for the embedding distribution and beta target as used.
    The loss in Eq. 3 extends the entropy estimator to a KL-difference estimator; the background theorem is standard, but the extension is not proved.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SHREWD: Semantic Hierarchy-based Relational Embeddings for Weakly-supervised Deep Hashing." pith.science (2026). https://pith.science/paper/XWCMD42P

@misc{pith2026190805602,
  author       = {Pith},
  title        = {Pith review of: SHREWD: Semantic Hierarchy-based Relational Embeddings for Weakly-supervised Deep Hashing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWCMD42P}},
  note         = {Machine review of arXiv:1908.05602}
}
read the original abstract

Using class labels to represent class similarity is a typical approach to training deep hashing systems for retrieval; samples from the same or different classes take binary 1 or 0 similarity values. This similarity does not model the full rich knowledge of semantic relations that may be present between data points. In this work we build upon the idea of using semantic hierarchies to form distance metrics between all available sample labels; for example cat to dog has a smaller distance than cat to guitar. We combine this type of semantic distance into a loss function to promote similar distances between the deep neural network embeddings. We also introduce an empirical Kullback-Leibler divergence loss term to promote binarization and uniformity of the embeddings. We test the resulting SHREWD method and demonstrate improvements in hierarchical retrieval scores using compact, binary hash codes instead of real valued ones, and show that in a weakly supervised hashing setting we are able to learn competitively without explicitly relying on class labels, but instead on similarities between labels.

Figures

Figures reproduced from arXiv: 1908.05602 by the authors.

Figure 1
Figure 1. Hierarchical precision @k for CIFAR-100 (left) and ILSVRC-2012 (right) for 64-bit [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

  1. [1]

    Hierarchy-based Image Embeddings for Semantic Image Retrieval

    Bj \"o rn Barz and Joachim Denzler. Hierarchy-based Image Embeddings for Semantic Image Retrieval . September 2018

    work page 2018

  2. [2]

    HashNet: Deep Learning to Hash by Continuation

    Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S Yu. HashNet: Deep Learning to Hash by Continuation . February 2017

    work page 2017

  3. [3]

    Hierarchical semantic indexing for large scale image retrieval

    Jia Deng, Alexander C Berg, and Fei-Fei Li. Hierarchical semantic indexing for large scale image retrieval. CVPR, pp.\ 785--792, 2011

    work page 2011

  4. [4]

    Mean Local Group Average Precision (mLGAP): A New Performance Metric for Hashing-based Retrieval

    Pak Lun Kevin Ding, Yikang Li, and Baoxin Li. Mean Local Group Average Precision (mLGAP): A New Performance Metric for Hashing-based Retrieval . November 2018

    work page 2018

  5. [5]

    DeViSE - A Deep Visual-Semantic Embedding Model

    Andrea Frome, Gregory S Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. DeViSE - A Deep Visual-Semantic Embedding Model. NIPS, 2013

    work page 2013

  6. [6]

    Similarity Search in High Dimensions via Hashing

    Aristides Gionis, Piotr Indyk, and Rajeev Motwani. Similarity Search in High Dimensions via Hashing. VLDB, 1999

    work page 1999

  7. [7]

    Unsupervised Semantic Deep Hashing

    Sheng Jin. Unsupervised Semantic Deep Hashing . March 2018

    work page 2018

  8. [8]

    L. F. Kozachenko and N. N. Leonenko. Sample Estimate of the Entropy of a Random Vector . Probl. Peredachi Inf., 23:2 (1987), 9–16, 1987

    work page 1987

Show all 19 references
  1. [9]

    Learning multiple layers of features from tiny images

    A Krizhevsky and G Hinton. Learning multiple layers of features from tiny images . 2009

    2009

  2. [10]

    Dual Asymmetric Deep Hashing Learning

    Jinxing Li, Bob Zhang, Guangming Lu, and David Zhang. Dual Asymmetric Deep Hashing Learning . January 2018

    2018

  3. [11]

    Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks

    Xu Sun, Bingzhen Wei, Xuancheng Ren, and Shuming Ma. Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks . October 2017

    2017

  4. [12]

    Supervised Deep Hashing for Hierarchical Labeled Data

    Dan Wang, Heyan Huang, Chi Lu, Bo-Si Feng, Liqiang Nie, Guihua Wen, and Xian-Ling Mao. Supervised Deep Hashing for Hierarchical Labeled Data . April 2017

    2017

  5. [13]

    A Survey on Learning to Hash

    Jingdong Wang, Ting Zhang, Jingkuan Song, Nicu Sebe, and Heng Tao Shen. A Survey on Learning to Hash . June 2016

    2016

  6. [14]

    Hierarchical Hashing for Image Retrieval

    Cheng Yan, Xiao Bai, Jun Zhou, and Yun Liu. Hierarchical Hashing for Image Retrieval . In Computer Vision, pp.\ 111--125. Springer, Singapore, Singapore, October 2017

    2017

  7. [15]

    Semantic Hierarchy Preserving Deep Hashing for Large-scale Image Retrieval

    Xuefei Zhe, Le Ou-Yang, Shifeng Chen, and Hong Yan. Semantic Hierarchy Preserving Deep Hashing for Large-scale Image Retrieval . January 2019

    2019

  8. [16]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  9. [17]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  10. [18]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  11. [19]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.