REVIEW 5 major objections 6 minor 33 references
Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark Image Retrieval
T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Trademark images are better retrieved as trees of learned visual words than as global image descriptors, and the paper shows the tree representation beats previous shape-based methods on the MPEG-7 CE-1 and CE-2 benchmarks.
desk verdict A genuinely new hierarchical visual-words descriptor for trademark retrieval, but the main empirical claim is compromised by test-set hyperparameter tuning and contradicted by its own F1 table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the visual hierarchy: a rooted tree per image in which each node stores a visual word and edges encode inclusion and exclusion among component shapes. A shape A is included in a shape B if A is a hole inside object B or an object inside hole B, and shapes are siblings when both are included in a common shape. The hierarchy is built by recursively extracting foreground objects and holes with border following. Nodes are labeled by a k-means visual-words codebook learned from 29-dimensional shape features (Zernike moments, circularity, average bending energy, eccentricity, convexity), so the tree mixes local geometry at its nodes with global topology in its edges. Tree comparison uses AP-TED with rename cost $\delta_r(n_a,n_b)=\operatorname{dist}_E(\lambda_a,\lambda_b)$ and insert or remove cost $\delta_x(n)=\alpha \frac{2}{m(m-1)}\sum_{i<j}\operatorname{dist}_E(\lambda_i,\lambda_j)$ with $\alpha=\min\{\log_2^{-1} L,\log_2^{-1} D\}$, where $L$ is the sibling count and $D$ is the depth. This is the mechanism that turns learned shape distances into hierarchical dissimilarity. A second codebook, obtained by mean-shift clustering of the edit-distance matrix, supplies labels that let retrieval start in the most relevant hierarchy cluster and expand to nearby clusters.
What would settle it
Compare, for a query trademark, retrieval ranks between a same-class variant whose component shape is replaced by a perceptually similar shape and a variant whose replacement shape is Euclidean-close to the original's cluster centroid but visually different; if the second variant consistently outranks the first, the edit-cost model is not capturing perceptual similarity.
Extended reading notes
Core claim
The paper's central claim is that describing a binary trademark as a tree of component shapes, rather than as a holistic image descriptor or a fixed partition of the image plane, yields a retrieval descriptor that is both geometrically specific and topologically faithful. Each connected foreground object or background hole is represented by a 29-dimensional invariant feature vector, and k-means turns those vectors into a codebook of visual words; inclusion and exclusion among the components define the tree. Two such trees are compared with AP-TED, where renaming a node costs the Euclidean distance between the corresponding visual-word centroids and insertion or removal costs scale the mean pairwise centroid distance by a depth-and-sibling factor. The paper reports that on MPEG-7 CE-1 this descriptor reaches 99.79% precision up to 55% recall, and that on MPEG-7 CE-2 it attains an F1 score of 0.89, higher than the compared CNN baseline (0.81) and higher than the other evaluated shape-descriptor methods. On CE-1, HoVW is the best method up to 80% recall and comparable to the hierarchical-region baseline from then on.
Load-bearing premise
The method's ranking collapses if the distance between two learned shape categories, measured as Euclidean distance between their cluster centers, does not match how visually different those shapes look to a person.
Editorial extensions
If this is right
- On MPEG-7 CE-1, HoVW returns the first 11 same-class images almost perfectly, with 99.79% precision through 55% recall and 72% precision at the last correlated image.
- Because the feature vectors are invariant to rotation, translation, and scale and the tree encodes topology, the descriptor withstands linear and some nonlinear transformations without partitioning the image.
- The visual-hierarchies codebook gives a fast retrieval path: only entries sharing the query's hierarchy label are compared first, and neighboring labels are added only when more results are requested.
- On MPEG-7 CE-2, HoVW's F1 of 0.89 exceeds the 0.81 of the CNN baseline, indicating that a non-neural learned structural representation can win on small binary shape datasets.
- HoVW does not read text within marks and does not apply Gestalt grouping, so its claimed advantage is scoped to graphical device marks.
Reading between the lines
- This suggests the same decompose-into-shapes-and-nesting recipe could transfer to other binary image retrieval problems, such as logo databases, diagram search, or icon sets, where inclusion and exclusion structure is meaningful.
- The precision drop at high recall is attributed by the authors to chunked retrieval from nearby hierarchy labels; an intra-label ranking structure or a finer hierarchy codebook could plausibly recover that lost precision without changing the descriptor.
- A testable extension would be replacing the 29-dimensional hand-designed shape features with learned embeddings and seeing whether tree-edit retrieval improves further, since the edit costs would then inherit whatever geometry the embedding learns.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hierarchy-of-Visual-Words (HoVW), a learning-based trademark image retrieval method. Binary trademark images are decomposed into component shapes, each represented by a 29-dimensional feature vector combining Zernike moments, circularity, bending energy, eccentricity, and convexity. A k-means codebook maps shapes to visual words, and inclusion/exclusion topology among shapes is encoded as a tree whose nodes carry visual-word labels. A second codebook of visual hierarchies is learned by mean-shift clustering over a dissimilarity matrix computed with the AP-TED tree edit distance, whose rename/insert/remove costs are defined in Equations (8) and (9). Retrieval first searches within the query's hierarchy label and then expands to nearby labels. Experiments on MPEG-7 CE-1 and CE-2 compare HoVW with ZM, Anuar et al., Liu et al., and a CNN baseline, reporting precision-recall curves and F1 scores.
Significance. If the reported results are reliable, HoVW would be a useful contribution: it combines learned shape codebooks with a hierarchical topology representation, is invariant to rotation, translation, and scale by construction, and releases code for reproducibility. The use of a tree-structured descriptor driven by shape inclusion/exclusion is a sensible way to capture both local geometry and global layout. However, the central claim that HoVW outperforms previous TIR methods is not currently supported because the two main hyperparameters are selected using the same test databases, and because the paper's own Table I contradicts the abstract on CE-1. The method itself is plausible and the issues are fixable, but the evidence as presented does not establish the stated state-of-the-art claim.
major comments (5)
- [IV-B, Fig. 7, IV-C, IV-D] The codebook size k and mean-shift bandwidth h are chosen by maximizing MAP computed on the full MPEG-7 CE-1 and CE-2 databases, and the same databases are then used for the 10-fold cross-validation reported in Sections IV-C and IV-D. No nested validation or held-out parameter selection is described. The comparison baselines (ZM, Anuar et al., Liu et al.) are parameter-free and taken from their original papers, so HoVW is the only method whose tuning uses the test labels. This selection bias can inflate the reported MAP, precision-recall curves, and F1 scores. The authors should re-run the evaluation with k and h selected on training folds only, or fix the hyperparameters before any evaluation, and report the variance across folds.
- [III, Equation (9)] The insertion/removal cost in Equation (9) is defined with alpha = min{1/log_2 L, 1/log_2 D}. When L = 1 or D = 1, which occurs for a root node, an only child, or any node at depth one, log_2(1) = 0 and the expression is undefined. Since AP-TED must assign costs to every node during tree edit distance computation, the method as specified is not well-defined for common tree structures. The authors need to state the boundary convention (for example, limiting alpha to 1 when L = 1 or D = 1) and confirm that the reported results use that convention.
- [Abstract, Table I, IV-D] The abstract claims that HoVW 'outperforms previous TIR methods on the MPEG-7 CE-1 and MPEG-7 CE-2 image databases,' but Table I shows that on MPEG-7 CE-1 the F1 score of HoVW is 0.85 while Liu et al. achieve 0.87. Since F1 at full recall is one of the two main evaluation metrics in the paper, this is a direct contradiction of the stated claim. The text in Section IV-C is more careful ('HoVW outperforms Liu et al.'s approach up to recall of 80%'), but the abstract and conclusion need to be revised to report metric-specific results rather than an unqualified outperformance claim.
- [III, Equations (8)-(9)] The rename, insert, and remove costs are all based on Euclidean distances between visual-word centroids in the 29-dimensional feature space. This space concatenates Zernike moments with circularity, bending energy, eccentricity, and convexity, but no feature normalization or scaling is described. Without standardization, the Euclidean distance is likely dominated by the feature components with the largest numerical range, which would make both the k-means codebook and the tree edit costs sensitive to arbitrary feature scale. The authors should justify the distance metric, describe any normalization, or add a sensitivity analysis.
- [IV-C, IV-D, II] The comparison against previous TIR methods is incomplete in two ways. First, the paper does not compare with the Curvature Tree approach of Alajlan et al., which is the most closely related topology-aware TIR method and is cited in Section II; without this comparison, the 'outperforms previous TIR methods' claim is not fully supported. Second, the reported differences on CE-2 are described as 'slightly better' on 11 out of 20 points of the precision-recall curve, but no statistical significance tests, confidence intervals, or per-fold standard deviations are reported. Given the small differences, the authors should provide uncertainty estimates and a significance analysis.
minor comments (6)
- [I] In the last paragraph of Section I, 'the 14 first images retrieved are butterfly images' should probably read 'the first 14 images retrieved are butterfly images.'
- [II] There is a typo in 'Perez’set al.' which should be 'Perez et al.'
- [IV-D] The table caption and the text use 'MEPG-7' instead of 'MPEG-7' in several places.
- [IV-B] The sentence describing data augmentation says 'flipping the testing database entries,' which presumably should be 'training database entries'; applying augmentation to the test set would be a methodological problem.
- [IV-B, Fig. 7] The text states that the k versus MAP chart for MPEG-7 CE-2 is 'equivalent' to the one shown for CE-1, but the actual chart is not presented; for reproducibility, the authors should include the CE-2 chart or provide the numeric values.
- [III, Fig. 4, Similar images retrieval] The retrieval strategy refers to the 'cluster closest α' and to 'close labels' in the visual-hierarchy codebook, but the distance between hierarchy labels is never defined. The authors should specify how label proximity is computed.
Circularity Check
The performance claim is partially self-referential: hyperparameters k and h are selected by maximizing MAP on the same MPEG-7 CE-1/CE-2 benchmarks later reported as the evaluation, so the reported superiority is partly a selection artifact.
-
fitted input called prediction
[Section IV-B, "HoVW Implementation and Parameterization" and "Evaluation"; Fig. 7]
"The size of the visual words codebook was set to k = 800 for MPEG-7 CE-1 and to k = 600 for MPEG-7 CE-2 after looking for the maximum among the mean average precision (MAP) metric values [30] computed as function of the number of clusters in k-means, for k∈{ 100, 200,··· , 1200}. ... The bandwidth h of the mean shift clustering procedure was chosen after assuming h∈{ 0.1, 0.3, 0.5, 0.7, 0.9} for MPEG-7 CE-1 and h∈{ 1.1, 1.3, 1.5, 1.7, 1.9} for MPEG-7 CE-2 and then analyzing the respective MAP values. ... Evaluation."
The two free parameters of HoVW, the codebook size k and the mean-shift bandwidth h, are explicitly chosen by maximizing MAP on the MPEG-7 CE-1 and CE-2 databases, and the final precision-recall and F1 results are then reported on the same databases. No nested cross-validation, held-out validation set, or separate parameter-selection protocol is described. Consequently, the reported MAP and the derived precision-recall/F1 curves are the very quantities used as the selection objective, so the claim that HoVW outperforms the untuned baselines (Liu et al., Anuar et al., ZM) is at least partly an artifact of fitting hyperparameters to the evaluation benchmark.
full rationale
The derivation-level pipeline of HoVW is not circular: shape decomposition, Zernike and contour features, tree encoding via Suzuki-Abe border following, AP-TED tree edit distance, and the two codebooks are all defined independently of the experimental outcome. There is no self-citation chain, no imported uniqueness theorem, and no renaming of a known result; the method's representation is a genuine construction. The circularity burden is concentrated in the evaluation protocol. Section IV-B states that k and h were selected as the maxima of MAP computed on the same MPEG-7 CE-1/CE-2 databases used for the final 10-fold cross-validation, without any nested or held-out parameter selection. This makes the reported performance partly self-referential: the evaluation metric is the tuning objective. The comparison baselines are taken from their original papers and are not tuned in this protocol, so HoVW enjoys a selection advantage that is not evidence of generalization. This is a fitted-input-called-prediction pattern rather than a definitional equivalence, so the score is moderate rather than maximal. In addition, Table I already contradicts the abstract's unqualified claim: Liu et al. reach F1=0.87 on CE-1 while HoVW reaches 0.85, which is an internal inconsistency but not itself a circularity.
Assumptions & free parameters
free parameters (3)
- Visual words codebook size k =
800 (CE-1), 600 (CE-2)
- Mean shift bandwidth h for hierarchy codebook =
0.7 (CE-1), 1.7 (CE-2)
- Zernike polynomial order cap =
8
assumptions (4)
- domain assumption Connected components of foreground and background, extracted with Suzuki-Abe border following, yield the semantically meaningful component shapes of a trademark.
- domain assumption Tree edit distance with the cost functions in Equations 8 and 9 ranks perceptually similar trademarks closer than dissimilar ones.
- ad hoc to paper Hyperparameters k and h that maximize MAP on the benchmark databases also generalize to unseen queries.
- domain assumption Excluding uncategorized MPEG-7 CE-2 images yields an unbiased retrieval evaluation.
Cite this review
Pith. "Pith review of Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark Image Retrieval." pith.science (2026). https://pith.science/paper/HU2QYNMN
@misc{pith2026190802786,
author = {Pith},
title = {Pith review of: Hierarchy-of-Visual-Words: a Learning-based Approach for Trademark Image Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/HU2QYNMN}},
note = {Machine review of arXiv:1908.02786}
}
read the original abstract
In this paper, we present the Hierarchy-of-Visual-Words (HoVW), a novel trademark image retrieval (TIR) method that decomposes images into simpler geometric shapes and defines a descriptor for binary trademark image representation by encoding the hierarchical arrangement of component shapes. The proposed hierarchical organization of visual data stores each component shape as a visual word. It is capable of representing the geometry of individual elements and the topology of the trademark image, making the descriptor robust against linear as well as to some level of nonlinear transformation. Experiments show that HoVW outperforms previous TIR methods on the MPEG-7 CE-1 and MPEG-7 CE-2 image databases.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
TRADEMARK: multimedia image database system with intelligent human interface,
T. Kato, K. Fujimura, and H. S. Nonmember, “TRADEMARK: multimedia image database system with intelligent human interface,” Systems and Comput. Japan , vol. 21, pp. 33–46, 1990
work page 1990
-
[2]
Content-based retrieval for trademark registration,
J. K. Wu, C. P. Lam, B. M. Mehtre, Y . J. Gao, and A. D. Narasimhalu, “Content-based retrieval for trademark registration,” Multimed. Tools Appl., vol. 3, pp. 245–267, 1996
work page 1996
-
[3]
Evaluation of a trademark image retrieval system,
J. P. Eakins, M. E. Graham, and J. M. Boardman, “Evaluation of a trademark image retrieval system,” in Proc. Annual BCS-IRSG Conf. Inf. Retr . Research, 1997
work page 1997
-
[4]
C. Hung Wei, Y . Li, W. Y . Chau, and C. T. Li, “Trademark image retrieval using synthetic features for describing global shape and interior structure,” Pattern Recognit., vol. 42, pp. 386–394, 2009
work page 2009
-
[5]
Trademark image retrieval using an integrated shape descriptor,
F. M. Anuar, R. Setchi, and Y . Lai, “Trademark image retrieval using an integrated shape descriptor,” Expert Syst. Appl. , vol. 40, pp. 105–121, 2013
work page 2013
-
[6]
H. Qi, K. Li, Y . Shen, and W. Qu, “An effective solution for trademark image retrieval by combining shape description and feature matching,” Pattern Recognit., vol. 43, pp. 2017–2027, 2010
work page 2017
-
[7]
Trademark image retrieval using hierarchical region feature description,
F. Liu, B. Wang, and F. Zeng, “Trademark image retrieval using hierarchical region feature description,” in Proc. IEEE Intl. Conf. Image Process., 2017, pp. 3620–3624
work page 2017
-
[8]
Content-based binary image retrieval using the adaptive hierarchical density histogram,
P. Sidiropoulos, S. Vrochidis, and I. Kompatsiaris, “Content-based binary image retrieval using the adaptive hierarchical density histogram,” Pattern Recognit., vol. 44, pp. 739–750, 2011
work page 2011
Show all 33 references
-
[9]
Near-duplicate image recognition and content-based image retrieval using adaptive hierarchical geometric centroids,
M. Yang, G. Qiu, J. Huang, and D. Elliman, “Near-duplicate image recognition and content-based image retrieval using adaptive hierarchical geometric centroids,” in Proc. Intl. Conf. Pattern Recognit. , 2006, pp. 958–961
2006
-
[10]
Multi-object image retrieval based on shape and topology,
N. Alajlan, M. S. Kamel, and G. Freeman, “Multi-object image retrieval based on shape and topology,” Signal Process. Image Commun. , vol. 21, pp. 904–918, 2006
2006
-
[11]
Geometry-based image retrieval in binary image databases,
N. Alajlan, M. S. Kamel, and G. H. Freeman, “Geometry-based image retrieval in binary image databases,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, pp. 1003–1013, 2008
2008
-
[12]
Recognition-by-components: a theory of human image understanding,
I. Biederman, “Recognition-by-components: a theory of human image understanding,” Psychol. Rev., vol. 94, pp. 115–147, 1987
1987
-
[13]
Tree edit distance: robust and memory- efficient,
M. Pawlik and N. Augsten, “Tree edit distance: robust and memory- efficient,” Inf. Syst. , vol. 56, pp. 157–173, 2016
2016
-
[14]
Least squares quantization in PCM,
S. Lloyd, “Least squares quantization in PCM,” IEEE Trans. Inf. Theor ., vol. 28, pp. 129–137, 1982
1982
-
[15]
Mean shift: a robust approach toward feature space analysis,
D. Comaniciu and P. Meer, “Mean shift: a robust approach toward feature space analysis,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 24, pp. 603–619, 2002
2002
-
[16]
Matching hierarchical structures using association graphs,
M. Pelillo, K. Siddiqi, and S. W. Zucker, “Matching hierarchical structures using association graphs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, pp. 1105–1120, 1999
1999
-
[17]
Image classification based on bag of visual graphs,
F. B. Silva, S. Goldenstein, S. Tabbone, and R. S. Torres, “Image classification based on bag of visual graphs,” in Proc. IEEE Intl. Conf. Image Process., 2013, pp. 4312–4316
2013
-
[18]
Graph-based bag-of-words for classification,
F. B. Silva, R. O. Werneck, S. Goldenstein, S. Tabbone, and R. S. Torres, “Graph-based bag-of-words for classification,” Pattern Recognit. , vol. 74, pp. 266–285, 2017
2017
-
[19]
An affine invariant interest point detector,
K. Mikolajczyk and C. Schmid, “An affine invariant interest point detector,” in Proc. European Conf. Comput. Vis. – Part I , 2002, pp. 128–142
2002
-
[20]
Object recognition from local scale-invariant features,
D. G. Lowe, “Object recognition from local scale-invariant features,” in Proc. Intl. Conf. Comput. Vis. , 1999, vol. 2, pp. 1150–1157
1999
-
[21]
Trademark image retrieval using a combination of deep convolutional neural networks,
C. A. Perez, P. A. Est ´evez, F. J. Galdames, D. A. Schulz, J. P. Perez, D. Bast ´ıas, and D. R. Vilar, “Trademark image retrieval using a combination of deep convolutional neural networks,” in Proc. Intl. Joint Conf. on Neural Networks , 2018, pp. 1–7
2018
-
[22]
Similar trademark image retrieval based on convolutional neural network and constraint theory,
T. Lan, X. Feng, L. Li, and Z. Xia, “Similar trademark image retrieval based on convolutional neural network and constraint theory,” in Proc. Intl. Conf. on Image Process. Theor ., Tools and Appl. , 2018, pp. 1–6
2018
-
[23]
A fast two-dimensional median filtering algorithm,
T. Huang, G. Yang, and G. Tang, “A fast two-dimensional median filtering algorithm,” IEEE Trans. Acoust., Speech, Signal Process. , vol. 27, pp. 13–18, 1979
1979
-
[24]
Bilateral filtering for gray and color images,
C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. Intl. Conf. Comput. Vis. , 1998, pp. 839–846
1998
-
[25]
A threshold selection method from gray-level histograms,
N. Otsu, “A threshold selection method from gray-level histograms,” IEEE Trans. Syst. Man Cybern. , vol. 9, pp. 62–66, 1979
1979
-
[26]
Topological structural analysis of digitized binary images by border following,
S. Suzuki and K. Abe, “Topological structural analysis of digitized binary images by border following,” Comput. Vis. Graph. Image Process., vol. 30, pp. 32–46, 1985
1985
-
[27]
Invariant image recognition by Zernike moments,
A. Khotanzad and Y .H. Hong, “Invariant image recognition by Zernike moments,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 12, pp. 489– 497, 1990
1990
-
[28]
A survey of shape feature extraction techniques,
Y . Mingqiang, K. Kidiyo, and R. Joseph, “A survey of shape feature extraction techniques,” in Pattern Recognition, P. Yin, Ed., chapter 3. IntechOpen, 2008
2008
-
[29]
An analysis technique for biological shape. i*,
I. T. Young, J. E. Walker, and J. E. Bowie, “An analysis technique for biological shape. i*,” Inf. Control, vol. 25, pp. 357–370, 1974
1974
-
[30]
Liu and M
L. Liu and M. T. ¨Ozsu, Encyclopedia of Database Systems , Springer US, 2009
2009
-
[31]
A region-based shape descriptor using Zernike moments,
W. Kim and Y Kim, “A region-based shape descriptor using Zernike moments,” Sig. Proc.: Image Comm. , vol. 16, pp. 95–102, 2000
2000
-
[32]
Very deep convolutional networks for large-scale image recognition,
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014
2014
-
[33]
ImageNet: a large-scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: a large-scale hierarchical image database,” in Proc. CVPR , 2009, pp. 248–255
2009
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.