{"id":"d27918d3-446b-4e93-8f67-3a50346ee116","arxiv_id":"1908.02786","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hierarchy-of-visual-words tree descriptor, built from decomposed shapes and learned visual word codebooks, improves trademark image retrieval on MPEG-7 CE-1 and CE-2 benchmarks.","lead":"This paper introduces a trademark image retrieval method that breaks each binary image into component shapes, arranges them in a tree, and looks up similar trademarks by matching these trees. It reports higher retrieval precision than several older methods on standard MPEG-7 benchmarks, with caveats about how the method was tuned.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hyperparameters k and h are tuned on the same benchmark used for evaluation, so the claimed outperformance may be test-set overfitting; Table I already contradicts the abstract on CE-1.","rationale":"The reader's weakest_assumption focuses on the tree-edit-distance cost model, but the more decisive vulnerability is the evaluation protocol: Section IV-B tunes both free parameters by maximizing MAP on the same datasets used for the reported 10-fold CV. This directly undermines the central 'outperforms previous TIR methods' claim, because the baselines do not receive equivalent tuning. The reader's rationale did note the hyperparameter issue, but did not list it as the weakest assumption, hence partial agreement. I did not choose the tree-edit-distance perceptual assumption as the central concern because Euclidean distance on Zernike moments is a standard and defensible shape similarity, and even a perfect cost model would not rescue a benchmark that leaks test labels. The missing Curvature Tree and graph-BoW baselines are also important, but they are less decisive because they require additional experiments rather than exposing an internal inconsistency. The paper deserves credit for releasing code and using AP-TED; these strengths do not mitigate the test-set tuning problem. Since the issue is concrete, fixable by nested validation, and does not necessarily invalidate the descriptor, the appropriate verdict remains CONDITIONAL, matching the reader's verdict and requiring the authors to add a proper validation procedure and soften the abstract to match their measured results.","tokens_in":11079,"tokens_out":6110,"duration_ms":68861,"concrete_test":"Using the released HoVW code, re-run CE-1 and CE-2 with nested validation: for each of the 10 outer test folds, select k and h on the 9 training folds only (e.g., by inner 10-fold CV or a validation split), then evaluate on the held-out fold; report mean and standard deviation of MAP, precision-recall, and F1. If the nested MAP on CE-1 drops below Liu et al.'s reported curve/F1, the headline claim is an artifact of test-set tuning. As a secondary check, recompute the F1 comparison on CE-1 at recall=100% to make the abstract consistent with Table I.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim requires the reported 10-fold cross-validation results to reflect generalization. Section IV-B states that the visual-words codebook size k (800 for CE-1, 600 for CE-2) was selected as the maximum MAP over k in {100,...,1200}, and that the mean-shift bandwidth h was selected as the maximum MAP over h in {0.1,...,0.9} (CE-1) and {1.1,...,1.9} (CE-2), with Fig. 7 plotting MAP on the same MPEG-7 datasets. The evaluation section then reports 10-fold CV on those same datasets, without any nested or held-out parameter selection. Thus the two free parameters are effectively fit to the test labels. Because the comparison baselines (Liu et al., Anuar et al., ZM) are parameter-free and taken from their papers, HoVW enjoys an advantage that may be pure selection bias. This is not hypothetical: Table I shows Liu et al. achieving F1=0.87 on CE-1 versus HoVW's 0.85, so the abstract's unqualified 'outperforms previous TIR methods on MPEG-7 CE-1' is already false for the F1 metric at full recall. The central claim therefore rests on an evaluation protocol that both leaks test information and is internally inconsistent with the paper's own table.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hierarchy-of-Visual-Words (HoVW), a learning-based trademark image retrieval method. Binary trademark images are decomposed into component shapes, each represented by a 29-dimensional feature vector combining Zernike moments, circularity, bending energy, eccentricity, and convexity. A k-means codebook maps shapes to visual words, and inclusion/exclusion topology among shapes is encoded as a tree whose nodes carry visual-word labels. A second codebook of visual hierarchies is learned by mean-shift clustering over a dissimilarity matrix computed with the AP-TED tree edit distance, whose rename/insert/remove costs are defined in Equations (8) and (9). Retrieval first searches within the query's hierarchy label and then expands to nearby labels. Experiments on MPEG-7 CE-1 and CE-2 compare HoVW with ZM, Anuar et al., Liu et al., and a CNN baseline, reporting precision-recall curves and F1 scores.","tokens_in":11357,"tokens_out":5096,"duration_ms":56799,"significance":"If the reported results are reliable, HoVW would be a useful contribution: it combines learned shape codebooks with a hierarchical topology representation, is invariant to rotation, translation, and scale by construction, and releases code for reproducibility. The use of a tree-structured descriptor driven by shape inclusion/exclusion is a sensible way to capture both local geometry and global layout. However, the central claim that HoVW outperforms previous TIR methods is not currently supported because the two main hyperparameters are selected using the same test databases, and because the paper's own Table I contradicts the abstract on CE-1. The method itself is plausible and the issues are fixable, but the evidence as presented does not establish the stated state-of-the-art claim.","major_comments":[{"comment":"The codebook size k and mean-shift bandwidth h are chosen by maximizing MAP computed on the full MPEG-7 CE-1 and CE-2 databases, and the same databases are then used for the 10-fold cross-validation reported in Sections IV-C and IV-D. No nested validation or held-out parameter selection is described. The comparison baselines (ZM, Anuar et al., Liu et al.) are parameter-free and taken from their original papers, so HoVW is the only method whose tuning uses the test labels. This selection bias can inflate the reported MAP, precision-recall curves, and F1 scores. The authors should re-run the evaluation with k and h selected on training folds only, or fix the hyperparameters before any evaluation, and report the variance across folds.","section":"IV-B, Fig. 7, IV-C, IV-D"},{"comment":"The insertion/removal cost in Equation (9) is defined with alpha = min{1/log_2 L, 1/log_2 D}. When L = 1 or D = 1, which occurs for a root node, an only child, or any node at depth one, log_2(1) = 0 and the expression is undefined. Since AP-TED must assign costs to every node during tree edit distance computation, the method as specified is not well-defined for common tree structures. The authors need to state the boundary convention (for example, limiting alpha to 1 when L = 1 or D = 1) and confirm that the reported results use that convention.","section":"III, Equation (9)"},{"comment":"The abstract claims that HoVW 'outperforms previous TIR methods on the MPEG-7 CE-1 and MPEG-7 CE-2 image databases,' but Table I shows that on MPEG-7 CE-1 the F1 score of HoVW is 0.85 while Liu et al. achieve 0.87. Since F1 at full recall is one of the two main evaluation metrics in the paper, this is a direct contradiction of the stated claim. The text in Section IV-C is more careful ('HoVW outperforms Liu et al.'s approach up to recall of 80%'), but the abstract and conclusion need to be revised to report metric-specific results rather than an unqualified outperformance claim.","section":"Abstract, Table I, IV-D"},{"comment":"The rename, insert, and remove costs are all based on Euclidean distances between visual-word centroids in the 29-dimensional feature space. This space concatenates Zernike moments with circularity, bending energy, eccentricity, and convexity, but no feature normalization or scaling is described. Without standardization, the Euclidean distance is likely dominated by the feature components with the largest numerical range, which would make both the k-means codebook and the tree edit costs sensitive to arbitrary feature scale. The authors should justify the distance metric, describe any normalization, or add a sensitivity analysis.","section":"III, Equations (8)-(9)"},{"comment":"The comparison against previous TIR methods is incomplete in two ways. First, the paper does not compare with the Curvature Tree approach of Alajlan et al., which is the most closely related topology-aware TIR method and is cited in Section II; without this comparison, the 'outperforms previous TIR methods' claim is not fully supported. Second, the reported differences on CE-2 are described as 'slightly better' on 11 out of 20 points of the precision-recall curve, but no statistical significance tests, confidence intervals, or per-fold standard deviations are reported. Given the small differences, the authors should provide uncertainty estimates and a significance analysis.","section":"IV-C, IV-D, II"}],"minor_comments":[{"comment":"In the last paragraph of Section I, 'the 14 first images retrieved are butterfly images' should probably read 'the first 14 images retrieved are butterfly images.'","section":"I"},{"comment":"There is a typo in 'Perez’set al.' which should be 'Perez et al.'","section":"II"},{"comment":"The table caption and the text use 'MEPG-7' instead of 'MPEG-7' in several places.","section":"IV-D"},{"comment":"The sentence describing data augmentation says 'flipping the testing database entries,' which presumably should be 'training database entries'; applying augmentation to the test set would be a methodological problem.","section":"IV-B"},{"comment":"The text states that the k versus MAP chart for MPEG-7 CE-2 is 'equivalent' to the one shown for CE-1, but the actual chart is not presented; for reproducibility, the authors should include the CE-2 chart or provide the numeric values.","section":"IV-B, Fig. 7"},{"comment":"The retrieval strategy refers to the 'cluster closest α' and to 'close labels' in the visual-hierarchy codebook, but the distance between hierarchy labels is never defined. The authors should specify how label proximity is computed.","section":"III, Fig. 4, Similar images retrieval"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a real representation idea, but the headline experimental claim doesn't survive contact with the paper's own Table I.\n\nWhat's new: HoVW decomposes binary trademarks into component shapes (objects and holes), assigns each a codebook label learned by k-means on Zernike+contour features, and arranges the labels into an inclusion/exclusion tree. Tree similarity is computed with AP-TED and learned edit costs. That is a clean and original combination. The paper distinguishes it properly from region-partition methods, Curvature Tree, and graph-based BoVW, and ships implementation code. The method is described in enough detail to reimplement.\n\nWhat's soft: the authors choose the visual-words codebook size k and mean-shift bandwidth h by maximizing MAP on the very same MPEG-7 CE-1 and CE-2 datasets they then evaluate with 10-fold CV. That is test-set leakage, plain and simple. The k vs. MAP figure shows the curve is fairly flat from 800 to 1200, so the practical effect may be small, but the protocol is still wrong. They also don't report variance across folds, and they compare against numbers taken from other papers rather than running the baselines. More seriously, the abstract says HoVW 'outperforms previous TIR methods' on both databases, but Table I gives Liu et al. F1=0.87 on CE-1 against HoVW's 0.85. The precision-recall curves show HoVW winning at low recall and tying at high recall, so the claim should be qualified.\n\nNone of this sinks the method. The representation is sensible, the retrieval results are at least comparable to the state of the art, and the code is public. The fix is straightforward: add nested or separate validation for the hyperparameters, report fold-wise variance, run the closest structural baselines (Curvature Tree, graph BoVW) on the same folds, and align the abstract with what the numbers actually support.\n\nWho should read it: people working on binary image retrieval or shape descriptors. It is not a breakthrough, but it is a useful data point and a worked example of learning hierarchical codebooks.\n\nRecommendation: this deserves peer review, but only after the authors address the evaluation protocol. Send it to a competent referee with the expectation of major revision.\n\nBest,\n[Your name]","headline":"A genuinely new hierarchical visual-words descriptor for trademark retrieval, but the main empirical claim is compromised by test-set hyperparameter tuning and contradicted by its own F1 table.","tokens_in":11886,"tokens_out":2545,"would_cite":false,"duration_ms":25858,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trademark images are better retrieved as trees of learned visual words than as global image descriptors, and the paper shows the tree representation beats previous shape-based methods on the MPEG-7 CE-1 and CE-2 benchmarks.","keywords":["trademark image retrieval","hierarchy-of-visual-words","visual words codebook","tree edit distance","shape descriptors","MPEG-7 CE-1","MPEG-7 CE-2","binary image retrieval"],"falsifier":"Compare, for a query trademark, retrieval ranks between a same-class variant whose component shape is replaced by a perceptually similar shape and a variant whose replacement shape is Euclidean-close to the original's cluster centroid but visually different; if the second variant consistently outranks the first, the edit-cost model is not capturing perceptual similarity.","tokens_in":10876,"feed_emoji":"🌳","tokens_out":9490,"duration_ms":96904,"temperature":0.7,"pith_summary":"The paper is trying to establish that a binary trademark image is best represented for retrieval not by a global feature vector but by a tree of learned visual words: decompose the image into objects and holes, encode each component shape by a learned codebook word, and arrange those words according to inclusion and exclusion topology. The retrieval ranking then compares trees with a tree edit distance whose operation costs are derived from distances between learned shape clusters. The paper's experiments on MPEG-7 CE-1 and CE-2 report that this representation outperforms prior shape-based trademark retrieval methods and, on CE-2, also beats a CNN baseline. If that holds, trademark-infringement screening can use a learned structural representation that requires no manually defined similarity classes and no deep network training.","feed_headline":"Shape trees beat prior trademark retrieval on MPEG-7 benchmarks","feed_subtitle":"A learned codebook encodes shapes and their nesting, ranking logos by tree edit distance instead of global features.","key_machinery":"The load-bearing object is the visual hierarchy: a rooted tree per image in which each node stores a visual word and edges encode inclusion and exclusion among component shapes. A shape A is included in a shape B if A is a hole inside object B or an object inside hole B, and shapes are siblings when both are included in a common shape. The hierarchy is built by recursively extracting foreground objects and holes with border following. Nodes are labeled by a k-means visual-words codebook learned from 29-dimensional shape features (Zernike moments, circularity, average bending energy, eccentricity, convexity), so the tree mixes local geometry at its nodes with global topology in its edges. Tree comparison uses AP-TED with rename cost $\\delta_r(n_a,n_b)=\\operatorname{dist}_E(\\lambda_a,\\lambda_b)$ and insert or remove cost $\\delta_x(n)=\\alpha \\frac{2}{m(m-1)}\\sum_{i