A Fast Content-Based Image Retrieval Method Using Deep Visual Features

Hiroki Tanioka

REVIEW 4 major objections 6 minor 12 references

A Fast Content-Based Image Retrieval Method Using Deep Visual Features

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-indexing L2 norms turns cosine image retrieval into a single dot-product pass on an inverted index, eliminating re-ranking.

desk verdict Standard cosine-as-dot-product trick applied to Elasticsearch; math is right, but the scalability claim rests on untested sparsity assumptions and a toy evaluation. read the letter →

arxiv 1908.01505 v1 pith:6YVVC5DD submitted 2019-08-05 cs.CV cs.IRcs.LG

Hiroki Tanioka This is my paper · ORCID

classification cs.CVcs.IRcs.LG

keywords content-basedimageretrievaldeepvisualfeaturescosinesimilarityinvertedindexElasticsearchL2normalizationVGG-16meanaverageprecision

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that cosine similarity, not just inner product, can be the scoring function of an inverted-index search engine, by moving the expensive vector-length computation to indexing time. If the claim is right, content-based image retrieval with deep network features can run on disk-based indexes instead of main-memory-only systems, and the top results no longer need a second re-ranking pass. The paper reports the scheme on VGG-16 softmax features from ImageNet and finds that cosine scoring reaches perfect mean average precision on same-image queries once seven or more feature components are retained, while plain dot product stays near 0.05. Response times for cosine scoring stay in the same range as dot-product scoring, roughly 0.2 to 3 seconds for 100 top-k results, whereas the re-ranking baseline costs tens of seconds.

What carries the argument

The load-bearing object is the L2-normalized indexed feature vector, $y'_i = y_i / \|y\|$, stored in the index at ingest time along with the squared norm. The identity $\cos(x,y) \propto \sum_i x_i y'_i$ converts a cosine ranking into an inner-product ranking for a fixed query, which is exactly the kind of score an inverted-index engine can sum term by term. The implementation uses a script_score query on Elasticsearch that walks the stored per-synset scores of each candidate image, so no sequential norm computation and no top-k re-ranking is required.

What would settle it

Count the nonzero (or above a tiny threshold) components in the VGG-16 softmax vectors for a sample of ImageNet images: if the average count is close to 1,000 instead of a small fraction, the inverted index must touch nearly every dimension for every candidate, and the reported response times will not extend to larger collections. A scaling test that grows the collection by factors of 10 and watches whether retrieval time grows proportionally would settle the speed claim.

Watch

Extended reading notes

Core claim

The central discovery is a rearrangement of cosine similarity that makes it compatible with inverted-index scoring. Since the query vector is fixed during a search, the query norm is constant, so ranking by cosine similarity is the same as ranking by the dot product of the query with the L2-normalized indexed vector: $\cos(x,y) \propto x \cdot (y/\|y\|) = \sum_i x_i y'_i$. If the normalized components $y'_i$ and the squared norm $\|y\|^2$ are stored as document fields at indexing time, the search engine can compute this score with a single scripted dot-product pass. The paper further rewrites Manhattan and Euclidean distances as inner-product terms plus precomputed document statistics. Experiments on a dog-and-cat image set show that the cosine scheme matches or beats dot product and re-ranking cosine on mean average precision while staying in the dot product's response-time range.

Load-bearing premise

The method depends on each image's 1,000-dimensional feature vector being representable as a small number of indexable scores; if those vectors are dense, the disk-based inverted index will not speed up retrieval.

Editorial extensions

If this is right

Cosine similarity becomes a first-class score for inverted-index retrieval: any engine that can sum per-term scores can rank by cosine without re-ranking.
Retrieval accuracy on same-image queries is effectively perfect (MAP 1.0) once at least seven softmax components are retained, compared with about 0.05 for the inner-product baseline.
Manhattan and Euclidean distances are also expressible as inner products plus precomputed document statistics, so one indexing scheme supports several distance metrics.
Response times stay in the dot product's range (about 0.2 to 3 seconds for 100 top-k results), avoiding the roughly 35 to 53 second penalty of the re-ranking baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the query norm cancels from the ranking, a service that needs a literal cosine value can multiply the script score by $1/\|x\|$ at query time and still avoid re-ranking.
The same decomposition should apply to other embedding types, such as word2vec or document vectors, whenever the scoring metric is a cosine or a norm-expandable distance.
The method's speed rests on the indexed softmax vectors being sparse enough that only a few per-document scores are stored; a direct test would measure average retrieval time as the collection grows, which the paper does not report.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague —

What you should know: this is a short engineering note that applies a standard trick — cosine similarity as a dot product against L2-normalized vectors — to Elasticsearch. The math is correct, the exposition is clear, and avoiding the re-ranking step is a legitimate engineering goal. But the paper's central scalability claim rests on an untested sparsity assumption, and the evaluation is too thin to support the headline.

The derivation in Eqs. (5)–(6) is just the textbook identity cos(x,y) = x·(y/‖y‖) when x is fixed and y is pre-normalized. The authors implement this by storing the normalized components in the index and summing over query terms in a script_score. That is sound as far as it goes. The comparison against the dot+cos re-ranking baseline shows a large response-time gap, and the MAP numbers on the toy set suggest the method behaves sensibly at small scale.

Now the soft spots. The abstract says ImageNet, but the experiments use a small Kaggle Dogs vs Cats set, with no reported database size, index size, or postings density. More damaging, the method's speed story assumes the 1,000-dimensional VGG-16 softmax vector can be turned into a few index terms. If every document stores a score for all 1,000 classes, the script_score over match_all becomes a columnar scan of all documents — O(N·D) — and the inverted index buys nothing. The 'Feature Number' truncation is never specified as query-side or index-side; if the indexed vector is truncated and not renormalized, the score computed is no longer the exact cosine of Eq. (6). The paper reports no evidence about sparsification, so the claimed advantage over a linear scan is uninterpretable.

These problems do not invalidate the core identity, but they do undermine the paper's main claim. The MAP of 1.0 on a tiny set is meaningless for generalizing. As it stands, this is a plausible prototype write-up, not a complete performance study.

Who gets value: someone implementing or evaluating Elasticsearch for CBIR might use this as a starting point. I would not cite it in my own work, and I would not bring it to reading group. But the idea deserves a real referee if the authors respond to the sparsity concern: report index statistics, test with a public benchmark, and define the truncation/renormalization. I'd send it back for major revision, not desk-reject.

Referee Report

4 major / 6 minor

Summary. The paper proposes to compute cosine similarity in an Elasticsearch inverted-index engine by precomputing each document's L2 norm at indexing time, storing per-synset normalized scores, and then scoring query documents by the dot product of the query vector with the L2-normalized stored vector (Eqs. 5-6). It also sketches analogous treatments for Manhattan and Euclidean distances. Experiments compare dot product, Manhattan, Euclidean, direct cosine, and dot-plus-rerank cosine on a Kaggle Dogs vs Cats collection, reporting response time and MAP for varying feature numbers and resolution rates. The claimed contribution is fast and scalable image retrieval on an inverted-index engine without main-memory indexing or re-ranking.

Significance. If the scalability claim were established, the contribution would be practically useful: it would let cosine-based CBIR run on commodity inverted-index engines, avoiding memory-based systems and re-ranking. The mathematical identity in Eqs. (5)-(6) is sound, and the paper is transparent about its baseline comparisons. However, the current evaluation does not establish the central speed/scalability advantage: the sparsity assumption underlying inverted-index efficiency is nowhere defined or measured, and the experiments omit basic scale information such as N, index size, and postings density. The accuracy results are also based on a very small query set with no variance. With the missing measurements supplied, the core idea could be a useful engineering contribution, but as written it is not yet supported.

major comments (4)

[§III, Tables II–V, Fig. 2] The efficiency claim is not supported because the manuscript never defines or measures the sparsity of the indexed 1,000-dimensional VGG-16 softmax vectors. The query in Fig. 2 builds a script_score over the query's synsets with a match_all query; if every document stores a value for every synset, then every posting list has length N and the engine scores all N documents, so the cost is O(N·D) and no better than a columnar scan. The paper must state whether "Feature Number" in Tables II–V means query-side truncation, index-side truncation, or both; give the truncation or quantization rule; and report the number of images N, the index size, per-document term counts, and postings density. Without these, the response times in Table II cannot be interpreted as evidence of inverted-index sublinearity.
[Abstract and §III] The abstract states "We evaluate our approach with ImageNet Dataset and VGG-16 pre-trained model," but the experiments in Section III are run on the Kaggle Dogs vs Cats data [3]. Either the ImageNet evaluation should be performed and reported, or the abstract should be corrected. Additionally, the database size N is never reported, which further limits reproducibility.
[§II-D, §II-E, Eqs. (7)–(10)] The Manhattan and Euclidean proposals are only sketched. Eq. (10) is valid for ranking a fixed query because ‖x‖ is constant, but the manuscript never explains how L1 distance or the quantity L2_y^2 − 2 x·y is evaluated in Elasticsearch, which fields in Table I store the required statistics, or how distances are "converted to complement" for scoring. As Sections II-D and II-E are presented as part of the proposed method, this implementation gap should be filled.
[Tables III–V] All accuracy claims are based on 10 base query images with no variance information; MAP values of 1.000 (Tables III–IV) are reported without error bars, repetitions, or per-query breakdown. At minimum, standard deviations or per-query distributions should be added so the reader can judge whether the differences between scoring functions are meaningful.

minor comments (6)

[Table I] The row entries "s {synset wnid}", "c normalized score by L2 norm", and "ss s squared" are not explained; the relation between s, ss, and c (presumably c = s/√ss) should be stated explicitly, and it is unclear whether "ss" is actually used by any query.
[Eq. (3)] Eq. (3) contains malformed radical notation ("/radicaltp /radicalvertex ...") from the rendering pipeline; it should be a standard square-root symbol.
[Tables IV and Fig. 4] The term "resolution rate" is used without definition; state what resolution values 1.0, 0.8, etc. mean and how images were resized.
[Section III and Fig. 5] Figure 5 and Section III refer to "quartered partial images" but do not specify how the quarter crops were extracted or how ground truth for partial-image queries was determined.
[Tables II–V] The "dot+cos" baseline is not described; specify the top-k and re-ranking procedure used (e.g., the method of [7]), since its 35–53 s response times dominate Table II.
[Section III] The paper says "the prototype system implementation is naive using script score of Elasticsearch" but does not state the Elasticsearch version or mapping (e.g., whether doc values are enabled); this information is needed to assess the timing experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: cosine-L2 identity is derived from definitions; self-citation [9] is context only.

full rationale

The paper's central derivation is Eqs. (3)-(6), which rewrite cosine similarity as a dot product against an L2-normalized indexed vector: cos(x,y) is proportional to sum_i x_i (y_i / ||y||). This is a direct algebraic identity from the definitions of cosine similarity, dot product, and L2 norm; no parameter is fitted, and no predicted quantity is used to determine any constant. The experimental MAP and response-time values are measurements, not outputs forced by construction. The only self-citation, reference [9] (the author's own GitHub repository), is used to describe a previously proposed inner-product-based image search system and is not load-bearing for the proposed normalization identity or for the indexing scheme. The abstract's statement that the method is evaluated with ImageNet, while the experiments use Kaggle Dogs vs Cats, is a correctness/reproducibility discrepancy rather than a circularity. The untested sparsity assumption underlying the scalability claim is a legitimate technical concern but does not constitute circular reasoning. Therefore, no significant circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; VGG-16 weights are pretrained. The core identity is not circular. The main unverified addition is the sparse-indexability assumption for dense softmax vectors.

assumptions (3)

standard math Cosine similarity ranking is preserved when the query norm is omitted: rank by x·(y/‖y‖) instead of (x·y)/(‖x‖‖y‖).
Invoked in Eqs. (5)-(6); this is elementary linear algebra.
domain assumption VGG-16 softmax activations (1,000-dim) are effective visual features for retrieval.
The entire evaluation depends on this; the paper does not test other features or justify the choice beyond common practice (Fig. 1).
ad hoc to paper The dense 1,000-dim softmax vector can be stored and queried efficiently as sparse terms in an Elasticsearch inverted index.
Table I indexes per-synset scores without specifying thresholding or measuring density; the speed claim relies on this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Fast Content-Based Image Retrieval Method Using Deep Visual Features." pith.science (2026). https://pith.science/paper/6YVVC5DD

@misc{pith2026190801505,
  author       = {Pith},
  title        = {Pith review of: A Fast Content-Based Image Retrieval Method Using Deep Visual Features},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6YVVC5DD}},
  note         = {Machine review of arXiv:1908.01505}
}

read the original abstract

Fast and scalable Content-Based Image Retrieval using visual features is required for document analysis, Medical image analysis, etc. in the present age. Convolutional Neural Network (CNN) activations as features achieved their outstanding performance in this area. Deep Convolutional representations using the softmax function in the output layer are also ones among visual features. However, almost all the image retrieval systems hold their index of visual features on main memory in order to high responsiveness, limiting their applicability for big data applications. In this paper, we propose a fast calculation method of cosine similarity with L2 norm indexed in advance on Elasticsearch. We evaluate our approach with ImageNet Dataset and VGG-16 pre-trained model. The evaluation results show the effectiveness and efficiency of our proposed method.

Figures

Figures reproduced from arXiv: 1908.01505 by the authors.

**Figure 2.** Query using function score for cosine similarity on Elasticsearch. Thus, re-calculating score within top-k (k = 10, 100, 1, 000) search result and re-ranking the search result. C. Proposed system (Cosine) The calculation formula of cosine similarity is divided into two phases. The L2 norm of the vector is calculated and registered in the index at the indexing phase, shown in Talbe I. Then, cosine similarity is calcu… view at source ↗

**Figure 4.** Comparison Mean Average Precision (MAP) of resoluti [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

**Figure 5.** Quartered partial image queries for partial image se [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗

Figures from the paper (1 more)

**Figure 6.** Figure 6: Comparison Mean Average Precision (MAP) of feature n [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

[3]

(2014) Dogs vs

Kaggle . (2014) Dogs vs. cats. [Online]. Available: https://www.kaggle.com/c/dogs-vs-cats

work page 2014
[1]

Amato, P

G. Amato, P. Bolettieri, F. Carrara, F. Falchi, and C. Gennaro, ``Large-scale image retrieval with elasticsearch,'' in The 41st International ACM SIGIR Conference on Research &\#38; Development in Information Retrieval, ser. SIGIR '18. 1em plus 0.5em minus 0.4em New York, NY, USA: ACM, 2018, pp. 925--928. [Online]. Available: http://doi.acm.org/10.1145/32...

arXiv 2018
[2]

(2014) Large scale visual recognition challenge (ilsvrc)

ImageNet . (2014) Large scale visual recognition challenge (ilsvrc). [Online]. Available: http://www.image-net.org/challenges/LSVRC/

work page 2014
[4]

H. Liu , R. Wang , S. Shan , and X. Chen , ``Deep supervised hashing for fast image retrieval,'' in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 2064--2072

work page 2016
[5]

H. Liu, B. Li, X. Lv, and Y. Huang, ``Image retrieval using fused deep convolutional features,'' Procedia Computer Science, vol. 107, pp. 749 -- 754, 2017, advances in Information and Communication Technology: Proceedings of 7th International Congress of Information and Communication Technology (ICICT2017). [Online]. Available: http://www.sciencedirect.co...

work page 2017
[6]

Lux and S

M. Lux and S. A. Chatzichristofis, ``Lire: Lucene image retrieval: An extensible java cbir library,'' in Proceedings of the 16th ACM International Conference on Multimedia, ser. MM '08. 1em plus 0.5em minus 0.4em New York, NY, USA: ACM, 2008, pp. 1085--1088. [Online]. Available: http://doi.acm.org/10.1145/1459359.1459577

work page arXiv 2008
[7]

C. Mu, J. Zhao, G. Yang, J. Zhang, and Z. Yan, ``Towards practical visual search engine within elasticsearch,'' in The SIGIR 2018 Workshop On eCommerce co-located with the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, USA, July 12, 2018. , ser. CEUR Workshop Proceedings, J. ...

work page 2018
[8]

Sadeghi-Tehran, P

P. Sadeghi-Tehran, P. Angelov, N. Virlet, and M. J. Hawkesford, ``Scalable database indexing and fast image retrieval based on deep learning and hierarchically nested structure applied to remote sensing and plant biology,'' Journal of Imaging, vol. 5, no. 3, 2019. [Online]. Available: https://www.mdpi.com/2313-433X/5/3/33

work page 2019

Show all 12 references

[9]

Tanioka, ``Super easy way of building image search with keras,'' Aug

H. Tanioka, ``Super easy way of building image search with keras,'' Aug. 2017. [Online]. Available: https://github.com/taniokah/liarr2017

2017
[10]

`` word2vec : Tool for computing continuous distributed representations of words ,'' https://code.google.com/p/word2vec, [Online; accessed 11-September-2018]

2018
[11]

4" FUNCTION default.is.dash.repeated.names #1 FUNCTION default.name.format.string

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEE...

2007
[12]

write newline

" write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write ...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [3]

(2014) Dogs vs

Kaggle . (2014) Dogs vs. cats. [Online]. Available: https://www.kaggle.com/c/dogs-vs-cats

work page 2014

[2] [1]

Amato, P

G. Amato, P. Bolettieri, F. Carrara, F. Falchi, and C. Gennaro, ``Large-scale image retrieval with elasticsearch,'' in The 41st International ACM SIGIR Conference on Research &\#38; Development in Information Retrieval, ser. SIGIR '18. 1em plus 0.5em minus 0.4em New York, NY, USA: ACM, 2018, pp. 925--928. [Online]. Available: http://doi.acm.org/10.1145/32...

arXiv 2018

[3] [2]

(2014) Large scale visual recognition challenge (ilsvrc)

ImageNet . (2014) Large scale visual recognition challenge (ilsvrc). [Online]. Available: http://www.image-net.org/challenges/LSVRC/

work page 2014

[4] [4]

H. Liu , R. Wang , S. Shan , and X. Chen , ``Deep supervised hashing for fast image retrieval,'' in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 2064--2072

work page 2016

[5] [5]

H. Liu, B. Li, X. Lv, and Y. Huang, ``Image retrieval using fused deep convolutional features,'' Procedia Computer Science, vol. 107, pp. 749 -- 754, 2017, advances in Information and Communication Technology: Proceedings of 7th International Congress of Information and Communication Technology (ICICT2017). [Online]. Available: http://www.sciencedirect.co...

work page 2017

[6] [6]

Lux and S

M. Lux and S. A. Chatzichristofis, ``Lire: Lucene image retrieval: An extensible java cbir library,'' in Proceedings of the 16th ACM International Conference on Multimedia, ser. MM '08. 1em plus 0.5em minus 0.4em New York, NY, USA: ACM, 2008, pp. 1085--1088. [Online]. Available: http://doi.acm.org/10.1145/1459359.1459577

work page arXiv 2008

[7] [7]

C. Mu, J. Zhao, G. Yang, J. Zhang, and Z. Yan, ``Towards practical visual search engine within elasticsearch,'' in The SIGIR 2018 Workshop On eCommerce co-located with the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, USA, July 12, 2018. , ser. CEUR Workshop Proceedings, J. ...

work page 2018

[8] [8]

Sadeghi-Tehran, P

P. Sadeghi-Tehran, P. Angelov, N. Virlet, and M. J. Hawkesford, ``Scalable database indexing and fast image retrieval based on deep learning and hierarchically nested structure applied to remote sensing and plant biology,'' Journal of Imaging, vol. 5, no. 3, 2019. [Online]. Available: https://www.mdpi.com/2313-433X/5/3/33

work page 2019