{"id":"038b501d-1bc0-4ef1-b0d1-78ae69f9ed01","arxiv_id":"1908.01505","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Cosine similarity for image retrieval can be computed inside Elasticsearch by storing L2-normalized deep features at index time, removing the need for re-ranking.","lead":"This paper describes a way to run image similarity search on the Elasticsearch engine by storing each image's normalized visual features, so that cosine similarity becomes a simple dot product. It is a modest engineering improvement that avoids an extra re-ranking step, tested on a small dog and cat image dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scalability claim rests on an untested term-sparsity assumption: VGG-16 softmax vectors are dense, so without a defined truncation/quantization rule the inverted index degenerates to a linear scan; the paper reports no index size or postings density.","rationale":"The reader's weakest_assumption correctly identifies the central soft spot: the method assumes the deep feature vectors can be made indexable as sparse terms, yet no sparsification rule or index-size measurement is provided. My reading of the implementation in Fig. 2 strengthens this concern: the scoring script dereferences doc['s.<id>.c'].value for each query term, so if the 1,000-dimensional softmax vector is dense, Elasticsearch is effectively performing a per-document field scan rather than exploiting inverted-index pruning. The math of the proposed cosine transformation itself is correct, and the idea of precomputing the L2 norm is reasonable, so this is not a soundness objection to Eqs. (5)-(6). It is an empirical and architectural objection to the paper's central 'fast and scalable' claim. The paper also says it evaluates on ImageNet while the experiments use Kaggle Dogs vs Cats, but that inconsistency is secondary to the missing sparsity evidence. Because the reader already conditioned the verdict on this issue, no change in verdict is needed; the condition should be made explicit: either provide a density/sparsity analysis and index-size report, or demonstrate on a realistic large collection that response time is sublinear in the number of indexed images.","tokens_in":6277,"tokens_out":6681,"duration_ms":76210,"concrete_test":"Run a density audit on the actual VGG-16 features: sample 1,000 images, compute the softmax vectors, and report the average number of components above thresholds (0, 1e-4, 1e-3, 1e-2 of the max component) together with the resulting Elasticsearch postings-list lengths. If the mean density is close to 1,000 components at any threshold used, the inverted index degenerates to a linear scan, and the reported response times cannot generalize to large collections; this would falsify the scalability claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the implicit claim that a 1,000-dimensional VGG-16 softmax vector can be represented as a small set of Elasticsearch terms such that the inverted-index engine provides a real speed advantage. The mathematics of Eqs. (5)-(6) is sound: for a fixed query vector x, ranking by x·(y/||y||) is equivalent to ranking by cosine similarity. But the implementation in Fig. 2 computes a sum over the query's 'synsets' of doc['s.<id>.c'].value. If every indexed image stores a value for most or all of the 1,000 classes, then every query term touches a posting list of length N, match_all returns all N documents, and the cost is O(N·D): a columnar scan, not sublinear inverted-index search. The paper never defines a sparsification threshold, a quantization rule, or whether the 'Feature Number' in Tables II-V means query-side truncation, index-side truncation, or both. If the stored vector is truncated and renormalized, the score computed is no longer the exact cosine of Eq. (6). No index size, postings density, or per-document term count is reported, so the claimed speed advantage over the dot+cos re-ranking baseline is uninterpretable outside the 25k-image test set. The abstract's ImageNet evaluation is also absent (the experiments use Kaggle Dogs vs Cats), but the more damaging gap is the missing sparsity evidence, which directly determines whether the method scales.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to compute cosine similarity in an Elasticsearch inverted-index engine by precomputing each document's L2 norm at indexing time, storing per-synset normalized scores, and then scoring query documents by the dot product of the query vector with the L2-normalized stored vector (Eqs. 5-6). It also sketches analogous treatments for Manhattan and Euclidean distances. Experiments compare dot product, Manhattan, Euclidean, direct cosine, and dot-plus-rerank cosine on a Kaggle Dogs vs Cats collection, reporting response time and MAP for varying feature numbers and resolution rates. The claimed contribution is fast and scalable image retrieval on an inverted-index engine without main-memory indexing or re-ranking.","tokens_in":6518,"tokens_out":6356,"duration_ms":62350,"significance":"If the scalability claim were established, the contribution would be practically useful: it would let cosine-based CBIR run on commodity inverted-index engines, avoiding memory-based systems and re-ranking. The mathematical identity in Eqs. (5)-(6) is sound, and the paper is transparent about its baseline comparisons. However, the current evaluation does not establish the central speed/scalability advantage: the sparsity assumption underlying inverted-index efficiency is nowhere defined or measured, and the experiments omit basic scale information such as N, index size, and postings density. The accuracy results are also based on a very small query set with no variance. With the missing measurements supplied, the core idea could be a useful engineering contribution, but as written it is not yet supported.","major_comments":[{"comment":"The efficiency claim is not supported because the manuscript never defines or measures the sparsity of the indexed 1,000-dimensional VGG-16 softmax vectors. The query in Fig. 2 builds a script_score over the query's synsets with a match_all query; if every document stores a value for every synset, then every posting list has length N and the engine scores all N documents, so the cost is O(N·D) and no better than a columnar scan. The paper must state whether \"Feature Number\" in Tables II–V means query-side truncation, index-side truncation, or both; give the truncation or quantization rule; and report the number of images N, the index size, per-document term counts, and postings density. Without these, the response times in Table II cannot be interpreted as evidence of inverted-index sublinearity.","section":"§III, Tables II–V, Fig. 2"},{"comment":"The abstract states \"We evaluate our approach with ImageNet Dataset and VGG-16 pre-trained model,\" but the experiments in Section III are run on the Kaggle Dogs vs Cats data [3]. Either the ImageNet evaluation should be performed and reported, or the abstract should be corrected. Additionally, the database size N is never reported, which further limits reproducibility.","section":"Abstract and §III"},{"comment":"The Manhattan and Euclidean proposals are only sketched. Eq. (10) is valid for ranking a fixed query because ‖x‖ is constant, but the manuscript never explains how L1 distance or the quantity L2_y^2 − 2 x·y is evaluated in Elasticsearch, which fields in Table I store the required statistics, or how distances are \"converted to complement\" for scoring. As Sections II-D and II-E are presented as part of the proposed method, this implementation gap should be filled.","section":"§II-D, §II-E, Eqs. (7)–(10)"},{"comment":"All accuracy claims are based on 10 base query images with no variance information; MAP values of 1.000 (Tables III–IV) are reported without error bars, repetitions, or per-query breakdown. At minimum, standard deviations or per-query distributions should be added so the reader can judge whether the differences between scoring functions are meaningful.","section":"Tables III–V"}],"minor_comments":[{"comment":"The row entries \"s {synset wnid}\", \"c normalized score by L2 norm\", and \"ss s squared\" are not explained; the relation between s, ss, and c (presumably c = s/√ss) should be stated explicitly, and it is unclear whether \"ss\" is actually used by any query.","section":"Table I"},{"comment":"Eq. (3) contains malformed radical notation (\"/radicaltp /radicalvertex ...\") from the rendering pipeline; it should be a standard square-root symbol.","section":"Eq. (3)"},{"comment":"The term \"resolution rate\" is used without definition; state what resolution values 1.0, 0.8, etc. mean and how images were resized.","section":"Tables IV and Fig. 4"},{"comment":"Figure 5 and Section III refer to \"quartered partial images\" but do not specify how the quarter crops were extracted or how ground truth for partial-image queries was determined.","section":"Section III and Fig. 5"},{"comment":"The \"dot+cos\" baseline is not described; specify the top-k and re-ranking procedure used (e.g., the method of [7]), since its 35–53 s response times dominate Table II.","section":"Tables II–V"},{"comment":"The paper says \"the prototype system implementation is naive using script score of Elasticsearch\" but does not state the Elasticsearch version or mapping (e.g., whether doc values are enabled); this information is needed to assess the timing experiments.","section":"Section III"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short engineering contribution. The derivation is correct, but the experimental section is not at journal level: N and index size are not reported, there are no error bars, and the dataset in the experiments does not match the abstract. The author should be asked for a substantial revision. Note also that reference [9] is the author's own GitHub repository, which is not a peer-reviewed source; the novelty relative to Amato et al. [1] and Mu et al. [7] should be clarified. If the journal's bar for novelty is high, the contribution as currently written may be better suited to a workshop or short paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nWhat you should know: this is a short engineering note that applies a standard trick — cosine similarity as a dot product against L2-normalized vectors — to Elasticsearch. The math is correct, the exposition is clear, and avoiding the re-ranking step is a legitimate engineering goal. But the paper's central scalability claim rests on an untested sparsity assumption, and the evaluation is too thin to support the headline.\n\nThe derivation in Eqs. (5)–(6) is just the textbook identity cos(x,y) = x·(y/‖y‖) when x is fixed and y is pre-normalized. The authors implement this by storing the normalized components in the index and summing over query terms in a script_score. That is sound as far as it goes. The comparison against the dot+cos re-ranking baseline shows a large response-time gap, and the MAP numbers on the toy set suggest the method behaves sensibly at small scale.\n\nNow the soft spots. The abstract says ImageNet, but the experiments use a small Kaggle Dogs vs Cats set, with no reported database size, index size, or postings density. More damaging, the method's speed story assumes the 1,000-dimensional VGG-16 softmax vector can be turned into a few index terms. If every document stores a score for all 1,000 classes, the script_score over match_all becomes a columnar scan of all documents — O(N·D) — and the inverted index buys nothing. The 'Feature Number' truncation is never specified as query-side or index-side; if the indexed vector is truncated and not renormalized, the score computed is no longer the exact cosine of Eq. (6). The paper reports no evidence about sparsification, so the claimed advantage over a linear scan is uninterpretable.\n\nThese problems do not invalidate the core identity, but they do undermine the paper's main claim. The MAP of 1.0 on a tiny set is meaningless for generalizing. As it stands, this is a plausible prototype write-up, not a complete performance study.\n\nWho gets value: someone implementing or evaluating Elasticsearch for CBIR might use this as a starting point. I would not cite it in my own work, and I would not bring it to reading group. But the idea deserves a real referee if the authors respond to the sparsity concern: report index statistics, test with a public benchmark, and define the truncation/renormalization. I'd send it back for major revision, not desk-reject.","headline":"Standard cosine-as-dot-product trick applied to Elasticsearch; math is right, but the scalability claim rests on untested sparsity assumptions and a toy evaluation.","tokens_in":7079,"tokens_out":3249,"would_cite":false,"duration_ms":32414,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-indexing L2 norms turns cosine image retrieval into a single dot-product pass on an inverted index, eliminating re-ranking.","keywords":["content-based image retrieval","deep visual features","cosine similarity","inverted index","Elasticsearch","L2 normalization","VGG-16","mean average precision"],"falsifier":"Count the nonzero (or above a tiny threshold) components in the VGG-16 softmax vectors for a sample of ImageNet images: if the average count is close to 1,000 instead of a small fraction, the inverted index must touch nearly every dimension for every candidate, and the reported response times will not extend to larger collections. A scaling test that grows the collection by factors of 10 and watches whether retrieval time grows proportionally would settle the speed claim.","tokens_in":6000,"feed_emoji":"🖼️","tokens_out":9377,"duration_ms":89907,"temperature":0.7,"pith_summary":"The paper tries to establish that cosine similarity, not just inner product, can be the scoring function of an inverted-index search engine, by moving the expensive vector-length computation to indexing time. If the claim is right, content-based image retrieval with deep network features can run on disk-based indexes instead of main-memory-only systems, and the top results no longer need a second re-ranking pass. The paper reports the scheme on VGG-16 softmax features from ImageNet and finds that cosine scoring reaches perfect mean average precision on same-image queries once seven or more feature components are retained, while plain dot product stays near 0.05. Response times for cosine scoring stay in the same range as dot-product scoring, roughly 0.2 to 3 seconds for 100 top-k results, whereas the re-ranking baseline costs tens of seconds.","feed_headline":"Cosine image search runs directly on an inverted index","feed_subtitle":"Precomputed L2 norms turn cosine similarity into a dot product, so retrieval skips the slow re-ranking step.","key_machinery":"The load-bearing object is the L2-normalized indexed feature vector, $y'_i = y_i / \\|y\\|$, stored in the index at ingest time along with the squared norm. The identity $\\cos(x,y) \\propto \\sum_i x_i y'_i$ converts a cosine ranking into an inner-product ranking for a fixed query, which is exactly the kind of score an inverted-index engine can sum term by term. The implementation uses a script_score query on Elasticsearch that walks the stored per-synset scores of each candidate image, so no sequential norm computation and no top-k re-ranking is required.","core_discovery":"The central discovery is a rearrangement of cosine similarity that makes it compatible with inverted-index scoring. Since the query vector is fixed during a search, the query norm is constant, so ranking by cosine similarity is the same as ranking by the dot product of the query with the L2-normalized indexed vector: $\\cos(x,y) \\propto x \\cdot (y/\\|y\\|) = \\sum_i x_i y'_i$. If the normalized components $y'_i$ and the squared norm $\\|y\\|^2$ are stored as document fields at indexing time, the search engine can compute this score with a single scripted dot-product pass. The paper further rewrites Manhattan and Euclidean distances as inner-product terms plus precomputed document statistics. Experiments on a dog-and-cat image set show that the cosine scheme matches or beats dot product and re-ranking cosine on mean average precision while staying in the dot product's response-time range.","pith_inferences":["Because the query norm cancels from the ranking, a service that needs a literal cosine value can multiply the script score by $1/\\|x\\|$ at query time and still avoid re-ranking.","The same decomposition should apply to other embedding types, such as word2vec or document vectors, whenever the scoring metric is a cosine or a norm-expandable distance.","The method's speed rests on the indexed softmax vectors being sparse enough that only a few per-document scores are stored; a direct test would measure average retrieval time as the collection grows, which the paper does not report."],"forward_implications":["Cosine similarity becomes a first-class score for inverted-index retrieval: any engine that can sum per-term scores can rank by cosine without re-ranking.","Retrieval accuracy on same-image queries is effectively perfect (MAP 1.0) once at least seven softmax components are retained, compared with about 0.05 for the inner-product baseline.","Manhattan and Euclidean distances are also expressible as inner products plus precomputed document statistics, so one indexing scheme supports several distance metrics.","Response times stay in the dot product's range (about 0.2 to 3 seconds for 100 top-k results), avoiding the roughly 35 to 53 second penalty of the re-ranking baseline."],"supporting_citations":[{"why":"Shows that Elasticsearch can serve large-scale image retrieval, the engine and indexing direction this paper builds on.","marker":"[1]"},{"why":"Proposes cosine similarity through top-k re-ranking inside Elasticsearch; the method here replaces that re-ranking with a precomputed norm.","marker":"[7]"},{"why":"Describes the earlier VGG-16 image search system whose inner-product scoring motivated switching to cosine.","marker":"[9]"},{"why":"Supplies the ImageNet pretrained VGG-16 feature space whose 1,000-class softmax output defines the image descriptors.","marker":"[2]"},{"why":"Provides the dog-and-cat image collection used to measure mean average precision and response time.","marker":"[3]"},{"why":"An alternative dimensionality-compression retrieval method that the paper positions against, motivating index-side normalization.","marker":"[4]"}],"fun_headline_variants":["Cosine similarity folds into dot product for instant image search","Image retrieval skips re-ranking via precomputed L2 norms","Elasticsearch cosine search without the re-ranking bottleneck","Inverted index computes cosine similarity via one dot product"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on each image's 1,000-dimensional feature vector being representable as a small number of indexable scores; if those vectors are dense, the disk-based inverted index will not speed up retrieval.","fun_headline_variants_meta":{"raw":{"variants":["Cosine similarity folds into dot product for instant image search","Image retrieval skips re-ranking via precomputed L2 norms","Elasticsearch cosine search without the re-ranking bottleneck","Inverted index computes cosine similarity via one dot product"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000531,"raw_usage":{"total_tokens":2512,"prompt_tokens":854,"completion_tokens":1658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":1592}},"tokens_in":470,"tokens_out":1658,"duration_ms":12336,"temperature":1.0,"reasoning_tokens":1592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:10:47.411778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the nonzero (or above a tiny threshold) components in the VGG-16 softmax vectors for a sample of ImageNet images: if the average count is close to 1,000 instead of a small fraction, the inverted index must touch nearly every dimension for every candidate, and the reported response times will not extend to larger collections. A scaling test that grows the collection by factors of 10 and watches whether retrieval time grows proportionally would settle the speed claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposes cosine similarity through top-k re-ranking inside Elasticsearch; the method here replaces that re-ranking with a precomputed norm."},{"cited_title":"Tanioka, ``Super easy way of building image search with keras,'' Aug","cited_arxiv_id":null,"evidence_quote":"Describes the earlier VGG-16 image search system whose inner-product scoring motivated switching to cosine."},{"cited_title":"(2014) Large scale visual recognition challenge (ilsvrc)","cited_arxiv_id":null,"evidence_quote":"Supplies the ImageNet pretrained VGG-16 feature space whose 1,000-class softmax output defines the image descriptors."},{"cited_title":"(2014) Dogs vs","cited_arxiv_id":null,"evidence_quote":"Provides the dog-and-cat image collection used to measure mean average precision and response time."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"An alternative dimensionality-compression retrieval method that the paper positions against, motivating index-side normalization."}],"review_version":1}