Pith. sign in

REVIEW 3 major objections 5 minor 23 references

SIR: Similar Image Retrieval for Product Search in E-Commerce

T0 review · 3 major / 5 minor · reviewed 2026-08-27 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining image and text retrieval raises e-commerce variant recall by 24 percent, using binarized deep-learning embeddings in a production search system.

desk verdict A transparent industrial systems paper whose headline complementarity claim is undercut by an unequal candidate-budget comparison; worth peer review as an application study, not for algorithmic novelty. read the letter →

arxiv 2009.13836 v1 pith:BQT557EU submitted 2020-09-29 cs.CV cs.AIcs.IRcs.LG

Theban Stanley , Nihar Vanjara , Yanxin Pan , Ekaterina Pirogova , Swagata Chakraborty , Abon Chaudhuri This is my paper · ORCID
classification cs.CVcs.AIcs.IRcs.LG
keywords similarimageretrievalproductsearche-commerceapproximatenearestneighborlocalitysensitivehashingembeddingbinarizationElasticsearchvariantdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents SIR, a similar-image retrieval system that lets users search a catalog of millions of products by example rather than by a predefined label. The paper claims that binarized VGG16 embeddings, split into subcodes and indexed in a full-text search engine, retrieve visually similar products with accuracy close to unmodified embeddings while cutting query latency by up to 70 percent. In a variant-detection application, image-based candidates fetch items that text search misses, and combining image and text retrieval raises candidate recall by 24 percent over text alone. The authors argue this makes SIR well suited to ephemeral, hard-to-define search themes such as offensive-product detection, where training a supervised model is too slow.

What carries the argument

The central mechanism is an LSH-style encoding pipeline: each product image becomes a high-dimensional VGG16 embedding, which is then binarized through random projections and split into 256 or 512 subcodes. These subcodes are stored in an Elasticsearch index, whose bitwise Hamming-distance operations and pigeonhole-based early abandonment turn floating-point nearest-neighbor search into sub-linear approximate retrieval. The pipeline runs as a rolling process over a Kafka stream of new and updated images, keeping the index current without recomputing a global representation such as PCA. This lets a query image return near-real-time results on a 1.5-million-image index.

What would settle it

Rerun the variant-candidate experiment with an exhaustive ground-truth comparison: check whether any known relevant variant appears beyond rank 1000 by searching the full index rather than assuming the top 1000 contains all relevant items. If any relevant item is found outside the top 1000, the reported approximate recall and the 24 percent improvement over text search would be inflated.

Watch

Extended reading notes

Core claim

The central claim is that an approximate nearest-neighbor search over binarized deep-learning embeddings, implemented on a full-text search index, is accurate and fast enough to drive product discovery in a large, constantly changing e-commerce catalog. Using VGG16 embeddings from the final fully connected layer, the system applies a locality-sensitive hashing variant that binarizes the vector with random projections and splits it into subcodes, enabling Elasticsearch to enforce early abandonment via the pigeonhole principle and to compute Hamming distances with bit operations. In the authors' measurements, VGG16 with 256 subcodes retains approximately 98 percent of raw-embedding recall while reducing mean query time by about 70 percent. In the variant-grouping application, SIR retrieved candidates that text search missed, and the combined image-plus-text candidate set increased recall by 24 percent over text alone; in the offensive-product application, the same index supports both streaming checks on new items and full-catalog sweeps.

Load-bearing premise

All reported recall figures assume that every relevant product appears somewhere in the top 1000 retrieved results; if some relevant items fall beyond that cutoff, the recall numbers and the claimed 13–24 percent improvements would be inflated.

Editorial extensions

If this is right

  • Variant grouping can use SIR's image candidates to push recall of the high-recall stage beyond what text-only retrieval achieves, giving the precision classifiers a more complete candidate pool.
  • Ephemeral compliance themes can be searched the moment they arise using one or two example images, without the delay of collecting labels and training a classifier.
  • Applying text filters before image search cuts query latency as the catalog grows, so a multimodal image-plus-text index is a practical scaling path.
  • Subcoded binarization preserves most of the accuracy of raw embeddings while reducing storage and query cost dramatically, making deep similarity search feasible on commodity text-search infrastructure.
  • The same rolling index can support data augmentation, helping data scientists find examples for under-represented classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The three-month rolling index implicitly assumes that recent catalog images are the ones that matter for these applications; a testable implication is that search quality for older products would degrade, which may or may not matter for the stated use cases.
  • The complementarity of image and text retrieval suggests that in marketplaces where different sellers describe the same product differently, visual similarity is a more stable signal than text; this could generalize beyond the paper's variant grouping to duplicate detection or product matching.
  • Because the exact precision, recall, and latency numbers come from private production data, they are not independently reproducible; a public benchmark on a fashion or product image dataset with the same pipeline would show whether the 24 percent improvement holds outside this catalog.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents SIR, a deployed image-similarity retrieval system for an e-commerce catalog. Product images are converted to VGG16 embeddings, binarized with random projections, split into subcodes, and indexed in Elasticsearch for approximate nearest-neighbor search. The paper reports two production applications: detection of non-compliant or offensive products for ephemeral themes, and generation of variant candidates for product grouping. For variant detection, the authors claim that image-based retrieval alone achieves 13% higher recall than a text-based system, and that combining image and text candidates improves recall by 24% over text alone. The evaluation includes precision/recall comparisons over embedding types and subcoding levels, plus query-latency measurements.

Significance. The paper is a useful systems and application contribution. It demonstrates a concrete, scalable recipe—CNN embeddings, random-projection binarization, subcoding, and Elasticsearch—for visual search over millions of images, and it is honest about deployment constraints and limitations, including the tablet-content failure in Figure 7d and the volatility of compliance themes. If the 13% and 24% variant-recall gains survive equal-budget evaluation, the image-text complementarity result would be practically important for e-commerce variant grouping. However, the paper does not release code or data, and the key quantitative claims currently rest on an unequal-candidate-budget comparison and an unvalidated approximate-recall assumption, so the significance is that of a promising system report rather than a fully established result.

major comments (3)
  1. [Section 4.2, Performance Analysis] The comparison of candidate generators uses unequal budgets: approximately 1,000 image-based candidates versus approximately 500 text-based candidates per reference item. Since recall@K is monotonically non-decreasing in K, the reported 13% image-alone advantage, and at least part of the 24% combined gain, could be a budget effect rather than evidence of visual complementarity. The authors acknowledge the discrepancy but do not correct for it. They should re-run the comparison at equal candidate budgets (e.g., 500 vs 500), or use the recall-versus-candidate-count curves in Figures 5 and 6 to interpolate the text recall at 1,000 candidates; this is needed to support the paper's central claim that image search retrieves candidates that complement text search.
  2. [Section 4.1, Analysis of Search Quality] The 'approximate recall@1000' is defined by assuming 'that all the relevant documents are either returned within top 1000 or not.' This is an upper-bound assumption: relevant items ranked beyond 1000 are never observed, so any such items would inflate the reported recall values (e.g., 0.989 for VGG16 in Table 1). The authors should either justify the assumption with a sampled check of how often relevant items appear beyond rank 1000, or report a conservative recall that treats unobserved items as potentially relevant.
  3. [Section 4.2, Performance Analysis] The headline 13% and 24% recall improvements are reported as point estimates without confidence intervals, error bars, or statistical tests, despite being computed over roughly 5,000 variant groups. Given the diversity of product categories, the gain could be concentrated in a few categories; a per-category breakdown or bootstrap intervals should be reported to establish that the aggregate improvement is stable.
minor comments (5)
  1. [Section 5] There is a typo: 'can We also intend to scale the embedding computation' should read 'We also intend to scale the embedding computation.'
  2. [Tables 1 and 3] Model names are inconsistent: Table 1 reports 'Inception-v2' and Table 3 reports 'Inception-v3'; please unify the naming.
  3. [Figure 3] The caption does not describe the axes or the meaning of 'data size'; please add axis labels and clarify whether the horizontal axis refers to the number of indexed images or some other quantity.
  4. [Section 4.1] The ground-truth setup ('10000 offensive images ... related to 3600 query images') is ambiguous: it is not clear whether each query has a variable number of relevant images or whether the 10,000 images form a shared pool. Please clarify the construction.
  5. [General] No code, data, or configuration files are released, which limits reproducibility; at minimum, the embedding extraction and subcoding parameters should be fully specified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: SIR's retrieval quality is measured against external ground truth, and the core indexing components are cited from external work rather than defined in terms of the paper's conclusions.

full rationale

The paper makes no first-principles prediction; it reports an engineered system built from pre-trained VGG16 embeddings, random-projection binarization, subcodes, and Elasticsearch, and evaluates retrieval quality against ground-truth sets of offensive images and production variant groups. The core algorithm analysis is explicitly delegated to external work (Mu et al., ref [16]), and no parameter is fitted to the reported evaluation sets and then renamed as a prediction. The approximate recall@1000 assumption is stated transparently in Section 4.1, and the image-vs-text candidate comparison in Section 4.2 uses 1000 image versus 500 text candidates; while this is a legitimate measurement-fairness concern that could affect the reported 13-24% recall differences, it is not circularity because recall@K is a measured outcome rather than an input that defines the system's outputs. The only self-citations are prior Walmart product-image papers used for supervised-model alternatives (refs [5] and [8]); they are not load-bearing for SIR's central retrieval claim. Hence no quoted step reduces a claimed result to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The claims rest on engineering choices (embedding backbone, subcode count, candidate counts, rolling window), on external results for LSH and subcoded search, and on the assumption that internal ground truth labels are reliable.

free parameters (4)
  • Embedding backbone = VGG16
    Chosen after comparing five networks on internal data; the reported quality numbers depend on this model selection.
  • Number of subcodes = 512 and 256
    Hand-tuned trade-off between latency and precision; Tables 2 and 3 directly depend on this choice.
  • Candidate counts N and K in suggestion-of-suggestions = Not specified
    Values chosen during experiments; the recall gains in Section 4.2 depend on these counts.
  • Rolling index window = 3 months
    Chosen to cover recently added products; affects recall for compliance screening.
assumptions (4)
  • standard math Random projection LSH preserves approximate cosine similarity
    The binarization in Section 3.2 relies on this result, citing Gionis et al. [9].
  • standard math Subcoding with early abandonment enables sub-linear search
    The performance claims for Elasticsearch depend on the subcoded search algorithm from Mu et al. [16].
  • domain assumption ImageNet-pretrained VGG16 embeddings transfer to product images
    Section 3.1 uses the VGG16 final layer without fine-tuning; the entire retrieval quality rests on this transfer.
  • domain assumption Internal ground truth for offensive themes and variant groups is correct
    All precision and recall numbers are computed against internal labels, and no audit of that ground truth is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SIR: Similar Image Retrieval for Product Search in E-Commerce." pith.science (2026). https://pith.science/paper/BQT557EU

@misc{pith2026200913836,
  author       = {Pith},
  title        = {Pith review of: SIR: Similar Image Retrieval for Product Search in E-Commerce},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQT557EU}},
  note         = {Machine review of arXiv:2009.13836}
}
read the original abstract

We present a similar image retrieval (SIR) platform that is used to quickly discover visually similar products in a catalog of millions. Given the size, diversity, and dynamism of our catalog, product search poses many challenges. It can be addressed by building supervised models to tagging product images with labels representing themes and later retrieving them by labels. This approach suffices for common and perennial themes like "white shirt" or "lifestyle image of TV". It does not work for new themes such as "e-cigarettes", hard-to-define ones such as "image with a promotional badge", or the ones with short relevance span such as "Halloween costumes". SIR is ideal for such cases because it allows us to search by an example, not a pre-defined theme. We describe the steps - embedding computation, encoding, and indexing - that power the approximate nearest neighbor search back-end. We also highlight two applications of SIR. The first one is related to the detection of products with various types of potentially objectionable themes. This application is run with a sense of urgency, hence the typical time frame to train and bootstrap a model is not permitted. Also, these themes are often short-lived based on current trends, hence spending resources to build a lasting model is not justified. The second application is a variant item detection system where SIR helps discover visual variants that are hard to find through text search. We analyze the performance of SIR in the context of these applications.

Figures

Figures reproduced from arXiv: 2009.13836 by the authors.

Figure 1
Figure 1. System architecture of similar image retrieval tool (SIR). The top block [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Process of generation of subcodes from an image embedding followed by [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Query performance on different index sizes with and without text based [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The query product (on the right) and the variant candidate (on the left) [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Recall of any retrieval system saturates as we keep increasing number [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Image and text together increases recall of text retrieval system with [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: A number of examples demonstrating product search capability of SIR [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

  1. [1]

    https://kafka.apache.org/ (2011)

    Apache Software Foundation: Apache kafka. https://kafka.apache.org/ (2011)

    work page 2011

  2. [2]

    Babenko, A., Slesarev, A., Chigorin, A., Lempitsky, V.: Neural codes for image retrieval (2014)

    work page 2014

  3. [3]

    In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition

    Cao, Y., Long, M., Liu, B., Wang, J.: Deep cauchy hashing for hamming space retrieval. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 1229–1237 (2018)

    work page 2018

  4. [4]

    In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    Cao, Y., Long, M., Wang, J., Liu, S.: Deep visual-semantic quantization for effi- cient image retrieval. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 916–925 (2017)

    work page 2017

  5. [5]

    A Smart System for Selection of Optimal Product Images in E-Commerce

    Chaudhuri, A., Messina, P., Kokkula, S., Subramanian, A., Krishnan, A., Gandhi, S., Magnani, A., Kandaswamy, V.: A smart system for selection of optimal product images in e-commerce. CoRR abs/1811.07996 (2018), http://arxiv.org/abs/ 1811.07996

    work page Pith review arXiv 2018

  6. [6]

    2453–2453

    Craswell, N.: R-Precision, pp. 2453–2453. Springer US, Boston, MA (2009) 14 T. Stanley et al

    work page 2009

  7. [7]

    https://www.elastic.co/elasticsearch/ (2010)

    Elastic.co: Elasticsearch. https://www.elastic.co/elasticsearch/ (2010)

    work page 2010

  8. [8]

    In: IEEE Winter Conference on Ap- plications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020

    Gandhi, S., Kokkula, S., Chaudhuri, A., Magnani, A., Stanley, T., Ahmadi, B., Kandaswamy, V., Ovenc, O., Mannor, S.: Scalable detection of offensive and non- compliant content / logo in product images. In: IEEE Winter Conference on Ap- plications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020. pp. 2236–2245. IEEE (2020)

    work page 2020

Show all 23 references
  1. [9]

    Gionis, A., Indyk, P., Motwani, R.: Similarity search in high dimensions via hash- ing. pp. 518–529 (1999)

    1999

  2. [10]

    CoRR abs/1512.03385 (2015), http://arxiv.org/abs/1512.03385

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015), http://arxiv.org/abs/1512.03385

    2015 arXiv

  3. [11]

    CoRR abs/1505.07647 (2015), http://arxiv.org/abs/ 1505.07647

    Jing, Y., Liu, D.C., Kislyuk, D., Zhai, A., Xu, J., Donahue, J., Tavel, S.: Vi- sual search at pinterest. CoRR abs/1505.07647 (2015), http://arxiv.org/abs/ 1505.07647

    2015 arXiv

  4. [12]

    arXiv preprint arXiv:1702.08734 (2017)

    Johnson, J., Douze, M., J´ egou, H.: Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734 (2017)

    2017 arXiv

  5. [13]

    Malkov, Y.A., Yashunin, D.A.: Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs (2016)

    2016

  6. [14]

    Cambridge University Press, USA (2008)

    Manning, C.D., Raghavan, P., Sch¨ utze, H.: Introduction to Information Retrieval. Cambridge University Press, USA (2008)

    2008

  7. [15]

    CoRR abs/1906.10095 (2019), http://arxiv.org/abs/1906.10095

    Mu, C., Yang, B., Yan, Z.: An empirical comparison of FAISS and FENSHSES for nearest neighbor search in hamming space. CoRR abs/1906.10095 (2019), http://arxiv.org/abs/1906.10095

    2019 arXiv

  8. [16]

    Mu, C., Zhao, J., Yang, G., Yang, B., Yan, Z.: Fast and exact nearest neighbor search in hamming space on full-text search engines (2019)

    2019

  9. [17]

    CoRR abs/1409.1556 (2014)

    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)

    2014 arXiv

  10. [18]

    CoRR abs/1512.00567 (2015), http: //arxiv.org/abs/1512.00567

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the in- ception architecture for computer vision. CoRR abs/1512.00567 (2015), http: //arxiv.org/abs/1512.00567

    2015 arXiv

  11. [19]

    CoRR abs/1905.11946 (2019), http://arxiv.org/abs/1905.11946

    Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR abs/1905.11946 (2019), http://arxiv.org/abs/1905.11946

    2019 arXiv

  12. [20]

    CoRR abs/1706.03154 (2017), http://arxiv.org/ abs/1706.03154

    Yang, F., Kale, A., Bubnov, Y., Stein, L., Wang, Q., Kiapour, M.H., Piramuthu, R.: Visual search at ebay. CoRR abs/1706.03154 (2017), http://arxiv.org/ abs/1706.03154

    2017 arXiv

  13. [21]

    Zauner, C.: Implementation and benchmarking of perceptual image hash functions (2010)

    2010

  14. [22]

    In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    Zhang, Y., Pan, P., Zheng, Y., Zhao, K., Zhang, Y., Ren, X., Jin, R.: Visual search at alibaba. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. p. 9931001. KDD 18 (2018)

    2018

  15. [23]

    In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence

    Zhu, H., Long, M., Wang, J., Cao, Y.: Deep hashing network for efficient sim- ilarity retrieval. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. p. 24152421. AAAI’16, AAAI Press (2016)

    2016

Pith tools

Reviewed August 27, 2026 · model on record in the stance chip above.