Pith. sign in

REVIEW 3 major objections 7 minor 21 references

Multimodal semantic retrieval for product search

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding product images to text embeddings improves retrieval precision in product search.

desk verdict Modest but honest extension of Amazon's multimodal retrieval work; the either/or claim needs statistical discipline before the numbers can be trusted. read the letter →

arxiv 2501.07365 v3 pith:A6KINNFL submitted 2025-01-13 cs.IR cs.LG

Dong Liu , Esther Lopez Ramos This is my paper · ORCID
classification cs.IRcs.LG
keywords multimodalretrievalproductsearchsemanticdensee-commercecontrastivelearningCLIPfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a product's image, not just its title text, carries useful signal for semantic (dense-vector) retrieval in e-commerce search. It builds several fused representations: a text-only baseline, a 4-tower model combining a text encoder and a pretrained vision-language encoder, and a 3-tower variant, and evaluates them on millions of products. The central result is that the best fused variant raises exact-match precision from 52.7% to 54.0% and lowers irrelevant results from 13.6% to 11.9%, at the cost of some purchase recall. A sympathetic reader would care because product images are known to matter for shoppers, and the question is whether they can be exploited without hurting scalable retrieval.

What carries the argument

The central objects are the two encoder stacks: a text-only bi-encoder model for query and product text, and CLIP, a pretrained vision-language model whose text and image towers are independently initialized. The paper's mechanism is a fusion module—concatenation, alpha-weighted concatenation, or an MLP—that combines text and image embeddings into a single product vector, trained with the normalized temperature-scaled cross entropy (NT-Xent) contrastive loss. A 4-tower model fuses text-only embeddings with CLIP text for the query and with CLIP image for the product; a 3-tower model drops the CLIP text tower and lets an MLP align the CLIP image embedding to the text-only vector space. Fine-tuning choices (whether the text encoder and the MLP are jointly trained, and whether extra negative examples are seen) determine whether the gain lands on recall or on precision.

What would settle it

Re-run the best 4-tower joint-fine-tuned configuration and the text-only baseline on the same 38,268 queries with five different random seeds and bootstrap-resample the query set; if the 95% confidence intervals for exact-match precision and recall overlap between the two models, the claimed improvement fails. A cheaper observational check: take the exclusive matches of the multimodal model and have human annotators label them independently of the automation used in the paper; if exact-plus-substitute rates drop to near baseline levels, the relevance gain is an artifact of the annotation model.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that multimodal representations—where a product is embedded by fusing text embeddings with image embeddings from a pretrained vision-language encoder—can beat a text-only semantic retrieval model on either purchase recall or relevance accuracy, depending on how the fusion and fine-tuning are done. The best configuration, a 4-tower model with joint fine-tuning of the text encoder and an MLP fusion module, improves exact-match precision from 52.7% to 54.0%, cuts irrelevant predictions from 13.6% to 11.9%, and degrades recall@100 from 78.1% to 73.3%. Exclusive matches—products retrieved by the multimodal model but not by the text baseline—remain mostly exact or substitute, which the paper reads as evidence that images add genuinely new relevant results.

Load-bearing premise

The whole conclusion rests on the assumption that the evaluation metrics—purchase records as recall ground truth and the relevance annotation model for precision—measure retrieval quality well enough that a 1.3-point precision gain and 0.5-point recall gain are real signals rather than evaluation noise, and the paper reports no error bars or significance tests.

Editorial extensions

If this is right

  • If correct, e-commerce search engines can add image signal to dense retrieval without switching to expensive late-interaction scoring; cosine similarity over the fused vectors stays compatible with large-scale nearest-neighbor indexes.
  • The winning configuration implies that aligning a public vision encoder to an in-domain text encoder matters more than simply concatenating features; joint fine-tuning is the load-bearing step.
  • The 3-tower result shows a cheaper architecture (no CLIP text tower) can nearly match the 4-tower model after joint fine-tuning, so image-only injection may be sufficient in practice.
  • Because the gains are concentrated in exact and substitute matches among exclusive retrievals, image-aware retrieval may be most useful for queries where the right product is visually distinctive but poorly described in text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported differences are single point estimates without error bars or significance tests; a natural next step is to run multiple seeds or bootstrap over queries to see whether the 1.3-point exact-match gain is stable.
  • A testable extension: train the 4-tower model with query images (as in the related 4-tower work) and measure whether image-to-image matching further improves substitute and complement relevance, which the paper did not study.
  • The net-analysis numbers suggest a possible evaluation protocol for retrieval models: report precision on exclusive matches, since it separates new information from re-ranking the same set; future work could adopt this as a standard metric.
  • The paper's premise that purchase records proxy relevance is strongest for purchased items; an independent human-annotation study on a held-out query set would directly test whether the relevance gains generalize beyond the annotation model used here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies whether adding product images to a text-only semantic retrieval model improves e-commerce product search. It proposes two families of multimodal architectures: a 4-tower model (4tMM) that fuses BiBERT text embeddings with CLIP text/image embeddings, and a 3-tower model (3tMM) that fuses product text and image embeddings without a CLIP text tower. The models are evaluated on an internal e-commerce dataset with 38,268 queries and 3,384,067 products, using recall@100 against purchase ground truth and relevance labels from a proprietary annotation model. The main empirical claim is that a multimodal scheme can improve either purchase recall or relevance accuracy relative to the text-only BiBERT baseline. A further 'exclusive matches' analysis reports the quality of products retrieved by multimodal models but not by BiBERT.

Significance. If the point estimates are reliable, the paper offers a scalable empirical comparison of image-plus-text fusion strategies for dense product retrieval, and the 4tMM/3tMM contrast speaks to a practical question about encoder alignment. The use of a large product index and standard FAISS KNN retrieval makes the benchmark concrete, and the general architecture recipe is described enough to reproduce. The main limitation is that the evidence for the central claim is thin: the headline gains are 0.5 percentage points of recall and 1.3 percentage points of exact precision, and no uncertainty estimates are provided. The paper is therefore more convincing as a descriptive study of possible architectures than as a demonstration that images reliably add signal beyond text.

major comments (3)
  1. [Sections 4.2-4.3, Table 2] The central claim rests on two small point-estimate gains: 4tMM cat improves recall@100 from 78.1 to 78.6 (+0.5), and 4tMM with joint fine-tuning improves Exact from 52.7 to 54.0 (+1.3) while its recall falls to 73.3. Because the claim is disjunctive and Table 2 evaluates eleven model rows, selecting the best row on each metric effectively becomes a multiple-comparison search. Without confidence intervals, paired significance tests, or at least bootstrap error bars over queries, the reported differences are within the range that could be produced by evaluation noise. Please report query-level uncertainty and either apply a multiple-comparison correction or explicitly rephrase the conclusion as 'there exist configurations that improve one metric on this dataset.'
  2. [Section 4.4, Table 4] The rows '4eMM BiBERT-MLP jointly trained' and '3eMM BiBERT-MLP jointly trained' report identical Net Recall (56.2), Net Exact (57.6), Net Substitute (28.7), and Net Complement (2.2), with Irrelevant differing only by 0.1. Since these two architectures have different overall metrics in Table 2, the exclusive-match results need either a correction or an explicit explanation (for example, if rounding produces identical values). As printed, the table suggests a copy-paste or labeling artifact and undermines confidence in the Section 4.4 analysis.
  3. [Section 4.2] The relevance labels are produced by a proprietary annotation model called Sentinel, but the paper provides no validation of this model against human judgments. Since Exact precision is one of the two headline metrics in the abstract's claim, the relevance comparison is only as trustworthy as that labeler. Please report at least a sample-based human-agreement measure, or explicitly state the limitation and avoid presenting the relevance numbers as validated retrieval quality.
minor comments (7)
  1. [Throughout] The model names are inconsistent: '4eMM' and '3eMM' appear alongside '4tMM' and '3tMM'; please standardize to one notation.
  2. [Table 2] The checkmark columns are ambiguous because two rows labeled '4tMM ✓ ✓' have different metrics; add a legend or an explicit per-row description of which components are tuned/trained.
  3. [Section 2.1, Eq. (1)] The index notation for the positive document is inconsistent: the text says '𝒅+_j denotes the i-th relevant target', but the subscript should be tied to the query index; please fix.
  4. [Sections 4.1 and 4.2] There are several typos and formatting issues: 'statics' should be 'statistics', 'datset' should be 'dataset', 'bettwen' should be 'between', 'learning rate of1e−4' is missing a space and preposition, and 'pytorch lighgning' should be 'PyTorch Lightning'.
  5. [Section 4.4, Table 3] The sentence 'the overlapping predictions are low' is at odds with a mean overlap of 39 out of 100 between BiBERT and 4tMM cat; please rephrase or specify what 'low' means in this context.
  6. [Section 4.4] The definition of Net Purchase Recall should state explicitly whether the denominator is the total number of purchases per query or the number of purchases among the exclusive predictions; the current wording is ambiguous.
  7. [Section 2.2] The CLIP model is cited to reference [5], which is a Data Filtering Networks paper; please clarify which specific CLIP checkpoint or variant (e.g., DataComp) was used, since this affects reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are direct empirical measurements on a held-out evaluation set, with no derivation reducing to fitted inputs or self-citations.

full rationale

The paper makes an empirical claim that some multimodal variant improves either purchase recall or relevance precision over the BiBERT text-only baseline. This is supported by held-out test-set measurements in Table 2: training and evaluation time windows do not overlap, all models are scored on the same 38,268 queries and 3.38M products, and recall/precision are computed from purchase records and the Sentinel relevance annotator rather than from any equation fitted to those numbers. The fusion equations (Eq. 2-5) define model architectures; they are not used to derive the claimed improvements. The 'Net' metrics in Section 4.4 are descriptive post-hoc analyses of exclusive matches, not predictions, so they cannot reduce to their own inputs. The paper cites prior work from the same company (e.g., [20]) as related work and for context, but the central conclusion does not depend on any theorem, ansatz, or fitted value imported from those citations. Concerns about missing error bars or multiple-comparison selection among the 11 configurations are statistical validity issues, not circularity. No self-definitional, fit-renamed-as-prediction, or self-citation-load-bearing step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the validity of internal relevance labels and the transferability of CLIP representations to e-commerce. No new entities are introduced.

assumptions (3)
  • domain assumption Purchase records and the Sentinel annotation model provide reliable relevance labels.
    The recall and precision metrics are computed against these labels; if they are noisy or biased, the reported improvements may not reflect true retrieval quality. Introduced in Sections 4.1 and 4.2.
  • domain assumption CLIP's public pretrained representations transfer to e-commerce images and text.
    The visual encoder is used without fine-tuning on e-commerce data; the paper's conclusions depend on CLIP features being meaningful for product images. Assumed in Section 3.
  • domain assumption The evaluation dataset (one month of interactions) is representative and non-overlapping with training.
    The test set is a single non-overlapping month; if it differs from typical traffic, the results may not generalize. Stated in Section 4.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal semantic retrieval for product search." pith.science (2026). https://pith.science/paper/A6KINNFL

@misc{pith2026250107365,
  author       = {Pith},
  title        = {Pith review of: Multimodal semantic retrieval for product search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6KINNFL}},
  note         = {Machine review of arXiv:2501.07365}
}
read the original abstract

Semantic retrieval (also known as dense retrieval) based on textual data has been extensively studied for both web search and product search application fields, where the relevance of a query and a potential target document is computed by their dense vector representation comparison. Product image is crucial for e-commerce search interactions and is a key factor for customers at product explorations. However, its impact on semantic retrieval has not been well studied yet. In this research, we build a multimodal representation for product items in e-commerce search in contrast to pure-text representation of products, and investigate the impact of such representations. The models are developed and evaluated on e-commerce datasets. We demonstrate that a multimodal representation scheme for a product can show improvement either on purchase recall or relevance accuracy in semantic retrieval. Additionally, we provide numerical analysis for exclusive matches retrieved by a multimodal semantic retrieval model versus a text-only semantic retrieval model, to demonstrate the validation of multimodal solutions.

Figures

Figures reproduced from arXiv: 2501.07365 by the authors.

Figure 1
Figure 1. Two multimodal model architectures for semantic retrival [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 13 canonical work pages

  1. [20]

    Bringing multimodality to amazon visual search system

    Xinliang Zhu, Sheng-Wei Huang, Han Ding, Jinyu Yang, Kelvin Chen, Tao Zhou, Tal Neiman, Ouye Xie, Son Tran, Benjamin Yao, et al. Bringing multimodality to amazon visual search system. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 6390–6399, 2024

    work page 2024

  2. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  3. [2]

    Extreme multi-label learning for semantic matching in product search

    Wei-Cheng Chang, Daniel Jiang, Hsiang-Fu Yu, Choon Hui Teo, Jiong Zhang, Kai Zhong, Kedarnath Kolluri, Qie Hu, Nikhil Shandilya, Vyacheslav Ievgrafov, et al. Extreme multi-label learning for semantic matching in product search. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2643–2651, 2021

    work page 2021

  4. [3]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597–1607. PMLR, 2020. WWW Companion ’25, April 28-May 2, 2025, Sydney, NSW, Australia Dong et al

    work page 2020

  5. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

  6. [5]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023

    arXiv 2023

  7. [6]

    Learning deep structured semantic models for web search using clickthrough data

    Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2333–2338, 2013

    work page 2013

  8. [7]

    Bert: Pre- training of deep bidirectional transformers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2, 2019

    work page 2019

Show all 21 references
  1. [8]

    Relevance-guided supervi- sion for openqa with colbert

    Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervi- sion for openqa with colbert. Transactions of the association for computational linguistics, 9:929–944, 2021

    2021

  2. [9]

    Vilt: Vision-and-language transformer without convolution or region supervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. InInternational conference on machine learning, pages 5583–5594. PMLR, 2021

    2021

  3. [10]

    Embracing structure in data for billion-scale semantic product search

    Vihan Lakshman, Choon Hui Teo, Xiaowen Chu, Priyanka Nigam, Abhinandan Patni, Pooja Maknikar, and SVN Vishwanathan. Embracing structure in data for billion-scale semantic product search. arXiv preprint arXiv:2110.06125, 2021

    2021 arXiv

  4. [11]

    BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and gen- eration

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and gen- eration. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings...

    2022

  5. [12]

    Embedding-based product retrieval in taobao search

    Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. Embedding-based product retrieval in taobao search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3181–3189, 2021

    2021

  6. [13]

    Deep self-adaptive hashing for image retrieval

    Qinghong Lin, Xiaojun Chen, Qin Zhang, Shangxuan Tian, and Yudong Chen. Deep self-adaptive hashing for image retrieval. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 1028– 1037, 2021

    2021

  7. [14]

    One picture is worth a thousand words? the pricing power of images in e-commerce

    Christof Naumzik and Stefan Feuerriegel. One picture is worth a thousand words? the pricing power of images in e-commerce. InProceedings of The Web Conference 2020, WWW ’20, page 3119–3125, New York, NY, USA, 2020. Association for Computing Machinery

    2020

  8. [15]

    Dc-bert: Decoupling question and document for efficient contextual encoding

    Ping Nie, Yuyu Zhang, Xiubo Geng, Arun Ramamurthy, Le Song, and Daxin Jiang. Dc-bert: Decoupling question and document for efficient contextual encoding. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 1...

    2020

  9. [16]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  10. [17]

    Introduction to information retrieval, volume 39

    Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge, 2008

    2008

  11. [18]

    Rep- bert: Contextualized text embeddings for first-stage retrieval

    Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. Rep- bert: Contextualized text embeddings for first-stage retrieval. arXiv preprint arXiv:2006.15498, 2020

    2006 arXiv

  12. [19]

    Binary neural network hashing for image retrieval

    Wanqian Zhang, Dayan Wu, Yu Zhou, Bo Li, Weiping Wang, and Dan Meng. Binary neural network hashing for image retrieval. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 1318–1327, 2021

    2021

  13. [21]

    Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks

    Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.