Pith. sign in

REVIEW 4 major objections 7 minor 29 references

Semantic-enhanced Modality-asymmetric Retrieval for Online E-commerce Search

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding item images to a text-only retrieval tower raises recall@50 from 0.641 to 0.690 on an industrial click-log dataset.

desk verdict A practical multimodal retrieval extension with a credible offline story and a real dataset release, but the online gain claim is weakened by adding extra candidates only in the test arm. read the letter →

arxiv 2506.20330 v1 pith:OIHWW72K submitted 2025-06-25 cs.IR

Zhigong Zhou , Ning Ding , Xiaochuan Fan , Yue Shang , Yiming Qiu , Jingwei Zhuo , Zhiwei Ge , Songlin Wang
show 3 more authors
Lin Liu Sulong Xu Han Zhang
This is my paper · ORCID
classification cs.IR
keywords e-commercesearchsemanticretrievalmultimodalmodality-asymmetriccross-modalalignmenttransformercross-attentionadaptiveimagegatingtwo-towermodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that in e-commerce search, a text-only user query can retrieve better when the item's image is fused into its representation, even though the query never contains an image. The proposed model, SMAR, uses a two-stage training scheme: a multi-task pretraining stage aligns text-query, item-text, item-image, and fused multimodal embeddings, and a fine-tuning stage gates image usage by a category-aware predictor. On a released industrial click-log dataset, SMAR reaches recall@50 of 0.690 versus 0.641 for the text-only DPSR baseline, a 4.9% relative gain, with larger relative gains on the not-fashion split; an online A/B test shows higher gross merchandise value and user conversion rate. If the result holds, it means retrieval systems can exploit product images without requiring image queries or symmetric multimodal inputs.

What carries the argument

The load-bearing mechanism is the item multimodal tower, which fuses text and image by feeding text hidden states as queries and image hidden states as keys/values into a transformer layer, so the image contributes only where text attends to it. Around this tower, SMAR organizes three pretraining tasks—semantic projection between query text and item text, modality alignment between query text and item image, and asymmetric alignment between query text and the fused item—combined in a weighted loss $\alpha\mathcal{L}_t + \beta\mathcal{L}_i + \gamma\mathcal{L}_m$. In fine-tuning, a learned prediction header $P$, supervised on a fashion dataset, decides per query whether to use the multimodal item embedding ($P=1$) or the text-only embedding ($P=0$), with loss $P\mathcal{L}'_m + (1-P)\mathcal{L}'_t$. These pieces are what carry the reported recall gain.

What would settle it

Re-train SMAR and DPSR on the first 50 days of the released click logs, freeze all hyperparameters before touching the final split, and evaluate on days 51-60; if the recall@50 advantage of SMAR over DPSR disappears or reverses, the reported gain is an artifact of the split rather than a general property of the method.

Watch

Extended reading notes

Core claim

SMAR's central claim is that the asymmetry itself—text-only query, text-plus-image item—can be turned into an advantage rather than a nuisance, provided the model separates the contributions of each modality and lets the query decide when the image should matter. Concretely, the paper reports that the full model improves recall@50 from 0.641 (DPSR) to 0.690 on the overall test set, and that the gain is not from any single component: removing the modality alignment task, the semantic projection task, or the fusion/alignment tasks each lowers recall, as does removing cross-attention from the item multimodal tower. The paper also reports online A/B improvements of +0.285% GMV and +0.174% UCVR overall, and +1.112% and +0.437% in fashion categories, and it releases the industrial dataset for reproduction.

Load-bearing premise

The reported gains assume that the loss weights for text, image, and combined tasks, and the fashion-supervised rule for when to use the image, generalize to new queries; the paper reports no held-out validation protocol or significance tests for these choices.

Editorial extensions

If this is right

  • Item images can be added to a retrieval tower without requiring the query side to carry images, so existing text-only query pipelines can be upgraded in place.
  • The multi-task pretraining objectives each contribute; dropping the modality-alignment task, the semantic-projection task, or the fusion-alignment task reduces recall, so the gain is not attributable to one loss.
  • Cross-attention fusion is doing real work: removing it from the item multimodal tower lowers recall, indicating the image signal enters through text-conditioned attention rather than simple concatenation.
  • Adaptive gating matters for business metrics: online, the gains are concentrated in fashion categories, consistent with the idea that image information should be used selectively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same cross-attention fusion plus query-side gating recipe could transfer to other asymmetric retrieval settings, such as voice queries against product pages or text queries against video thumbnails.
  • The offline table shows the largest relative recall gain on the not-fashion split (0.575 versus 0.486), while the online A/B gain is largest in fashion; reconciling this difference would clarify where the image signal actually matters in live traffic.
  • Because synthetic queries outnumber real queries by more than 100 to 1, an ablation that trains without the synthetic augmentation would reveal how much of the gain comes from fusion and how much from the augmented training distribution itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes SMAR, a two-stage retrieval model for the modality-asymmetric setting in e-commerce search, where queries are text-only and items have both text and image modalities. Stage 1 is a multi-task pre-training objective combining a text semantic projection loss, an image alignment loss, and a multimodal alignment loss. Stage 2 is a fine-tuning model with an adaptive prediction header P that decides, per query, whether to use the multimodal item embedding or the text-only item embedding. The authors report offline results on an industrial dataset showing that SMAR improves R@50, P@50, and F1@50 over the DPSR baseline, and online A/B test results claiming GMV and UCVR gains. They also release the dataset publicly.

Significance. If the offline results hold up under scrutiny, this is a useful empirical contribution to multimodal retrieval in e-commerce, and the public release of an industrial dataset is a concrete positive for reproducibility and future research. The two-stage design and the adaptive per-query gating of image information are sensible and go beyond simple concatenation of modalities. However, the online A/B test, which is cited in the abstract and conclusion as evidence of business impact, is confounded because the test group receives 1000 additional candidates relative to the control. The offline metrics are independent of that confound and are the strongest evidence, but they currently lack statistical validation. The paper also leaves several implementation details underspecified, including the loss weights and the supervision of the prediction header.

major comments (4)
  1. [Section 3.4, Table 3] The online A/B test changes two variables simultaneously: the retrieval model (SMAR vs. the baseline) and the number of candidates passed to downstream ranking. The text explicitly says 'our model SMAR retrieves another 1000 candidates in addition to the results in baseline group,' meaning the control group receives no extra candidates. Consequently, the reported GMV +0.285% and UCVR +0.174% improvements, and the larger fashion-category lifts, could be produced by the larger candidate pool alone, regardless of whether SMAR's multimodal embeddings are semantically better. This confound invalidates the attribution of the online gains to SMAR's retrieval quality. The abstract and conclusion cite these online results as evidence of real-world success, so the claim cannot stand as written. The authors should run a controlled A/B experiment with equal candidate counts in both groups, or reframe the online section as a test of candidate expansion and avoid claiming that the gains demonstrate the value of multimodal fusion.
  2. [Section 3.2, Table 1] The text states that SMAR 'outperforms baseline models significantly,' but no statistical support is provided: there are no error bars, no standard deviations, no confidence intervals, and no significance tests, nor is the number of evaluation runs stated. Several of the reported improvements are small in absolute terms (for example, overall R@50 of 0.690 for SMAR versus 0.681 for SMAR-nm; fashion R@50 of 0.770 versus 0.749 for DPSR). Without variance information or significance testing, the word 'significantly' is not substantiated. Please report multiple runs or bootstrap intervals and add appropriate significance tests for the main comparisons in Table 1.
  3. [Section 3.3, Fig. 3] The sensitivity analysis in Fig. 3 varies one of alpha, beta, gamma at a time while fixing the others to 1, but the paper never reports the concrete values of alpha, beta, and gamma used in the final model that produces Table 1. Since Eq. (13) defines the loss and these weights directly determine the training objective, the omission is a reproducibility gap. Please state the final weight values and describe how they were selected (for example, on a validation split).
  4. [Section 2.4.1 and Eq. (14)] The prediction header P is central to the claimed adaptive behavior, but its supervision is underspecified. The text only says it is 'supervised by a fashion dataset,' without defining the label space, the loss function, the training procedure, or how P's output is combined with the click-through loss in Eq. (14). Additionally, the assumption that a binary fashion/not-fashion split is a sufficient proxy for whether images are useful for a query is not validated. Please provide complete details of P's training and, ideally, an analysis or ablation demonstrating that P's decisions correlate with actual image usefulness for retrieval.
minor comments (7)
  1. [Abstract] There are several typos: 'infomration' should be 'information' and 'Scenorio' should be 'Scenario.'
  2. [Section 2.2, Eq. (8)] The notation in Eq. (8) is terse: H_m = transformer_m(H_t, H_i, H_i) means Q=H_t and K=V=H_i, but this should be stated explicitly for readability.
  3. [Figure 2] Figure 2 is extremely dense and the text labels are small. Please provide a higher-resolution figure or a labeled breakdown so that the four towers and the flow of losses are readable.
  4. [Section 3.1] The training setup says 'batch size 128 per gpu' with '2 Tesla A100 GPU cards.' Please clarify whether the effective batch size is 256 and how batch negative sampling is performed across GPUs.
  5. [Table 1] The table caption contains a typo: 'Comparision' should be 'Comparison.'
  6. [Section 3.4] The paper cites [17] for the A/B infrastructure, but the key issue is the experimental design. Please clarify the duration of the experiment and the traffic split (15% is mentioned, but the split between test and control within that 15% is not stated).
  7. [References] The GitHub link in the footnote should be verified; if the dataset is truly open-sourced, the repository should be listed with a stable identifier and an explicit license.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SMAR's central claim is an empirical retrieval-accuracy comparison on held-out industrial data, and the cited prior work supplies standard training components rather than the claimed result.

full rationale

The paper does not derive SMAR's superiority from its own definitions. Its central evidence is Table 1 (R@50/P@50/F1@50) on an industrial dataset split into overall/fashion/not-fashion, where SMAR is compared against a deployed baseline DPSR and several ablations. Those numbers are empirical outcomes of training and inference, not algebraic consequences of the loss definitions. The pretraining loss (Eq. 13) and fine-tuning loss (Eq. 14) are standard weighted combinations of cross-entropy objectives; the adaptive header P is supervised with a separate fashion dataset, so the final model is not by construction equal to the supervision. Self-citations such as [14] (synthetic-query sampling), [23] (DPSR baseline and negative sampling), and [8,20,22,23] (click-log training) are used as engineering ingredients or baselines, not as proof of the headline gain; none of them asserts the SMAR result. The main caveats are experimental-design weaknesses rather than circularity: the hyperparameter sweep in Fig. 3 reports test-set Recall without a validation split, and Section 3.4's A/B test gives the SMAR group 1000 extra candidates ('our model SMAR retrieves another 1000 candidates in addition to the results in baseline group'), confounding model quality with candidate coverage. These affect causal attribution of the online lifts, but they do not make the offline comparison an instance of a claim reducing to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The model introduces no new theoretical entities. The free parameters are standard loss weights, and the axioms are common assumptions in industrial retrieval. The most fragile choice is the fashion-supervised gating head, which relies on an unspecified dataset.

free parameters (3)
  • alpha (text loss weight) = not reported, swept from 0.1 to 20
    Controls contribution of query-item text alignment in the pretraining loss; tuned via validation sweeps in Fig. 3.
  • beta (image loss weight) = not reported, swept from 0.1 to 20
    Controls contribution of query-image alignment in the pretraining loss; tuned via validation sweeps in Fig. 3.
  • gamma (multimodal loss weight) = not reported, swept from 0.1 to 20
    Controls contribution of query-multimodal alignment in the pretraining loss; tuned via validation sweeps in Fig. 3.
assumptions (3)
  • domain assumption Click logs provide reliable implicit relevance labels for query-item pairs.
    Section 2.4.2 uses user click logs to train the fine-tuning model, assuming clicks indicate relevance.
  • domain assumption Transformer encoders can represent text and image features in a shared semantic space through cross-attention.
    The model architecture in Section 2.2 relies on transformers for all modalities and on cross-attention for fusion.
  • ad hoc to paper A binary fashion/not-fashion split is a sufficient proxy for whether images are useful for a query.
    Section 2.4.1 uses a 'fashion dataset' to supervise the prediction header P, but the dataset composition is not described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantic-enhanced Modality-asymmetric Retrieval for Online E-commerce Search." pith.science (2026). https://pith.science/paper/OIHWW72K

@misc{pith2026250620330,
  author       = {Pith},
  title        = {Pith review of: Semantic-enhanced Modality-asymmetric Retrieval for Online E-commerce Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OIHWW72K}},
  note         = {Machine review of arXiv:2506.20330}
}
read the original abstract

Semantic retrieval, which retrieves semantically matched items given a textual query, has been an essential component to enhance system effectiveness in e-commerce search. In this paper, we study the multimodal retrieval problem, where the visual information (e.g, image) of item is leveraged as supplementary of textual information to enrich item representation and further improve retrieval performance. Though learning from cross-modality data has been studied extensively in tasks such as visual question answering or media summarization, multimodal retrieval remains a non-trivial and unsolved problem especially in the asymmetric scenario where the query is unimodal while the item is multimodal. In this paper, we propose a novel model named SMAR, which stands for Semantic-enhanced Modality-Asymmetric Retrieval, to tackle the problem of modality fusion and alignment in this kind of asymmetric scenario. Extensive experimental results on an industrial dataset show that the proposed model outperforms baseline models significantly in retrieval accuracy. We have open sourced our industrial dataset for the sake of reproducibility and future research works.

Figures

Figures reproduced from arXiv: 2506.20330 by the authors.

Figure 1
Figure 1. Illustration of the scenarios in which image im [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed model SMAR. Now we employ cross entropy loss over the training dataset as 𝑝 = 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 ({𝑓 (𝑞𝑖 , 𝑠𝑖) | 𝑠𝑖 ∈ {𝑠 + 𝑖 } ∪ 𝑁𝑖 }), (11) L (𝐷) = ∑︁ (𝑞𝑖 ,𝑠+ 𝑖 ,𝑁𝑖 ) ∈𝐷 ∑︁ 𝑠𝑗 ∈ {𝑠 + 𝑗 }∪𝑁𝑗 𝑟(𝑞𝑗 , 𝑠𝑗) 𝑙𝑜𝑔(𝑝𝑗). (12) For the three tasks in 2.3.1, we apply each scoring function in Formula 11 and denote the loss as L𝑡 (𝐷), L𝑖 (𝐷) and L𝑚 (𝐷). Con￾sidering the different contributions of text and image, we… view at source ↗
Figure 3
Figure 3. Effect of the weights 𝛼, 𝛽, 𝛾 for loss functions on overall dataset, fashion dataset and not-fashion dataset. For each line in the figures, we adjust one parameter(for example 𝛼) and set another two to 1 (for example 𝛽 = 1 and 𝛾 = 1) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 21 canonical work pages

  1. [1]

    Long Chen, Yuhang Zheng, and Jun Xiao. 2022. Rethinking data augmentation for robust visual question answering. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI. Springer, 95–112

    work page 2022

  2. [2]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    arXiv 2018

  3. [3]

    Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang. 2020. Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2251–2260

    work page 2020

  4. [4]

    MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga

  5. [5]

    Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17980–17989

    work page 2022

  6. [6]

    Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management. 2333–2338

    work page 2013

  7. [7]

    Xingyu Jiang, Jiayi Ma, Guobao Xiao, Zhenfeng Shao, and Xiaojie Guo. 2021. A review of multimodal image matching: Methods and applications. Information Fusion 73 (2021), 22–71

    work page 2021

  8. [8]

    Yunjiang Jiang, Han Zhang, Yiming Qiu, Yun Xiao, Bo Long, and Wen-Yun Yang

    work page

Show all 29 references
  1. [9]

    Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. 2021. Embedding-based Product Retrieval in Taobao Search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 3181–3189

    2021

  2. [10]

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al . 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August...

    2020

  3. [11]

    Junyang Lin, An Yang, Yichang Zhang, Jie Liu, Jingren Zhou, and Hongxia Yang

  4. [12]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  5. [13]

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretrain- ing task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems 32 (2019)

    2019

  6. [14]

    Yiming Qiu, Chenyu Zhao, Han Zhang, Jingwei Zhuo, Tianhao Li, Xiaowei Zhang, Songlin Wang, Sulong Xu, Bo Long, and Wen-Yun Yang. 2022. Pre-training Tasks for User Intent Detection and Embedding Retrieval in E-commerce Search. arXiv preprint arXiv:2208.06150 (2022)

    2022 arXiv

  7. [15]

    Parikshit Sondhi, Mohit Sharma, Pranam Kolari, and ChengXiang Zhai. 2018. A Taxonomy of Queries for E-commerce Search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 1245– 1248

    2018

  8. [16]

    Daria Sorokina and Erick Cantu-Paz. 2016. Amazon search: The joy of ranking products. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 459–460

    2016

  9. [17]

    Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010. Over- lapping experiment infrastructure: More, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. 17–26

    2010

  10. [18]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  11. [19]

    Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017. Visual question answering: A survey of methods and datasets. Computer Vision and Image Understanding 163 (2017), 21–40

    2017

  12. [20]

    Xinlin Xia, Shang Wang, Han Zhang, Songlin Wang, Sulong Xu, Yun Xiao, Bo Long, and Wen-Yun Yang. 2021. SearchGCN: Powering Embedding Retrieval by Graph Convolution Networks for E-Commerce Search. In Proceedings of the 44th International ACM SIGIR Conference on Research and Dev...

    2021

  13. [21]

    Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang

  14. [22]

    Han Zhang, Hongwei Shen, Yiming Qiu, Yunjiang Jiang, Songlin Wang, Sulong Xu, Yun Xiao, Bo Long, and Wen-Yun Yang. 2021. Joint Learning of Deep Retrieval Model and Product Quantization based Embedding Index. In Proceedings of the 44th International ACM SIGIR Conference on Rese...

    2021

  15. [23]

    Han Zhang, Songlin Wang, Kang Zhang, Zhiling Tang, Yunjiang Jiang, Yun Xiao, Weipeng Yan, and Wen-Yun Yang. 2020. Towards personalized and semantic retrieval: An end-to-end solution for E-commerce search via embedding learning. In Proceedings of the 43rd International ACM SIGI...

    2020

  16. [24]

    Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai

  17. [2018]

    In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

    Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1079–1088. A Presenter Profiles ZhiGong Zhou is a researcher in the Department of Search and Recommendation at JD.com. H...

  18. [2019]

    ACM Computing Surveys (CsUR) 51, 6 (2019), 1–36

    A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR) 51, 6 (2019), 1–36

    2019

  19. [2020]

    arXiv preprint arXiv:2003.13198 (2020)

    Interbert: Vision-and-language interaction for multi-modal pretraining. arXiv preprint arXiv:2003.13198 (2020)

    2020 arXiv

  20. [2021]

    In Proceedings of the AAAI conference on artificial intelligence, Vol

    Ernie-vil: Knowledge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 3208–3216

  21. [2022]

    arXiv preprint arXiv:2203.05082 (2022)

    Givens Coordinate Descent Methods for Rotation Matrix Learning in Trainable Embedding Indexes. arXiv preprint arXiv:2203.05082 (2022)

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.