Pith. sign in

REVIEW 5 major objections 6 minor 22 references

Semantic Ads Retrieval at Walmart eCommerce with Language Models Progressively Trained on Multiple Knowledge Domains

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-tower BERT model with category pretraining and human-in-the-loop progressive training improves ads-retrieval relevance by up to 16 percent and ad revenue by 5.16 percent in production on Walmart.com.

desk verdict A real deployed system with honest engineering claims, but the offline numbers have a possible train/eval leakage problem that the paper does not address. read the letter →

arxiv 2502.09089 v1 pith:OOHUHERJ submitted 2025-02-13 cs.IR

Zhaodong Wang , Weizhi Du , Md Omar Faruk Rokon , Pooshpendu Adhikary , Yanbing Xue , Jiaxuan Xu , Jianghong Zhou , Kuang-chih Lee
show 1 more author
Musen Wen
This is my paper · ORCID
classification cs.IR
keywords sponsoredsearchadsretrievalBERTtwo-towerSiamesenetworkhuman-in-the-loopprogressivetraininghardnegativemininge-commerce
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sponsored ads on Walmart.com are matched to search queries by a retrieval model that must pick candidate products for later ranking. This paper claims that a two-stage training pipeline produces a retrieval model that improves query-ad matching over the company's previous DSSM-based model. The pipeline first teaches a BERT-style language model Walmart's product categories, then trains a two-tower Siamese encoder on several datasets at once, with a human-in-the-loop rule that shifts training weight toward domains where the model is still weak. Offline evaluation shows NDCG gains of up to 16.10 percent and a reduction in the irrelevant-ads rate by about 4 percent; online A/B tests show ad revenue up 5.16 percent. The paper's point is that the specific combination of category pretraining, hard-negative mining, and progressive fusion training is what produces these gains.

What carries the argument

The load-bearing mechanism is the two-stage progressive training architecture. Stage one runs multi-class classification of queries and items into Walmart 'Department' and 'Product Type' categories with DistilBERT and cross-entropy loss. Stage two adds a pooling layer to produce 384-dimensional embeddings and trains a two-tower Siamese network on multiple datasets through a dynamic fusion sampling rule: each domain's sampling weight is $w_i = 1/(1+\exp(10x_i - 5))$, where $x_i$ is the normalized discounted cumulative gain (nDCG) from human-labeled data, so well-handled domains are down-weighted and weak domains get more training. The encoder is trained with mean-squared-error loss on cosine-similarity labels plus a triplet loss with in-batch hard negatives, where hard negatives are items often displayed for a query but seldom clicked.

What would settle it

Train the same pipeline with the 12 million ad-log pseudo-labels replaced by human relevance labels on a matched subsample, and run the offline approximate-nearest-neighbor retrieval evaluation: if the NDCG@5 improvement over DSSM remains around 16 percent, then click-count pseudo-labeling is not what produces the reported relevance gains.

Watch

Extended reading notes

Core claim

The central claim is that the proposed pipeline — DistilBERT pretrained on Walmart department and product-type labels, then trained in a two-tower Siamese network with cosine-similarity and triplet losses over natural-language-inference, search-engine, human-labeled, and ad-click datasets — improves both relevance and business metrics for sponsored-search retrieval on Walmart.com relative to the DSSM baseline. The authors report that the model outperforms DSSM by up to 16.10 percent in NDCG@5 offline, reduces the irrelevant-ads rate by up to 4.07 percent, and in online A/B tests increases search-page ad revenue by 5.16 percent, CTR by 7.2 percent, and CPMV by 4.9 percent. They also report that removing hard negatives degrades NDCG by 3.6 percent, evidence that the hard-negative labeling scheme is load-bearing.

Load-bearing premise

The ad-log training data are labeled by treating an item as relevant if it attracts many clicks in a designated period, so the entire 12-million-pair training signal assumes clicks are a truthful proxy for topical relevance; position bias, brand effects, and price sensitivity could distort that signal.

Editorial extensions

If this is right

  • The two-tower BERT model was deployed as the production ads-retrieval model on Walmart.com in June 2023, so the reported gains reflect live-traffic behavior, not just offline simulations.
  • Retrieval-stage relevance gains partially propagate through the funnel: online NDCG on final ad placements improved by roughly 9-12 percent, smaller than the offline retrieval gains because ranking and re-ranking follow retrieval.
  • Hard-negative mining from search logs (items displayed but rarely clicked) contributes a measurable 3.6 percent NDCG gain, so retrieval pipelines should construct such negatives explicitly.
  • The dynamic fusion weighting rule offers a reusable way to balance multiple training domains without manual curriculum design, by down-weighting domains where human-labeled nDCG is already high.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The click-count pseudo-labeling is a candidate for position-bias correction; the paper provides no analysis of display position, brand, or price effects, so a testable extension is to re-label the 12 million pairs with position-adjusted click weights and compare NDCG.
  • The sigmoid weighting equation $w_i = 1/(1+\exp(10x_i - 5))$ is a specific parameterization; the steepness 10 and offset 5 are not experimentally justified, so the same framework might perform differently with other coefficients.
  • The category-pretraining-plus-two-tower recipe likely transfers to other large e-commerce marketplaces with hierarchical taxonomies, but the paper only demonstrates it on Walmart data, so a cross-platform replication would clarify how much of the gain is Walmart-specific.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper describes an end-to-end sponsored-product retrieval system for Walmart.com built on a DistilBERT-based two-tower Siamese network. The training pipeline has three main components: (i) language-model pretraining on Walmart product-category labels, (ii) progressive fusion training over multiple knowledge domains (SNLI/MultiNLI, Google site-restricted query-title pairs, human-evaluated search logs, and click-based ad-log pseudo-labels), and (iii) a human-in-the-loop weighting mechanism that adjusts per-domain sampling weights from nDCG scores. The authors report offline retrieval gains over a DSSM baseline (up to +16.10% NDCG@5, with IAR reductions) and online A/B improvements in NDCG, CTR, CPMV, and ad revenue, and state that the system was deployed in production in June 2023.

Significance. If the reported results are reliable, this is a valuable industrial case study: it combines public NLI data, Google site-restricted query-title pairs, human-labeled search logs, and click-based ad-log pseudo-labels into a two-tower embedding model that was deployed at scale, with consistent positive movements across relevance and business metrics. The hard-negative labeling method and the progressive fusion sampling mechanism are concrete, reusable ideas, and the deployment/monitoring narrative is a strength. However, the empirical core is currently under-supported: the offline evaluation may overlap with training data, no uncertainty quantification is reported for the offline numbers, and the online A/B testing lacks statistical detail. Because the central claims rest on these measurements, the paper needs substantial additional evidence before the results can be accepted at face value.

major comments (5)
  1. [§4.2 / §2.2 / Table 2] The central offline claim of +16.10% NDCG@5 rests on 2,000 queries sampled from Walmart traffic, while Section 2.2 describes training corpora drawn from the same ecosystem, including 8M human-evaluated search-log pairs and 20M Google site:walmart.com query-title pairs. The paper never states whether overlapping queries, query-item pairs, or near-duplicate titles were removed between training and this evaluation. Without a deduplication protocol or a temporal split, the reported gains can be inflated by the model memorizing labels from training pairs that reappear in the evaluation set. Please report overlap statistics and either exclude overlapping examples or use a split that guarantees disjoint queries.
  2. [§4.2 / Table 2] The offline results are single point estimates with no variance, confidence intervals, or significance tests. With 2,000 evaluation queries, the differences of +16.10% (NDCG@5), -4.07% (IAR@5), and the other rows in Table 2 may not be statistically reliable. Please provide bootstrap confidence intervals, error bars, and ideally the absolute NDCG/IAR values for both models, not only relative improvements.
  3. [§4.3 / Tables 3–5] The online A/B testing claims of +5.16% ad revenue and relevance improvements up to +11.87% are reported without sample sizes, experiment duration, confidence intervals, or guardrail metrics. The phrase 'large-scale' is not supported by the reported evidence. Please include the statistical details and specify whether the differences are significant at conventional levels.
  4. [§2.2, 'Data Labeling for Ad Logs'] The pseudo-labeling rule treats an item as relevant if it attracts a high number of clicks within a period, and hard negatives are items that are displayed often but rarely clicked. These heuristics confound relevance with position bias, brand awareness, and price attractiveness; the 'hard negative' definition in particular may label relevant but poorly placed items as negatives. Please validate the pseudo-labels against a human-labeled sample and report agreement or a bias analysis.
  5. [§2.1 / Eq. (1)] The human-in-the-loop weight adjustment uses nDCG scores computed from human-labeled data, and Section 4.2's offline evaluation also uses third-party human relevance labels. The paper does not state whether these two label sets are disjoint. If the same labels or queries are used both to set the domain sampling weights and to evaluate the final model, the evaluation is not independent. Please clarify the provenance of the two label sets and confirm that no evaluation label was used for model selection or weight tuning.
minor comments (6)
  1. [§4.1] In the first sentence of Section 4.1, 'matrices' should be 'metrics', and Eq. (6) should explicitly define the indicator function I(i).
  2. [§4.2] 'Ground-true relevance labels' should be 'ground-truth relevance labels'.
  3. [Table 5 caption] The caption says 'on search page' but the rows describe item page modules; the caption should be corrected.
  4. [Eq. (1) and Eq. (3)] The sigmoid constants 10 and 5 in Eq. (1) and the margin alpha in Eq. (3) are free parameters with no stated values; please report the chosen values and provide a brief sensitivity check.
  5. [Introduction / Reference [13]] The statement that the model in [13] 'did not significantly contribute to the improvement of the advertising system' is presented as observed fact without a citation; either cite the source or rephrase as the authors' own unpublished experience.
  6. [§4.4] The monitoring protocol is described verbally but no post-deployment numbers are given; either add a compact monitoring table or state explicitly that these numbers are omitted.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: reported offline gains are measured on a separately sampled 2,000-query set with third-party human labels, and online A/B metrics are external to the training objective.

full rationale

The paper's empirical claims are not derived from its own fitted parameters by construction. The offline relevance evaluation in Section 4.2 randomly samples 2,000 queries from Walmart traffic, retrieves 20 items per query via Faiss approximate nearest neighbor search, and scores them with third-party human labels, producing NDCG and IAR comparisons against a DSSM baseline. The online A/B results in Section 4.3 measure NDCG, CTR, CPMV, and ad revenue on live search and item-page placements. Neither evaluation is computed from the training labels by an equation in the paper. Equation (1) only adjusts domain sampling weights using nDCG from human-labeled data; it does not define the reported metrics or the evaluation set. The pseudo-labeling of ad logs and hard-negative construction are training-signal choices, not predictions that are then verified by the same labels. The only overlapping-author citation, [20], supports a general motivational statement about ad revenue and is not load-bearing for the model architecture or the empirical claims. The absence of an explicit deduplication statement between the 8M human-evaluated training pairs and the 2,000 evaluation queries is a data-hygiene and validity risk, but without evidence that the same query-item pairs were used for weighting and evaluation, it is not a circularity by construction. The pipeline is therefore self-contained against external human judgments and live business metrics, and no circular derivation chain is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The reported gains depend on several hand-chosen hyperparameters (triplet margin, sigmoid constants in Eq. 1, embedding dimension, pseudo-label click threshold) and on domain assumptions about the reliability of click signals, Google search relevance, third-party human annotations, and the validity of nDCG-weighted sampling. No sensitivity or ablation analysis is provided for any of these choices, so a reader cannot tell how robust the headline numbers are.

free parameters (5)
  • Triplet margin alpha = not disclosed
    Margin in the triplet loss (Eq. 3) controls the required distance between positive and negative pairs; no ablation or value is given.
  • Sigmoid constants 10 and 5 in Eq. (1) = 10 and 5
    The human-in-the-loop sampling weight is a sigmoid with fixed slope and center; the paper gives no derivation or sensitivity analysis for these constants.
  • Embedding dimension = 384
    The pooled sentence embedding dimension is set to 384; this is a free design choice that affects capacity and retrieval speed.
  • Pseudo-label click threshold = not disclosed
    The definition of 'high number of clicks within a designated period' for ad-log relevance labeling is left unspecified, making the pseudo-label generation non-reproducible.
  • Per-domain sampling weights w_i = not disclosed; adjusted per Eq. (1)
    The progressive fusion weights are updated based on nDCG scores of human-labeled data; they are tuned rather than derived.
assumptions (5)
  • domain assumption User clicks are a valid proxy for query-item relevance.
    Pseudo-labels for ad logs are generated from click counts (Section 2.2).
  • domain assumption Google's site:walmart.com search results reflect query-to-product relevance.
    20M query-item pairs from Google search are used as a training domain (Section 2.2, dataset 3).
  • domain assumption Human annotations on the third-party platform are ground truth.
    Offline evaluation uses third-party human labels with a three-level relevance scale (Section 4.2).
  • domain assumption The nDCG computed on human-labeled data is a reliable signal for adjusting sampling weights.
    Eq. (1) and Figure 2 use nDCG scores to re-balance domain sampling; this assumes the human-labeled set is representative and stable.
  • domain assumption The two-tower BERT embedding space with cosine similarity supports effective ANN retrieval.
    The whole retrieval pipeline relies on this standard but unverified-for-this-domain assumption (Section 2.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantic Ads Retrieval at Walmart eCommerce with Language Models Progressively Trained on Multiple Knowledge Domains." pith.science (2026). https://pith.science/paper/OOHUHERJ

@misc{pith2026250209089,
  author       = {Pith},
  title        = {Pith review of: Semantic Ads Retrieval at Walmart eCommerce with Language Models Progressively Trained on Multiple Knowledge Domains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOHUHERJ}},
  note         = {Machine review of arXiv:2502.09089}
}
read the original abstract

Sponsored search in e-commerce poses several unique and complex challenges. These challenges stem from factors such as the asymmetric language structure between search queries and product names, the inherent ambiguity in user search intent, and the vast volume of sparse and imbalanced search corpus data. The role of the retrieval component within a sponsored search system is pivotal, serving as the initial step that directly affects the subsequent ranking and bidding systems. In this paper, we present an end-to-end solution tailored to optimize the ads retrieval system on Walmart.com. Our approach is to pretrain the BERT-like classification model with product category information, enhancing the model's understanding of Walmart product semantics. Second, we design a two-tower Siamese Network structure for embedding structures to augment training efficiency. Third, we introduce a Human-in-the-loop Progressive Fusion Training method to ensure robust model performance. Our results demonstrate the effectiveness of this pipeline. It enhances the search relevance metric by up to 16% compared to a baseline DSSM-based model. Moreover, our large-scale online A/B testing demonstrates that our approach surpasses the ad revenue of the existing production model.

Figures

Figures reproduced from arXiv: 2502.09089 by the authors.

Figure 1
Figure 1. Two-stage progressive training architecture of retrieval embedding model. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Human-in-the-loop knowledge fusion sampling. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Department classification label of exampled item [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Pipeline architecture of generating Ads embedding [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Retrieval service architecture based on Vespa search [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Diagram of Faiss-based simulation relevance evaluation [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 10 canonical work pages

  1. [1]

    Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning

  2. [2]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. New York, NY, USA

    2016

  3. [3]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805

    arXiv 2018

  4. [4]

    Weizhi Du, Qichen Fu, and Zhengyu Huang. 2022. A Self-Supervised Deep Model for Focal Stacking. In CLEO: Applications and Technology . Optica Publishing Group, JTh3A–10

    work page 2022

  5. [5]

    Weizhi Du, Eunjeong Hyeon, Hanzhang Pei, Zhengyu Huang, YeonJoon Cheong, Siyuan Zheng, and Almantas Galvanauskas. 2021. Improved machine learn- ing algorithms for optimizing coherent pulse stacking amplification. In CLEO: QELS_Fundamental Science. Optica Publishing Group, JTh3A–1

    work page 2021

  6. [6]

    Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. ACM International Conference on Information and Knowledge Management (CIKM). https://www.microsoft.com/en- us/research/publication/learning-deep-structured-semantic-models-for-web- search-usin...

    work page 2013

  7. [7]

    Zhengyu Huang, Weizhi Du, and Theodore B Norris. 2022. Unsupervised Learn- ing Based Focal Stack Camera Depth Estimation. In CLEO: Applications and Technology. Optica Publishing Group, JW3A–5

    work page 2022

  8. [8]

    Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20, 4 (2002), 422–446

    2002

Show all 22 references
  1. [9]

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with GPUs. CoRR abs/1702.08734 (2017). arXiv:1702.08734 http://arxiv. org/abs/1702.08734 The Web Conference 2024, Singapore Zhaodong Wang and Weizhi Du, et al

    2017 arXiv

  2. [10]

    Uday Kamath, John Liu, and James Whitaker. 2019. Deep learning for NLP and speech recognition. Vol. 84. Springer

    2019

  3. [11]

    Peiyang Liu, Sen Wang, Xi Wang, Wei Ye, and Shikun Zhang. 2021. Quadruplet- BERT: An Efficient Model For Embedding-Based Large-Scale Retrieval. In Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag...

    2021 doi

  4. [12]

    Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval. CoRR abs/2002.06275 (2020). arXiv:2002.06275 https://arxiv.org/abs/2002.06275

    2020 arXiv

  5. [13]

    Alessandro Magnani, Feng Liu, Suthee Chaidaroon, Sachin Yadav, Praveen Reddy Suram, Ajit Puthenputhussery, Sijie Chen, Min Xie, Anirudh Kashi, Tony Lee, et al. 2022. Semantic retrieval at walmart. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data...

    2022

  6. [14]

    Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Frame- work: BM25 and Beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389. https://doi.org/10.1561/1500000019

    2009 doi

  7. [15]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Dis- tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108 (2019)

    2019 arXiv

  8. [16]

    J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007. Collaborative filtering recommender systems. The adaptive web: methods and strategies of web personalization (2007), 291–324

    2007

  9. [17]

    Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. 2018. Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba. CoRR abs/1803.02349 (2018). arXiv:1803.02349 http://arxiv.org/abs/ 1803.02349

    2018 arXiv

  10. [18]

    Brovman, and Sriganesh Madhvanath

    Tian Wang, Yuri M. Brovman, and Sriganesh Madhvanath. 2021. Personalized Embedding-based e-Commerce Recommendations at eBay. CoRR abs/2102.06156 (2021). arXiv:2102.06156 https://arxiv.org/abs/2102.06156

    2021 arXiv

  11. [19]

    Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 (2017)

    2017 arXiv

  12. [20]

    Yanbing Xue, Bo Liu, Weizhi Du, Jayanth Korlimarla, and Musen Men. 2023. Practical Lessons on Optimizing Sponsored Products in eCommerce. arXiv preprint arXiv:2304.09107 (2023)

    2023 arXiv

  13. [21]

    Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Adi- tee Ajit Kumthekar, Zhe Zhao, Li Wei, and Ed Chi (Eds.). 2019. Sampling-Bias- Corrected Neural Modeling for Large Corpus Item Recommendations

    2019

  14. [2015]

    arXiv preprint arXiv:1508.05326 (2015)

    A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 (2015)

    2015 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.