REVIEW 4 major objections 4 minor 23 references
Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rank
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A simple two-tower deep learning ranker beat OTTO's production LambdaMART model in an 8-week A/B test.
desk verdict A rare, useful online A/B validation of a simple DNN against a production GBDT ranker, but the central claim rests on an under-reported experiment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Two-Tower architecture: a backbone network encodes product features into an item embedding, a separate linear layer encodes context features, and relevance scores are computed as dot products of the two embeddings. Because item embeddings can be precomputed, the model remains cheap at inference time. Paired with this is a normalized Softmax cross-entropy loss that converts click and order labels into normalized probability distributions and combines the two losses with weight alpha = 0.5. This combination, rather than the more complex Cross-Encoder or Transformer variants, produced the winning offline and online results.
What would settle it
Re-run the comparison with user-level randomization, logging of concurrent product launches, and a novelty-control period; if the click and revenue uplifts vanish or reverse once those confounds are removed, the paper's central claim fails.
Extended reading notes
Core claim
In the paper's own terms, the central discovery is that a simple Two-Tower DNN with Softmax CE loss outperforms OTTO's production-grade LGBM LambdaMART on live traffic. Offline, the selected model improved NDCG for clicks by 4.32% and Average Item Value by 2.28% over baseline with no change in NDCG for orders; online, it delivered a statistically significant 1.86% uplift in total clicks (p < 0.0001) and a 0.56% uplift in revenue (p < 0.01) over eight weeks, with units sold stable. The authors interpret this as evidence that deep learning approaches, when properly tuned and evaluated for production, can serve as a viable alternative to GBDT-based models in industrial ranking systems.
Load-bearing premise
The A/B test's measured click and revenue uplifts are treated as caused by the DNN, which requires that the 8-week test compared like with like: the same traffic, the same users, no other system changes, and no novelty effect.
Editorial extensions
If this is right
- Practitioners can adopt a simple DNN ranker and expect engagement and revenue gains over a tuned LambdaMART baseline on live e-commerce traffic.
- The winning model's inference cost is low because item embeddings are precomputed, making the architecture practical in a large-scale retrieval-to-ranking pipeline.
- Offline NDCG at cutoff 15 and Average Item Value tracked the online outcome, supporting the use of these offline metrics as proxies for live performance.
- More complex DNN architectures did not beat the simple Two-Tower model, so added architectural complexity was not required for the observed gains.
- The authors frame the result as evidence that DNNs are a viable alternative in this setting, not as a claim that DNNs dominate GBDTs universally.
Reading between the lines
- The result is likely sensitive to data scale and label sparsity: OTTO's large interaction logs and click-heavy labels may be exactly the regime where an embedding-based model's capacity pays off, so smaller catalogs might not see the same gap.
- The A/B test report lacks details on randomization unit and concurrent system changes, so a cautious reader should treat the point estimates as provisional until replication with user-level randomization.
- A natural next experiment is to tune alpha upward to weight orders more heavily, since the paper's NDCG-order results were noticeably weaker than its click results.
- If the pattern replicates across other marketplaces, the practical default for e-commerce search ranking could shift from GBDT ensembles toward simple embedding-based models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares deep learning ranking architectures (Two-Tower, Cross-Encoder, Transformer) trained with RankNet and Softmax Cross-Entropy losses against OTTO's production-grade LightGBM LambdaMART baseline. Offline experiments on a proprietary 43M-sample training set report NDCG and AIV improvements for DNNs, with the Two-Tower model trained with Softmax CE selected as the best candidate. An 8-week online A/B test reports statistically significant uplifts of +1.86% in total clicks (p < 0.0001) and +0.56% in revenue (p < 0.01) for the Two-Tower model, with units sold described as stable. The paper concludes that a simple DNN can be a viable alternative to GBDT-based rankers in industrial e-commerce search.
Significance. If the online A/B result is reliable, this paper provides valuable, rare industry-scale evidence that a relatively simple deep learning ranker can outperform a mature LambdaMART production system on live e-commerce traffic. The manuscript's strengths include the use of a large proprietary dataset, systematic comparison of three architectures and two loss functions, honest reporting of offline underperformance on order NDCG for most DNNs, and an actual online validation rather than only offline metrics. These are useful contributions to the practitioner-oriented LTR literature. However, the headline claim rests almost entirely on the online A/B test described in Section 6.2, and the reporting of that experiment is currently too incomplete to verify the significance or validity of the results.
major comments (4)
- [§6.2, Online Experiments] The A/B test reporting is insufficient to support the central causal claim. The manuscript does not specify the randomization unit (user, session, or request), the traffic split between variants, the number of observations, or the variance of the metrics; it also does not describe overlap control, pre-period balance, or whether any other system changes occurred during the 8-week window. These details are essential because the reported t-test may be invalid if the unit of analysis is a session (repeated sessions from the same user violate independence) or if the totals are not normalized by exposure. Please provide the full experimental design, sample sizes, and numeric confidence intervals, or justify why the t-test is appropriate.
- [§6.2, Online Experiments] The claim that 'units sold remained stable' is asserted without any statistical support. Parity in units sold is one of the three conclusions in the abstract, but no test statistic, p-value, or confidence interval is reported for this metric. The authors should either provide an equivalence test or a confidence interval that demonstrates parity, or soften the claim to state that no significant difference was observed.
- [§6.2, Figure 1] Figure 1, which the text says displays the 95% t-test confidence intervals for the online results, is missing from the manuscript. Since this figure is the only numerical source of uncertainty for the headline online metrics, it must be included, and the confidence intervals should also be stated in the text so that the reported p-values can be checked.
- [§6.1, Offline Experiments, Table 1] The offline improvements in NDCG and AIV are reported without any measure of uncertainty, such as confidence intervals, standard errors, or significance tests. Given that the model selection for the online test was made from these offline results, the lack of uncertainty estimates makes it difficult to assess whether the observed differences are robust or merely noise. Please add error bars or statistical tests for the offline metrics.
minor comments (4)
- [§4, Feature Embeddings] The notation for categorical embeddings uses f^cat_{\cdot,j} without making explicit whether the embedding is applied per product and how multiple categorical features are combined; a small clarification would improve reproducibility.
- [§5.2, Losses] The RankNet loss is described as a modification of \tilde{L}_{RN}, but the original \tilde{L}_{RN} is not explicitly defined in the text; please provide the exact formula or a reference to the equation in the cited work.
- [§6, Experimental Setup] The description of the dataset gives only sample counts (43M training, 700k test) and does not mention the number of users, products, or requests, nor the temporal gap between training and test periods; adding these details would help readers gauge the scale and the strength of the temporal split.
- [§1, Introduction] The phrase 'a single-layer feed-forward neural network with a hidden size of 32 matches the performance of their GBDT baseline' cites reference [8], but the connection to the baseline strength is discussed only briefly; a sentence explaining why this suggests a weak baseline would strengthen the motivation.
Circularity Check
No circularity: the paper reports empirical offline and online comparisons; the online A/B test is a genuine out-of-sample benchmark and no fitted quantity is relabeled as a prediction.
full rationale
This paper does not claim a derivational prediction; its central results are empirical comparisons of DNN architectures against a production LambdaMART baseline. The offline evaluation uses a temporal train-test split (Section 6), and the best offline model is then tested in an 8-week online A/B test (Section 6.2). This is a genuine out-of-sample benchmark against a production system, not a quantity derived from the same data that defines the result. The model selection is based on offline NDCG/AIV metrics, and the reported online uplifts are separate empirical measurements. No equation in the paper reduces a predicted quantity to a fitted input, and no parameter is fitted to the online outcome and then called a prediction. The only self-citation is reference [20], cited for using a temporal train-test split; that is a methodological citation and is not load-bearing for the paper's empirical claim. The omission of randomization unit, traffic allocation, confidence intervals, and the missing Figure 1 are reporting/transparency concerns about the validity of the A/B test, but they are not circularity: the reported result is still an independent out-of-sample measurement rather than a constructed equivalence. Therefore the correct circularity score is 0.
Assumptions & free parameters
free parameters (6)
- alpha (click/order loss weight) =
0.5
- hidden size h and backbone depth k =
h=1024, k=3
- embedding dimensions d_cat, d_text =
128, 512
- dropout rates =
0.0 (TT), 0.3 (CR), 0.5 (TR)
- learning rates =
0.001 (TT/CR), 0.0001 (TR)
- Production LGBM LambdaMART baseline hyperparameters =
lr=0.1, max_depth=12, num_leaves=25, 400 trees
assumptions (3)
- standard math The t-test provides valid significance levels for the A/B test uplifts.
- domain assumption NDCG at cutoff 15 and AIV are valid offline proxies for online user behavior.
- domain assumption The production LGBM model is a strong, well-tuned representative of GBDT-based LTR systems.
Cite this review
Pith. "Pith review of Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rank." pith.science (2026). https://pith.science/paper/QQCMJDAN
@misc{pith2026250720753,
author = {Pith},
title = {Pith review of: Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rank},
year = {2026},
howpublished = {\url{https://pith.science/paper/QQCMJDAN}},
note = {Machine review of arXiv:2507.20753}
}
read the original abstract
In e-commerce recommender and search systems, tree-based models, such as LambdaMART, have set a strong baseline for Learning-to-Rank (LTR) tasks. Despite their effectiveness and widespread adoption in industry, the debate continues whether deep neural networks (DNNs) can outperform traditional tree-based models in this domain. To contribute to this discussion, we systematically benchmark DNNs against our production-grade LambdaMART model. We evaluate multiple DNN architectures and loss functions on a proprietary dataset from OTTO and validate our findings through an 8-week online A/B test. The results show that a simple DNN architecture outperforms a strong tree-based baseline in terms of total clicks and revenue, while achieving parity in total units sold.
Figures
Reference graph
Works this paper leans on
-
[1]
Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H. Chi. 2018. Latent Cross: Making Use of Context in Recurrent Recommender Systems. In WSDM 2018: The Eleventh ACM International Conference on Web Search and Data Mining
work page 2018
-
[2]
Sebastian Bruch, Xuanhui Wang, Mike Bendersky, and Marc Najork. 2019. An Analysis of the Softmax Cross Entropy Loss for Learning-to-Rank with Binary Relevance. In Proceedings of the 2019 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2019) . 75–78
work page 2019
-
[3]
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. InProceedings of the 22nd international conference on Machine learning - ICML ’05 . ACM Press, Bonn, Germany, 89–96. doi:10.1145/1102351.1102363
arXiv 2005
-
[4]
Maarten Buyl, Paul Missault, and Pierre-Antoine Sondag. 2023. RankFormer: List- wise learning-to-rank using listwide labels. (2023). https://www.amazon.science/ publications/rankformer-listwise-learning-to-rank-using-listwide-labels
work page 2023
-
[5]
Olivier Chapelle and Yi Chang. 2011. Yahoo! Learning to Rank Challenge Overview. In Proceedings of the Learning to Rank Challenge (Proceedings of Ma- chine Learning Research, Vol. 14) , Olivier Chapelle, Yi Chang, and Tie-Yan Liu (Eds.). PMLR, Haifa, Israel, 1–24. https://proceedings.mlr.press/v14/chapelle11a. html
work page 2011
-
[6]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. New York, NY, USA
work page 2016
-
[7]
Philipp Hager, Romain Deffayet, Jean-Michel Renders, Onno Zoeter, and Maarten de Rijke. 2024. Unbiased Learning to Rank Meets Reality: Lessons from Baidu’s Large-Scale Search Dataset. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (Washington DC, USA) (SIGIR ’24). Association for Computin...
arXiv 2024
-
[8]
Malay Haldar, Mustafa Abdool, Prashant Ramanathan, Tao Xu, Shulin Yang, Huizhong Duan, Qing Zhang, Nick Barrow-Williams, Bradley C. Turnbull, Brendan M. Collins, and Thomas Legrand. 2019. Applying Deep Learning to Airbnb Search. In Proceedings of the 25th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD ...
arXiv 2019
Show all 23 references
-
[9]
Balázs Hidasi and Ádám Tibor Czapp. 2023. Widespread Flaws in Offline Evalu- ation of Recommender Systems. In Proceedings of the 17th ACM Conference on Recommender Systems (Singapore, Singapore) (RecSys ’23). Association for Com- puting Machinery, New York, NY, USA, 848–855. d...
2023
-
[10]
Ziniu Hu, Yang Wang, Qu Peng, and Hang Li. 2019. Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm. In The World Wide Web Confer- ence. ACM, San Francisco CA USA, 2830–2836. doi:10.1145/3308558.3313447
2019
-
[11]
Shubhra Kanti Karmaker Santu, Parikshit Sondhi, and ChengXiang Zhai. 2017. On Application of Learning to Rank for E-Commerce Search. InProceedings of the 40th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval (Shinjuku, Tokyo, Japan)(SIG...
2017
-
[12]
Haitao Li, Jia Chen, Weihang Su, Qingyao Ai, and Yiqun Liu. 2023. Towards Better Web Search Performance: Pre-training, Fine-tuning and Learning to Rank. arXiv:2303.04710 [cs.IR] https://arxiv.org/abs/2303.04710
2023 arXiv
-
[13]
Alessandro Magnani, Feng Liu, Suthee Chaidaroon, Sachin Yadav, Praveen Reddy Suram, Ajit Puthenputhussery, Sijie Chen, Min Xie, Anirudh Kashi, Tony Lee, and Ciya Liao. 2022. Semantic Retrieval at Walmart. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery ...
2022
-
[14]
Przemyslaw Pobrotyn, Tomasz Bartczak, Mikolaj Synowiec, Radoslaw Bialo- brzeski, and Jaroslaw Bojar. 2020. Context-Aware Learning to Rank with Self- Attention. CoRR abs/2005.10084 (2020). arXiv:2005.10084 https://arxiv.org/abs/ 2005.10084
2020 arXiv
-
[15]
Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. CoRR abs/1306.2597 (2013). http://arxiv.org/abs/1306.2597
2013 arXiv
-
[16]
Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Mike Bendersky, and Marc Najork. 2021. Are Neural Rankers still Out- performed by Gradient Boosted Decision Trees?. In International Conference on Learning Representations (ICLR)
2021
-
[17]
Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. 2020. When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (Virtual Event, Ireland) ...
2020 doi
-
[18]
Gomez, Łukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, ...
2017
-
[19]
Xiaojie Wang, Ruoyuan Gao, Anoop Jain, Graham Edge, and Sachin Ahuja. 2023. How well do offline metrics predict online performance of product ranking models? (2023). https://www.amazon.science/publications/how-well-do-offline- metrics-predict-online-performance-of-product-rank...
2023
-
[20]
Timo Wilm, Philipp Normann, Sophie Baumeister, and Paul-Vincent Kobow
-
[21]
Tao Yang, Chen Luo, Hanqing Lu, Parth Gupta, Bing Yin, and Qingyao Ai. 2022. Can Clicks Be Both Labels and Features? Unbiased Behavior Feature Collection and Uncertainty-aware Learning to Rank. In Proceedings of the 45th International ACM SIGIR Conference on Research and Devel...
2022
-
[22]
Lixin Zou, Haitao Mao andXiaokai Chu, Jiliang Tang, Wenwen Ye, Shuaiqiang Wang, and Dawei Yin. 2022. A Large Scale Search Dataset for Unbiased Learning to Rank. In NeurIPS 2022
2022
-
[2023]
InProceedings of the 17th ACM Conference on Recommender Systems (Singapore, Singapore) (RecSys ’23)
Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions. InProceedings of the 17th ACM Conference on Recommender Systems (Singapore, Singapore) (RecSys ’23). Association for Computing Machinery, New York, NY, USA, 1023–1026. doi:1...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.