REVIEW 4 major objections 2 minor 1 references
Clicks Versus Conversion: Choosing a Recommender's Training Objective in E-Commerce
T0 review · 4 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper reports that, in the authors' e-commerce shops, training a product recommender on order-submit rate instead of click-through rate produces a gross-merchandise-value uplift more than five times larger, with no measurable loss in n
desk verdict The abstract reports a 5x GMV advantage for order-submit-trained recommenders in a live A/B test; I can't check the evidence because the supplied text is corrupt and appears to be a different arXiv paper. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the training label: click-through rate, add-to-cart rate, or order-submit rate, used as the target for ranking recommendations. OSR is the named target closest to the business metric GMV, because a purchase is the event that generates revenue. The mechanism is the online A/B test itself: models that are identical except for their training objective are served to comparable traffic, so any GMV difference is attributed to the objective. This isolates the training objective as the intervention.
What would settle it
Re-run the comparison in the same shops with CTR- and OSR-trained models that share architecture, features, training data, and serving logic, and inspect the confidence interval on the GMV difference. If the OSR uplift is not more than five times the CTR uplift, or the interval overlaps zero, the central claim would be refuted.
Extended reading notes
Core claim
On the paper's own terms: in the shops studied, replacing CTR with OSR as the training objective for the product recommender increases GMV uplift by more than a factor of five, while leaving new-product discovery intact. The comparison is made by an online A/B test in which models trained on CTR, add-to-cart rate (ACR), and OSR are evaluated on live traffic. The paper further finds that the relative importance of features shifts across objectives, meaning the choice of objective changes not only the ranking but the signals the model relies on.
Load-bearing premise
The claim stands or falls on the online A/B test being clean: the recommender variants differ only in their training objective, traffic is randomly split, and the observed GMV gap is statistically significant rather than noise or a seasonal artifact.
Editorial extensions
If this is right
- In the shops studied, switching the recommender's training objective from CTR to OSR multiplies the GMV uplift by more than five.
- The GMV gain is not bought by narrowing the catalog: new-product discovery is preserved under the OSR-trained model.
- Because feature importances differ by objective, feature selection and engineering for a recommender should be tuned to the chosen training label.
- Click data's practical advantages—abundance and ease of collection—are weighed against a direct business metric, and the comparison favors the business metric.
Reading between the lines
- If the fivefold gap reproduces in other shops and catalogs, it suggests click-optimized recommenders are systematically leaving revenue on the table; the paper itself only claims the result for its own shops.
- A natural testable extension is a hybrid objective that keeps CTR for exploration and OSR for ranking, to see whether the two signals can be combined rather than chosen between.
- The preserved new-product discovery predicts that OSR-trained models surface a more diverse or longer-tail set of recommendations; measuring recommendation diversity directly would test this.
- The paper compares ACR as well as CTR and OSR; a further analysis of whether ACR sits between them in GMV impact would clarify the conversion funnel.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims to compare recommender training objectives in e-commerce—click-through rate (CTR), add-to-cart rate (ACR), and order-submit rate (OSR)—using an online A/B test. The headline finding is that optimizing for OSR yields a GMV uplift "more than five times larger" than optimizing for CTR, without harming new product discovery. The abstract also announces feature-importance differences across objectives. The full text provided, however, is largely unreadable in the supplied version, and the only legible header identifies a different paper (arXiv:2508.10373v1 [cs.DB], 14 Aug 2025), not the target arXiv:2508.10377 [cs.IR]. No experimental details, statistical results, or methodology for the A/B test are accessible in the supplied material.
Significance. If the central claim is correct, the paper would provide a practically important result for e-commerce recommender systems: a simple change in training objective (from click labels to order-submit labels) could yield a large, direct improvement in gross merchandise value. The use of an online A/B test is the right kind of evidence for this claim and would substantially strengthen the literature, which often relies on offline proxy metrics. The paper also addresses a concrete business metric and the exploration side effect, which are both decision-relevant. However, the significance is entirely conditional on the live-traffic experiment being well-controlled and statistically sound; the supplied manuscript does not allow that condition to be verified.
major comments (4)
- [Full text / header] The supplied full text is not the target paper. The only legible header reads "arXiv:2508.10373v1 [cs.DB] 14 Aug 2025," whereas the abstract and title correspond to arXiv:2508.10377 [cs.IR]. No methods, results, or statistics for the claimed online A/B test are readable. Since the central claim depends entirely on that experiment, the manuscript as supplied does not support its headline result.
- [Abstract] The phrase "GMV uplift more than five times larger" is ambiguous. It is not specified whether this is the ratio of absolute GMV levels, the ratio of percentage improvements over a common baseline, or a comparison of regression coefficients. Without a precise definition and a baseline, the magnitude cannot be interpreted. The abstract should state the metric definition and the baseline.
- [Abstract / experimental methodology] The manuscript provides no confidence intervals, p-values, sample sizes, traffic-split percentages, randomization procedure, or seasonality controls for the online A/B test. Even if the correct full text were supplied, the abstract alone is insufficient to support a quantitative claim of this strength. At minimum, the paper must report uncertainty and experimental design details.
- [Abstract] The claim that OSR optimization succeeds "without sacrificing new product discovery" is not supported by any metric, table, or statistical test in the readable material. A measure of new-item exposure or discovery (e.g., novelty, coverage, cold-start engagement) needs to be defined and compared across variants with uncertainty estimates.
minor comments (2)
- [Full text] The full text appears to contain large portions of a different arXiv paper, including references to database systems and a different title. The authors should verify that the correct PDF was uploaded and that all pages are legible.
- [Abstract] The term "Order-Submit-Rate" is capitalized inconsistently with "add-to-cart rate" and "click-through rate"; a consistent style would improve readability.
Circularity Check
No circularity identifiable; the central claim is an external A/B test result, not a derivation from the outcome it predicts.
full rationale
The target paper's central claim — that optimizing for OSR yields a GMV uplift more than five times larger than optimizing for CTR — is presented as the result of an online A/B test, i.e., an external empirical measurement rather than a constructed derivation. There is no equation in the supplied material that defines OSR in terms of GMV, no parameter fitted to a subset and then 'predicted' on a closely related quantity, and no load-bearing self-citation. The supplied full text is unfortunately a different arXiv paper (2508.10373, cs.DB) and is mostly unreadable, so no specific derivation chain can be inspected; however, this is a verification/correctness problem, not a circularity problem. The fact that GMV is downstream of order submission is a structural mechanism that could explain the finding, but the paper does not use that relationship tautologically: the magnitude and sign of the difference between CTR- and OSR-trained recommenders remain empirical. Per the hard rules, absence of evidence for circularity is a non-finding, so the score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The only systematic difference between A/B variants is the recommender's training objective (CTR, ACR, or OSR); architecture, features, training data, and serving are held fixed.
- domain assumption GMV uplift measured over the test window is attributable to the models rather than to seasonality, traffic mix, or other time-varying effects.
- domain assumption The authors' shops are treated as representative of e-commerce recommendation settings for the purpose of generalization.
Cite this review
Pith. "Pith review of Clicks Versus Conversion: Choosing a Recommender's Training Objective in E-Commerce." pith.science (2026). https://pith.science/paper/CBVJTU2A
@misc{pith2026250810377,
author = {Pith},
title = {Pith review of: Clicks Versus Conversion: Choosing a Recommender's Training Objective in E-Commerce},
year = {2026},
howpublished = {\url{https://pith.science/paper/CBVJTU2A}},
note = {Machine review of arXiv:2508.10377}
}
read the original abstract
Ranking product recommendations to optimize for a high click-through rate (CTR) or for high conversion, such as add-to-cart rate (ACR) and Order-Submit-Rate (OSR, view-to-purchase conversion) are standard practices in e-commerce. Optimizing for CTR appears like a straightforward choice: Training data (i.e., click data) are simple to collect and often available in large quantities. Additionally, CTR is used far beyond e-commerce, making it a generalist, easily implemented option. ACR and OSR, on the other hand, are more directly linked to a shop's business goals, such as the Gross Merchandise Value (GMV). In this paper, we compare the effects of using either of these objectives using an online A/B test. Among our key findings, we demonstrate that in our shops, optimizing for OSR produces a GMV uplift more than five times larger than when optimizing for CTR, without sacrificing new product discovery. Our results also provide insights into the different feature importances for each of the objectives.
Reference graph
Works this paper leans on
-
[1]
������������������ ����������� ������� �������� ������ �� ���������������� ���� ������� ����� ����� ������� ������� ����� ��� �������� ������� �� ���� �������� ���� ������� �� �������� ������� ��� ����������� ������ ����������� ������ ����� ���� ������� ���������� �� ���� ����� ���� ���� ���� ����� ��������� ����� ����������� ����� ��������������� ������ ...
work page Pith review arXiv 2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.