REVIEW 3 major objections 5 minor 19 references
Influence of Neighborhood on the Preference of an Item in eCommerce Search
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Ranking an item against its neighbors in search results improves mean reciprocal sale rank by up to 5.01 percent, with a neighborhood of three items working best.
desk verdict Industrial study of delta features for neighborhood-aware eCommerce ranking; offline gains look real but the fixed-neighborhood feedback loop makes deployment claims unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the delta feature. For a candidate item at position $k$, numerical prev features are defined as $D_k^{m,prev} = (1/m) \sum_{j=1}^{m} (f_{k-j} - f_k)/\gamma(j)$, and next features use $f_{k+j}$ in place of $f_{k-j}$; categorical features replace the difference with an indicator that is 1 when the values match and 0 otherwise; vector-based versions use $1-\cos(\alpha,\beta)$ as a distance. $\gamma(j)$ is a distance-weighted decay function that is set to 1 in the experiments, so closer and farther neighbors contribute equally. These features turn the ranked list into a per-item comparative summary, letting an otherwise item-by-item scorer behave as if it were group-aware. Because the baseline and proposed models differ only in whether delta features are added, the measured MRR changes are attributed to the neighborhood signal.
What would settle it
Compute the best model's predicted ranking on the test queries, rebuild the delta features from that predicted ranking instead of the logged one, and re-evaluate with the same model. If the 5.01% gain over baseline disappears or reverses, the result depends on the fixed baseline ordering rather than on stable neighborhood influence; an online A/B test measuring sale rank after deployment would settle the same question.
Extended reading notes
Core claim
The paper's central claim is that preference for an item in eCommerce search is partly determined by the items around it, and that this context can be captured by delta features: for each user-visible attribute, the difference between an item and the m items ranked above (prev) and below (next) it, averaged over the neighborhood. On logged search data from a large marketplace, adding these features to the production ranking model raises mean reciprocal sale rank by about 3 to 5 percent across configurations. The best configuration, prev and next features with a neighborhood of size 3, improves MRR by 5.01 percent relative to the baseline; prev-only and next-only versions at the same size improve it by 4.65 and 4.45 percent. Neighborhood size 1 produces a small or negative change, and size 5 underperforms size 3, leading the authors to conclude that the neighborhood of an item measurably influences its preference and that neighborhood size is a meaningful modeling choice.
Load-bearing premise
The experiments assume the ranked list used to construct delta features is the same fixed list the learned model is evaluated on; if the new ranker changes the order, it changes every item's neighborhood and thus the feature values the model was trained on.
Editorial extensions
If this is right
- Combining prev and next delta features outperforms either alone, so the context above and below an item carries complementary information about purchase preference.
- Neighborhood size behaves like a hyperparameter with an interior optimum: size 3 beats sizes 1 and 5, so the influence of neighbors does not simply increase with distance.
- Delta features can be built from user-visible attributes such as price, popularity, and retail standards, making the approach applicable wherever a logged ranked list exists.
- The reported improvements are in offline MRR, meaning the first item a shopper buys appears closer to the top of the ranking when neighborhood context is used.
- Delta features rank high in feature importance, indicating that the trained models rely on comparative context rather than treating these additions as incidental.
Reading between the lines
- The logged ranking that defines the neighborhoods is treated as fixed; a deployed ranker that reorders results changes its own delta-feature inputs, and the offline gain may shrink once this feedback loop is included.
- The feature construction is agnostic to the scoring model, so the same delta features should transfer to other learning-to-rank objectives and to other marketplaces, though the size-3 optimum may not transfer.
- Attribute-level delta weights give an operational reading of diversity: the attributes that most shift preference are the dimensions along which shoppers perceive a choice set as varied, a notion the paper names as future work.
- Using a nontrivial distance decay $\gamma(j)$ in construction might recover some of the lost performance at neighborhood size 5, since the current experiments treat all neighbors equally.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper investigates whether the neighborhood of an item in a ranked search result list influences user preference, as measured by item sale. The authors define "delta features" that compare an item with the items ranked immediately above and below it, offering numerical, categorical, and vector-based variants with a distance-weighted decay function γ(j). They train LambdaMART models on eBay search logs, using baseline features plus prev, next, or combined prev+next delta features for neighborhood sizes m=1, 3, and 5. The offline evaluation reports MRR improvements over a production baseline, with the combined prev+next model at m=3 giving a 5.01% gain. The authors conclude that both above and below neighbors influence an item's preference and that neighborhood size matters.
Significance. If the claimed effect is real and can be exploited at serving time, the work is practically relevant for eCommerce search: it introduces a simple, interpretable family of contextual features and systematically shows that neighborhood size affects offline MRR. Strengths include the explicit feature definitions in Section III, the use of a production baseline, bootstrap-resampled error bars in the figures, and a clear ablation over prev, next, and combined features. The paper is also candid in Section V about the open problem of determining an item's placement under its own ranking, but that candor exposes the main gap between the offline experiment and the deployment claim.
major comments (3)
- [IV-A/IV-B/V] The delta features are computed from the logged ranked list (Section IV-A) and the same list is used for evaluation. When the trained model is applied to produce a new ranking, the items above and below a candidate change, so the delta feature values change and the model's learned weights may no longer be appropriate. Section V explicitly defers "designing efficient methods to determine the placement of a candidate item based on its potential neighbors" to future work. The reported 5.01% MRR gain (Table I) is therefore an upper-bound measurement for a fixed neighborhood definition, not evidence that a deployed ranker can exploit the learned weights. A re-ranking simulation, in which the delta-feature model is applied iteratively and features are recomputed from the model's own output, is needed to support the deployment claim.
- [IV-A] The paper does not state whether Model Base includes raw rank or position features. If position information is absent, the delta features could be acting as a position-leakage signal, since they are deterministic functions of positions in the logged ranking. The observed MRR gain might then reflect the model learning to upweight item features that correlate with higher logged ranks rather than a genuine neighborhood-comparison effect. Adding a control model that includes the raw rank (or a position indicator) as a feature would isolate the contribution of the comparative content.
- [Table I] Table I reports percentage changes in MRR without confidence intervals or significance tests, even though the figures show bootstrap error bars. For a dataset of about 20,000 queries, a gain of 0.07% (next, m=1) is likely not statistically significant, and the differences among the m=3 and m=5 gains need uncertainty quantification. The headline numbers should be accompanied by confidence intervals or a paired test across bootstrap samples.
minor comments (5)
- [Section III] In the delta feature definitions, f_k denotes the feature value of the item at position k, but f is also used for the ranking function in Section I; please use distinct notation to avoid confusion.
- [Table I] The column heading "prev next prev next" repeats "prev next"; label the columns "prev", "next", and "prev+next".
- [Section V] The phrase "raked search result page" should be "ranked search result page".
- [Section I] The word "addtion" should be "addition".
- [Section IV-A] The decay function γ(j) is introduced in Section III but set to 1 for all experiments; state explicitly that the distance-weighted variant is not evaluated here, so the claim that "farther neighbors may have a different influence" is only qualitatively supported.
Circularity Check
No significant circularity: the MRR gains are measured on held-out data with delta features computed from logged rankings, and the paper's own future-work section acknowledges the deployment feedback loop.
full rationale
The paper's central claim is an empirical result: adding delta features computed from items ranked above and below a given item to a LambdaMART ranker improves offline MRR by up to 5.01% over a baseline trained on the same data without delta features. The delta features are constructed from user-perceivable item attributes (price, popularity, retail standards) and from the logged ranked list of impressed items; they are not constructed from the sale labels or from the model's own output. The evaluation uses a held-out 20% of queries, and the baseline and proposed models share the same hyperparameters and features except for the delta features. No equation in Section III defines the target (sale preference or MRR) in terms of the delta features, and no fitted parameter is renamed as a prediction. The paper does not rely on self-citations: the only cited methods are standard external works (LambdaMART, listwise ranking, click models) that are not load-bearing in a way that predetermines the result. The main validity concern is that test-time delta features are computed from the logged ranking rather than from the ranker's own reordered output, so a deployed model that changes the ranking would change its own feature values; however, the authors explicitly identify this as future work ('designing efficient methods to determine the placement of a candidate item based on its potential neighbors'). That is a deployment/serving mismatch and a limitation of the experimental setup, not a circular derivation. Under the stated hard rules, no circular step can be quoted with a specific reduction, so the appropriate finding is no significant circularity with score 0.
Assumptions & free parameters
free parameters (1)
- distance weighted decay function gamma(j) =
1 (no decay)
assumptions (3)
- domain assumption Implicit feedback via sale is a valid measure of item preference in the ranked list.
- domain assumption The logged ranked list of impressed items is an unbiased representation of what users saw.
- domain assumption The delta features constructed from the production ranking remain informative when the model is retrained and re-ranks the list.
Cite this review
Pith. "Pith review of Influence of Neighborhood on the Preference of an Item in eCommerce Search." pith.science (2026). https://pith.science/paper/5FZFHZLJ
@misc{pith2026190803825,
author = {Pith},
title = {Pith review of: Influence of Neighborhood on the Preference of an Item in eCommerce Search},
year = {2026},
howpublished = {\url{https://pith.science/paper/5FZFHZLJ}},
note = {Machine review of arXiv:1908.03825}
}
abstract
Surfacing a ranked list of items for a search query to help buyers discover inventory and make purchase decisions is a critical problem in eCommerce search. Typically, items are independently predicted with a probability of sale with respect to a given search query. But in a dynamic marketplace like eBay, even for a single product, there are various different factors distinguishing one item from another which can influence the purchase decision for the user. Users have to make a purchase decision by considering all of these options. Majority of the existing learning to rank algorithms model the relative relevance between labeled items only at the loss functions like pairwise or list-wise losses. But they are limited to point-wise scoring functions where items are ranked independently based on the features of the item itself. In this paper, we study the influence of an item's neighborhood to its purchase decision. Here, we consider the neighborhood as the items ranked above and below the current item in search results. By adding delta features comparing items within a neighborhood and learning a ranking model, we are able to experimentally show that the new ranker with delta features outperforms our baseline ranker in terms of Mean Reciprocal Rank (MRR). The ranking models with proposed delta features result in $3-5\%$ improvement in MRR over the baseline model. We also study impact of different sizes for neighborhood. Experimental results show that neighborhood size $3$ perform the best based on MRR with an improvement of $4-5\%$ over the baseline model.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
From ranknet to lambdarank to lambdamart: An overview,
C. J. Burges, “From ranknet to lambdarank to lambdamart: An overview,” Learning, vol. 11, no. 23-581, p. 81, 2010
work page 2010
-
[2]
Learning to rank: from pairwise approach to listwise approach,
Z. Cao, T. Qin, T.-Y . Liu, M.-F. Tsai, and H. Li, “Learning to rank: from pairwise approach to listwise approach,” in Proceedings of the 24th international conference on Machine learning . ACM, 2007, pp. 129–136
work page 2007
-
[3]
Listwise approach to learning to rank: theory and algorithm,
F. Xia, T.-Y . Liu, J. Wang, W. Zhang, and H. Li, “Listwise approach to learning to rank: theory and algorithm,” in Proceedings of the 25th international conference on Machine learning . ACM, 2008, pp. 1192– 1199
work page 2008
-
[4]
Craswell, Mean Reciprocal Rank
N. Craswell, Mean Reciprocal Rank . Boston, MA: Springer US, 2009, pp. 1703–1703
work page 2009
-
[5]
Learning to rank for information retrieval,
T.-Y . Liu et al., “Learning to rank for information retrieval,”F oundations and Trends® in Information Retrieval , vol. 3, no. 3, pp. 225–331, 2009
work page 2009
-
[6]
A short introduction to learning to rank,
H. Li, “A short introduction to learning to rank,” IEICE TRANSACTIONS on Information and Systems , vol. 94, no. 10, pp. 1854–1862, 2011
work page 2011
-
[7]
Query chains: learning to rank from implicit feedback,
F. Radlinski and T. Joachims, “Query chains: learning to rank from implicit feedback,” in Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining . ACM, 2005, pp. 239–248
work page 2005
-
[8]
Accurately interpreting clickthrough data as implicit feedback,
T. Joachims, L. A. Granka, B. Pan, H. Hembrooke, and G. Gay, “Accurately interpreting clickthrough data as implicit feedback,” inSigir, vol. 5, 2005, pp. 154–161
work page 2005
Show all 19 references
-
[9]
Learning user interaction models for predicting web search result preferences,
E. Agichtein, E. Brill, S. Dumais, and R. Ragno, “Learning user interaction models for predicting web search result preferences,” in Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval . ACM, 2006, pp. 3–10
2006
-
[10]
Learning to order things,
W. W. Cohen, R. E. Schapire, and Y . Singer, “Learning to order things,” in Advances in Neural Information Processing Systems , 1998, pp. 451– 457
1998
-
[11]
Optimizing search engines using clickthrough data,
T. Joachims, “Optimizing search engines using clickthrough data,” in Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , ser. KDD ’02. New York, NY , USA: ACM, 2002, pp. 133–142. [Online]. Available: http://doi.acm.org/10.1145...
2002
-
[12]
On application of learning to rank for e-commerce search,
S. K. Karmaker Santu, P. Sondhi, and C. Zhai, “On application of learning to rank for e-commerce search,” in Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval , ser. SIGIR ’17. New York, NY , USA: ACM, 2017, pp. 475...
2017
-
[13]
Learning for search result diversification,
Y . Zhu, Y . Lan, J. Guo, X. Cheng, and S. Niu, “Learning for search result diversification,” in Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval . ACM, 2014, pp. 293–302
2014
-
[14]
Diversifying search results,
R. Agrawal, S. Gollapudi, A. Halverson, and S. Ieong, “Diversifying search results,” in Proceedings of the Second ACM International Conference on Web Search and Data Mining , ser. WSDM ’09. New York, NY , USA: ACM, 2009, pp. 5–14. [Online]. Available: http://doi.acm.org/10.114...
2009
-
[15]
Reversals of preference between bids and choices in gambling decisions
S. Lichtenstein and P. Slovic, “Reversals of preference between bids and choices in gambling decisions.” Journal of experimental psychology, vol. 89, no. 1, p. 46, 1971
1971
-
[16]
Accurately interpreting clickthrough data as implicit feedback,
T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay, “Accurately interpreting clickthrough data as implicit feedback,” in Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval , 2005, pp. 154–161
2005
-
[17]
Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search,
T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay, “Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search,” ACM Transactions on Information Systems, vol. 25, no. 2, 2007
2007
-
[18]
An experimental comparison of click position-bias models,
N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey, “An experimental comparison of click position-bias models,” in Proceedings of the 2008 International Conference on Web Search and Data Mining , 2008, pp. 87–94
2008
-
[19]
A user browsing model to predict search engine click data from past observations,
G. Dupret and B. Piwowarski, “A user browsing model to predict search engine click data from past observations,” in Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval , 2008, pp. 331–338
2008
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.