{"id":"aeb5aef6-66b7-4914-8b94-31e3d89a1bb4","arxiv_id":"1908.03825","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Adding pairwise comparison features between an item and its neighbors in the search ranking improves offline MRR for eBay search by 3-5 percent, with neighborhood size 3 performing best.","lead":"eBay researchers added features that compare each search result item with the items ranked just above and below it, and retrained a ranking model. Offline tests on eBay search logs show the model with these neighborhood features improves mean reciprocal sale rank by 3 to 5 percent, suggesting that shoppers' buying decisions are influenced by nearby items in the results list.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The delta features depend on the ranked list used to construct them; if the new ranker reorders results, the features change, so the reported 5.01% offline MRR gain may not survive deployment, and the paper defers this to future work.","rationale":"The most load-bearing assumption is the stationarity of the neighborhood. The whole feature construction maps a ranked list to item features; the learned model is then evaluated on the same ranked list used to generate the features. A deployed ranker would break this loop. This is not a question of whether the offline MRR table is internally correct; it is a question of whether the result supports the paper's stated takeaway that 'neighborhood must be accounted for while placing an item in a ranked search result page.' The authors themselves flag the missing placement method in Section V. A secondary concern is the lack of an explicit significance test between neighborhood sizes 3 and 5, but the bootstrap error bars at least give some evidence; the feedback loop is more fundamental. The reader's conditional verdict already captures this, so no change is needed.","tokens_in":6903,"tokens_out":3114,"duration_ms":37802,"concrete_test":"Rerun the evaluation with a fixed-point procedure: start from the Model Base ranking on the validation set, compute delta features from that ranking, apply the trained prev-next model to re-rank, recompute delta features from the new ranking, and iterate until the ranking converges or for a fixed number of steps. Report MRR on the final self-consistent ranking. If the 5.01% gain over Model Base is not preserved, the offline improvement is contingent on the logged ranking and does not validate deployment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section IV-A, delta features are computed from the logged 'ranked list of top items impressed for the query,' and the same list is used for evaluation. Section III defines delta features as functions of the items at positions k±j above or below the current item. This makes the feature vector depend on the ranking that produced the log. When the trained delta-feature model is used to produce a new ranking, the neighborhoods change, which changes the feature values that the model scores. The paper does not simulate this feedback loop or propose a serving-time algorithm; Section V explicitly lists 'designing efficient methods to determine the placement of a candidate item based on its potential neighbors' as future work. The measured MRR gain is therefore an upper bound for a fixed neighborhood definition, not evidence that the learned weights transfer to the model's own ranking. The central claim that neighborhood influences preference may still be true, but the experiment as reported does not establish that a deployed ranker can exploit it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether the neighborhood of an item in a ranked search result list influences user preference, as measured by item sale. The authors define \"delta features\" that compare an item with the items ranked immediately above and below it, offering numerical, categorical, and vector-based variants with a distance-weighted decay function γ(j). They train LambdaMART models on eBay search logs, using baseline features plus prev, next, or combined prev+next delta features for neighborhood sizes m=1, 3, and 5. The offline evaluation reports MRR improvements over a production baseline, with the combined prev+next model at m=3 giving a 5.01% gain. The authors conclude that both above and below neighbors influence an item's preference and that neighborhood size matters.","tokens_in":7065,"tokens_out":3998,"duration_ms":41159,"significance":"If the claimed effect is real and can be exploited at serving time, the work is practically relevant for eCommerce search: it introduces a simple, interpretable family of contextual features and systematically shows that neighborhood size affects offline MRR. Strengths include the explicit feature definitions in Section III, the use of a production baseline, bootstrap-resampled error bars in the figures, and a clear ablation over prev, next, and combined features. The paper is also candid in Section V about the open problem of determining an item's placement under its own ranking, but that candor exposes the main gap between the offline experiment and the deployment claim.","major_comments":[{"comment":"The delta features are computed from the logged ranked list (Section IV-A) and the same list is used for evaluation. When the trained model is applied to produce a new ranking, the items above and below a candidate change, so the delta feature values change and the model's learned weights may no longer be appropriate. Section V explicitly defers \"designing efficient methods to determine the placement of a candidate item based on its potential neighbors\" to future work. The reported 5.01% MRR gain (Table I) is therefore an upper-bound measurement for a fixed neighborhood definition, not evidence that a deployed ranker can exploit the learned weights. A re-ranking simulation, in which the delta-feature model is applied iteratively and features are recomputed from the model's own output, is needed to support the deployment claim.","section":"IV-A/IV-B/V"},{"comment":"The paper does not state whether Model Base includes raw rank or position features. If position information is absent, the delta features could be acting as a position-leakage signal, since they are deterministic functions of positions in the logged ranking. The observed MRR gain might then reflect the model learning to upweight item features that correlate with higher logged ranks rather than a genuine neighborhood-comparison effect. Adding a control model that includes the raw rank (or a position indicator) as a feature would isolate the contribution of the comparative content.","section":"IV-A"},{"comment":"Table I reports percentage changes in MRR without confidence intervals or significance tests, even though the figures show bootstrap error bars. For a dataset of about 20,000 queries, a gain of 0.07% (next, m=1) is likely not statistically significant, and the differences among the m=3 and m=5 gains need uncertainty quantification. The headline numbers should be accompanied by confidence intervals or a paired test across bootstrap samples.","section":"Table I"}],"minor_comments":[{"comment":"In the delta feature definitions, f_k denotes the feature value of the item at position k, but f is also used for the ranking function in Section I; please use distinct notation to avoid confusion.","section":"Section III"},{"comment":"The column heading \"prev next prev next\" repeats \"prev next\"; label the columns \"prev\", \"next\", and \"prev+next\".","section":"Table I"},{"comment":"The phrase \"raked search result page\" should be \"ranked search result page\".","section":"Section V"},{"comment":"The word \"addtion\" should be \"addition\".","section":"Section I"},{"comment":"The decay function γ(j) is introduced in Section III but set to 1 for all experiments; state explicitly that the distance-weighted variant is not evaluated here, so the claim that \"farther neighbors may have a different influence\" is only qualitatively supported.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is quite short for a journal paper and reads like an industry report. The main technical concern (feedback loop from re-ranking) is real and should be addressed before publication. The paper's contribution would be significantly strengthened by a serving-time simulation or a model that jointly scores candidates and their neighborhoods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of arXiv:1908.03825. The concrete new thing here is a clean, short industrial study of 'delta features' that compare an item to its neighbors in a ranked eCommerce list, and an offline MRR gain of about 5% for neighborhood size 3 at eBay. The paper is well written and honest; it actually tried prev, next, and combined, different sizes, and it reports bootstrap error bars in the figures. The result is genuinely new in the eCommerce ranking literature as far as I know, and the feature family is easy to understand and implement.\n\nThe soft spots are real but not disqualifying. The most serious is the fixed-neighborhood feedback loop you flagged: delta features are computed from the logged production ranking, and the same ranking is used for evaluation. If the new model reorders the list, the neighborhoods change and the features change. The paper explicitly defers this to future work ('designing efficient methods to determine the placement of a candidate item based on its potential neighbors'), so the 5.01% gain is an upper bound on what a deployed ranker can get without solving that circularity. I don't think that invalidates the claim that neighborhood information carries predictive signal, but it does mean 'neighborhood influences preference' is supported as an offline correlational finding, not as a deployable improvement.\n\nSecond, there is no position-bias control. The baseline lacks raw rank or position features, so part of the gain could come from the model learning that high-ranked items sell more. Adding a position feature set would settle this. Third, Table I gives percentage gains without confidence intervals, despite the bootstrap error bars in the figures; a single table with CIs would tighten the paper. Minor: the dataset is sampled from queries that ended in a sale, so the population is success-biased; and gamma(j) is set to 1, so the distance-weighting idea is untested.\n\nCitation pattern is fine and not self-promotional. The math is simple and correct. I'd send this to a serious referee rather than desk reject: it's a clean empirical result with a clear weakness that is explicitly acknowledged. A reasonable revision could add a position-bias baseline, report CIs in Table I, and either simulate the feedback loop or state the serving condition under which the gain holds. The paper is mainly for people working on eCommerce ranking and contextual features; a reading group could use it as a case study of offline-to-online gaps.","headline":"Industrial study of delta features for neighborhood-aware eCommerce ranking; offline gains look real but the fixed-neighborhood feedback loop makes deployment claims unproven.","tokens_in":7590,"tokens_out":3116,"would_cite":false,"duration_ms":35313,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ranking an item against its neighbors in search results improves mean reciprocal sale rank by up to 5.01 percent, with a neighborhood of three items working best.","keywords":["eCommerce search","learning to rank","delta features","neighborhood effects","item preference","mean reciprocal rank","implicit relevance feedback","comparative decision making"],"falsifier":"Compute the best model's predicted ranking on the test queries, rebuild the delta features from that predicted ranking instead of the logged one, and re-evaluate with the same model. If the 5.01% gain over baseline disappears or reverses, the result depends on the fixed baseline ordering rather than on stable neighborhood influence; an online A/B test measuring sale rank after deployment would settle the same question.","tokens_in":6717,"feed_emoji":"🛒","tokens_out":8956,"duration_ms":91001,"temperature":0.7,"pith_summary":"Shoppers on eCommerce sites usually do not decide on an item in isolation; they compare it with the items shown around it. This paper tries to prove that this comparative behavior can be exploited by a ranker: it constructs delta features that measure how an item differs, in user-visible attributes such as price and popularity, from the items ranked above and below it, and adds them to an existing baseline ranking model. On logged search data, models with these features improve mean reciprocal sale rank by 3 to 5 percent over the baseline, with the best configuration (both above and below neighbors, neighborhood size 3) gaining 5.01 percent. The claim matters because it says a meaningful portion of purchase preference is contextual and can be captured by feature engineering alone, without changing the loss function or the scoring architecture.","feed_headline":"Neighborhood-aware features lift eCommerce search ranking by 5%","feed_subtitle":"Comparing each item with the items around it in search results lifts ranking quality by up to 5.01%.","key_machinery":"The central object is the delta feature. For a candidate item at position $k$, numerical prev features are defined as $D_k^{m,prev} = (1/m) \\sum_{j=1}^{m} (f_{k-j} - f_k)/\\gamma(j)$, and next features use $f_{k+j}$ in place of $f_{k-j}$; categorical features replace the difference with an indicator that is 1 when the values match and 0 otherwise; vector-based versions use $1-\\cos(\\alpha,\\beta)$ as a distance. $\\gamma(j)$ is a distance-weighted decay function that is set to 1 in the experiments, so closer and farther neighbors contribute equally. These features turn the ranked list into a per-item comparative summary, letting an otherwise item-by-item scorer behave as if it were group-aware. Because the baseline and proposed models differ only in whether delta features are added, the measured MRR changes are attributed to the neighborhood signal.","core_discovery":"The paper's central claim is that preference for an item in eCommerce search is partly determined by the items around it, and that this context can be captured by delta features: for each user-visible attribute, the difference between an item and the m items ranked above (prev) and below (next) it, averaged over the neighborhood. On logged search data from a large marketplace, adding these features to the production ranking model raises mean reciprocal sale rank by about 3 to 5 percent across configurations. The best configuration, prev and next features with a neighborhood of size 3, improves MRR by 5.01 percent relative to the baseline; prev-only and next-only versions at the same size improve it by 4.65 and 4.45 percent. Neighborhood size 1 produces a small or negative change, and size 5 underperforms size 3, leading the authors to conclude that the neighborhood of an item measurably influences its preference and that neighborhood size is a meaningful modeling choice.","pith_inferences":["The logged ranking that defines the neighborhoods is treated as fixed; a deployed ranker that reorders results changes its own delta-feature inputs, and the offline gain may shrink once this feedback loop is included.","The feature construction is agnostic to the scoring model, so the same delta features should transfer to other learning-to-rank objectives and to other marketplaces, though the size-3 optimum may not transfer.","Attribute-level delta weights give an operational reading of diversity: the attributes that most shift preference are the dimensions along which shoppers perceive a choice set as varied, a notion the paper names as future work.","Using a nontrivial distance decay $\\gamma(j)$ in construction might recover some of the lost performance at neighborhood size 5, since the current experiments treat all neighbors equally."],"forward_implications":["Combining prev and next delta features outperforms either alone, so the context above and below an item carries complementary information about purchase preference.","Neighborhood size behaves like a hyperparameter with an interior optimum: size 3 beats sizes 1 and 5, so the influence of neighbors does not simply increase with distance.","Delta features can be built from user-visible attributes such as price, popularity, and retail standards, making the approach applicable wherever a logged ranked list exists.","The reported improvements are in offline MRR, meaning the first item a shopper buys appears closer to the top of the ranking when neighborhood context is used.","Delta features rank high in feature importance, indicating that the trained models rely on comparative context rather than treating these additions as incidental."],"supporting_citations":[{"why":"Supplies the gradient-boosted learning-to-rank algorithm used to train both the baseline and all delta-feature models.","marker":"[1]"},{"why":"Defines Mean Reciprocal Rank, the offline metric used to compare every model against the baseline.","marker":"[4]"},{"why":"Supplies the behavioral premise that people evaluate options differently when choices are presented together rather than separately.","marker":"[15]"},{"why":"Establishes that click decisions in search are affected by rank and by other documents shown, motivating neighborhood features.","marker":"[16]"},{"why":"Extends the implicit-feedback evidence to clicks and query reformulations, supporting the use of logged behavior for labels.","marker":"[17]"},{"why":"Introduces the cascade click model in which click probability at a rank depends on documents above, the direction captured by prev features.","marker":"[18]"},{"why":"Models browsing behavior where the distance to previously clicked documents affects subsequent clicks, the distance intuition behind delta features.","marker":"[19]"},{"why":"Makes the case for learning to rank in eCommerce search and motivates sale or conversion as the relevance target.","marker":"[12]"}],"fun_headline_variants":["Item context boosts eCommerce search ranking by up to 5%","Surrounding items matter: 5% better search ranking with neighbors","Neighborhood-aware ranking improves MRR by 5% in eCommerce search","For better search ranking, look at what's around each item","Delta features from item neighbors lift search MRR by 5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments assume the ranked list used to construct delta features is the same fixed list the learned model is evaluated on; if the new ranker changes the order, it changes every item's neighborhood and thus the feature values the model was trained on.","fun_headline_variants_meta":{"raw":{"variants":["Item context boosts eCommerce search ranking by up to 5%","Surrounding items matter: 5% better search ranking with neighbors","Neighborhood-aware ranking improves MRR by 5% in eCommerce search","For better search ranking, look at what's around each item","Delta features from item neighbors lift search MRR by 5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1476,"prompt_tokens":1011,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":627,"tokens_out":465,"duration_ms":5497,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:00:34.460801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the best model's predicted ranking on the test queries, rebuild the delta features from that predicted ranking instead of the logged one, and re-evaluate with the same model. If the 5.01% gain over baseline disappears or reverses, the result depends on the fixed baseline ordering rather than on stable neighborhood influence; an online A/B test measuring sale rank after deployment would settle the same question.","supporting_citations":[{"cited_title":"From ranknet to lambdarank to lambdamart: An overview,","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-boosted learning-to-rank algorithm used to train both the baseline and all delta-feature models."},{"cited_title":"Craswell, Mean Reciprocal Rank","cited_arxiv_id":null,"evidence_quote":"Defines Mean Reciprocal Rank, the offline metric used to compare every model against the baseline."},{"cited_title":"Reversals of preference between bids and choices in gambling decisions","cited_arxiv_id":null,"evidence_quote":"Supplies the behavioral premise that people evaluate options differently when choices are presented together rather than separately."},{"cited_title":"Accurately interpreting clickthrough data as implicit feedback,","cited_arxiv_id":null,"evidence_quote":"Establishes that click decisions in search are affected by rank and by other documents shown, motivating neighborhood features."},{"cited_title":"Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search,","cited_arxiv_id":null,"evidence_quote":"Extends the implicit-feedback evidence to clicks and query reformulations, supporting the use of logged behavior for labels."},{"cited_title":"An experimental comparison of click position-bias models,","cited_arxiv_id":null,"evidence_quote":"Introduces the cascade click model in which click probability at a rank depends on documents above, the direction captured by prev features."},{"cited_title":"A user browsing model to predict search engine click data from past observations,","cited_arxiv_id":null,"evidence_quote":"Models browsing behavior where the distance to previously clicked documents affects subsequent clicks, the distance intuition behind delta features."}],"review_version":1}