Pith. sign in

REVIEW 4 major objections 5 minor 20 references

Two-Stage Session-based Recommendations with Candidate Rank Embeddings

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that re-ranking the top-100 candidates of a session-based generator with Candidate Rank Embeddings improves Recall@20 and MRR@20 over the generator, and an online test showed a 2.84% relative CTR increase.

desk verdict A simple, well-ablated re-ranking trick with real offline and online gains; the missing test-time coverage report keeps the offline numbers from being fully interpretable. read the letter →

arxiv 1908.08284 v1 pith:N2FPHBG2 submitted 2019-08-22 cs.IR cs.LG

José Antonio Sánchez Rodríguez , Jui-Chieh Wu , Mustafa Khandwawala This is my paper · ORCID
classification cs.IRcs.LG
keywords session-basedrecommendationcandidaterankembeddingstwo-stagerecommenderre-rankingattentionnetworknext-clickpredictionRecall@20MRR@20
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a lightweight two-stage upgrade can make session-based recommendation work on a fashion similar-item task where deep sequence models fail out of the box. Its recipe is to take the top k candidates from an existing recommender, notably item-item collaborative filtering, and re-rank those candidates using the user's click session plus a new Candidate Rank Embedding that tells the re-ranker where each candidate stood in the original list. On the fashion dataset this lifts Recall@20 from 0.8106 to 0.8381 and MRR@20 from 0.2611 to 0.2981 over the generator, and an online test measured a 2.84% relative CTR increase. The same module also improves the STAMP and NARM session models on two public next-click datasets, which the paper reads as evidence that the rank-embedding trick transfers.

What carries the argument

The central object is the Candidate Rank Embedding matrix $W_{CR}\in\mathbb{R}^{k\times d_{CRE}}$: one learned vector per rank position in the candidate list, shared across all user sessions. The re-ranker scores the k candidates as $\mathrm{softmax}(V_C^\top h_e + W_{CR}^\top h_r)$, where $V_C$ holds the candidate item embeddings, $h_u = h_s\odot h_t$ is the session encoding from the STAMP encoder, and $h_e$, $h_r$ are two small feed-forward projections of $h_u$. The item term models what the user wants; the rank term models the user's preference for where the target sits in the generator's list, which lets the generator's global ranking act as a prior during training.

What would settle it

Look at the test sets and count how often the true next-clicked item appears in the generator's shortlist of k candidates; if that share is close to or below the reported Recall@20, the claimed gains are mostly reordering easy hits and will vanish wherever the generator's coverage is lower. A cheaper check: shuffle the rank embeddings per session; if Recall@20 does not drop, the rank position signal is not what carries the gain.

Watch

Extended reading notes

Core claim

The central claim is that the missing ingredient is not a better session model but a way to inject the candidate generator's ranking into a session-aware re-ranker. Candidate Rank Embeddings are positional embeddings over rank positions; added to item embeddings in the softmax, they let the re-ranker learn both what to recommend and where in the generator's list the user tends to click. With this addition, re-ranking the top-100 candidates of item-item collaborative filtering beats the generator itself on the Fashion-Similar dataset, and an online e-commerce test measured a 2.84% relative CTR increase with a p-value no larger than 0.05. The same re-ranker also improves the STAMP and NARM session models on YooChoose 1/4 and Diginetica.

Load-bearing premise

The load-bearing premise is that the generator's shortlist of k candidates usually contains the true next item, because the re-ranker only reorders that shortlist and can never bring back an item the generator left out.

Editorial extensions

If this is right

  • Any existing top-k recommender can be upgraded without retraining its generator; only the small re-ranker needs training.
  • Re-ranking can make session-based methods viable on domains where they underperform a simple collaborative filtering baseline.
  • The same rank-embedding module transfers to next-click prediction, improving both STAMP and NARM on YooChoose 1/4 and Diginetica.
  • Candidate-set size matters: gains are visible even at small k and flatten or drop once k is large, suggesting a practical sweet spot around k=100.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The rank embedding behaves like a learned position-bias term, so the same idea should transfer to any cascade in which a cheap generator proposes and a more expensive model reorders, not only session recommenders.
  • The plateau once k exceeds about 100 suggests low-ranked positions get too little training signal; sharing one embedding across several low ranks, which the paper mentions but does not test, is a straightforward extension.
  • The paper never reports how often the true target appears in the generator's top-k on the test sets, so the ceiling of the method is unquantified; reporting that coverage rate is the cheapest decisive experiment.
  • Because the module improved two different deep generators, it could also re-rank lists merged from several generators, a setting the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes a two-stage session-based recommendation architecture in which a pre-trained candidate generator (I2I-CF, STAMP, or NARM) produces a top-k list, and a STAMP-based re-ranker with Candidate Rank Embeddings (CREs) re-scores those candidates. The final list is the re-ranked top-k followed by the generator's remaining candidates. The re-ranker is trained with cross-entropy on the next-click target, using both item embeddings and positional rank embeddings. Experiments on the proprietary Fashion-Similar dataset and on YooChoose 1/4 and Diginetica report Recall@20 and MRR@20 improvements over the corresponding generators, and an online A/B test on Zalando reports a +2.84% relative CTR increase. The paper also includes a controlled ablation (with vs. without CRE) and a sensitivity analysis over k.

Significance. If the results hold, the paper makes a useful practical contribution: it shows a lightweight, generator-agnostic re-ranking layer that injects session context into an existing recommender and yields consistent offline and online improvements. The controlled ablation is a strength, as it isolates the contribution of the rank embeddings. The use of three different candidate generators and three datasets, including proprietary production data, strengthens the generality of the finding. The online validation is a notable strength, though the details are sparse. The main weaknesses are the absence of test-time candidate coverage statistics and the lack of uncertainty quantification in the offline metrics, both of which limit the interpretability of the claims.

major comments (4)
  1. [§5.5, Eq. (2)] The paper does not report the fraction of test instances whose true next item falls within the candidate set C (coverage at k=100), nor the oracle ceiling achievable by a perfect re-ranker over C. Because the final list's top-20 positions are drawn entirely from C whenever k >= 20, Recall@20 is upper-bounded by this coverage. Without this number, the reported gains (e.g., +0.0275 Recall@20 on Fashion-Similar) cannot be separated from headroom effects of the candidate generator, and the reader cannot judge how much of the possible improvement is achieved. Please report coverage for each dataset/generator combination, together with an oracle Recall@20 that assumes the target, when present in C, is ranked first.
  2. [Tables 1 and 2] Offline metrics are reported as point estimates with no variance, confidence intervals, or significance tests, yet the abstract and Section 5.5 refer to "significant improvements". With test sets containing hundreds of thousands of examples, even small differences may be statistically significant, but the claim needs explicit support. Please report standard errors or confidence intervals (e.g., from re-sampling test instances or multiple random seeds), or a paired significance test.
  3. [§5.2] The training of the re-ranker filters out examples whose target is outside the candidate set C. The paper should report the fraction of training examples retained after this filter for each dataset, as this fraction determines how representative the training distribution is and is directly related to the coverage issue in major comment 1. This number is cheap to compute and would help the reader interpret the method's behavior.
  4. [§4.2, Eq. (4)] The network architecture is under-specified. The dimensions of MLP1 and MLP2, the candidate rank embedding dimension d_cre, and the initialization of the item embedding matrix V_C are not given; the text says "all embeddings are initialized randomly" after stating that item embeddings are initialized with STAMP settings. Please provide a full hyperparameter table (hidden sizes, number of layers, d_cre, regularization) to allow reproducibility.
minor comments (5)
  1. [§5.6] The online test description is too brief. Please report the number of users/impressions, the duration of the test, the traffic allocation, and how the p-value was computed (e.g., which test was used).
  2. [§5.3] Large numbers should be formatted with thousands separators (e.g., 8,353,562) for readability.
  3. [Abstract] The phrase "significant improvements" should be qualified with the evaluation setting (offline metrics and the online CTR test) to avoid implying statistical significance for all metrics.
  4. [Figure 3] The curves are labeled only by epoch; please add a legend to indicate which line corresponds to RR-I2I-CF versus RRCRE-I2I-CF, as the colors/linestyles may be indistinguishable in print.
  5. [§4.2] The term "Candidate Rank Embedding" is introduced, but the relationship to positional embeddings used in sequence models (e.g., Transformer) is not discussed; citing that line of work would clarify novelty.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the re-ranker is a standard supervised model trained on held-out next-click targets, with candidate rank as an input feature rather than a construction of the target.

full rationale

The paper's central claim is that a two-stage re-ranker with Candidate Rank Embeddings improves Recall@20 and MRR@20 relative to candidate generators. The derivation chain is self-contained: the candidate generator G produces a ranked list C (Eq. 2), the re-ranker R scores candidates using a user-session encoder hu (Eq. 3) and a softmax over item and rank embeddings (Eq. 4), and R is trained with cross-entropy on the next-click target when it lies in C (Sections 4.2 and 5.2). The CRE is an input feature, namely the position of each candidate in G's list, and is not defined in terms of the target or fitted to the evaluation metric. The reported gains are measured on test data against the generator's original ordering, with a no-CRE ablation (RR vs RRCRE) in Figure 3, so the improvement is not forced by construction. The known limitation that targets outside the generator's top-k cannot be recovered is a coverage and ceiling issue, not a circular reduction; the paper does not hide this in its equations, and no self-citation is load-bearing. Therefore the paper merits a circularity score of 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the candidate generator producing high-quality top-k lists, the STAMP encoder transferring to re-ranking, and the training-filtered distribution matching the test distribution. These are empirical assumptions rather than formal axioms. No new physical or conceptual entities are introduced; the Candidate Rank Embedding matrix is a model parameter, not an independent postulate.

free parameters (4)
  • k, number of candidates re-ranked = 100 (range 5-200 tested)
    The re-ranker operates on the top k candidates from the generator. The choice of k trades off coverage against training signal; the paper tests multiple values in Figure 4 and sets k=100 for the main experiments. This is a hyperparameter chosen by hand.
  • Item embedding dimension d = not reported
    The dimension of the item embeddings V and the CRE matrix W_CR is not stated in the paper; it likely follows STAMP settings, but this is a free architectural choice that affects model capacity and is not justified.
  • MLP hidden sizes for MLP1 and MLP2 = not reported
    The two non-linear projections have learnable weights with hidden sizes that are not specified. These are part of the model architecture and are chosen without reported tuning.
  • Training epochs, learning rate, batch size = 5 epochs, 0.001, 512
    These training hyperparameters are stated in Section 5.2, but no sensitivity analysis or tuning procedure is provided.
assumptions (3)
  • domain assumption The STAMP encoder E_STAMP, taken from Liu et al. [11], is an effective user-session encoder for the re-ranking task.
    The re-ranker reuses E_STAMP without modification or re-derivation; the paper does not test alternative session encoders, so the method's performance is conditional on this encoder's suitability.
  • domain assumption Training the re-ranker only on sequences whose target is in the candidate set does not introduce a systematic bias that inflates test performance.
    Section 5.2 describes this training filter, but the paper does not quantify the coverage of test targets in the top-k set. If the filter changes the difficulty distribution, the learned rank preferences could be biased.
  • ad hoc to paper The candidate generator's ranking carries useful signal for the re-ranker beyond the identity of the items.
    The entire method rests on the hypothesis that rank positions encode latent information (e.g., co-occurrence strength). The ablation in Section 5.5 provides empirical support, but it remains an assumption about the transferability of this signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Two-Stage Session-based Recommendations with Candidate Rank Embeddings." pith.science (2026). https://pith.science/paper/N2FPHBG2

@misc{pith2026190808284,
  author       = {Pith},
  title        = {Pith review of: Two-Stage Session-based Recommendations with Candidate Rank Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N2FPHBG2}},
  note         = {Machine review of arXiv:1908.08284}
}
read the original abstract

Recent advances in Session-based recommender systems have gained attention due to their potential of providing real-time personalized recommendations with high recall, especially when compared to traditional methods like matrix factorization and item-based collaborative filtering. Nowadays, two of the most recent methods are Short-Term Attention/Memory Priority Model for Session-based Recommendation (STAMP) and Neural Attentive Session-based Recommendation (NARM). However, when these two methods were applied in the similar-item recommendation dataset of Zalando (Fashion-Similar), they did not work out-of-the-box compared to a simple Collaborative-Filtering approach. Aiming for improving the similar-item recommendation, we propose to concentrate efforts on enhancing the rank of the few most relevant items from the original recommendations, by employing the information of the session of the user encoded by an attention network. The efficacy of this strategy was confirmed when using a novel Candidate Rank Embedding that encodes the global ranking information of each candidate in the re-ranking process. Experimental results in Fashion-Similar show significant improvements over the baseline on Recall and MRR at 20, as well as improvements in Click Through Rate based on an online test. Additionally, it is important to point out from the evaluation that was performed the potential of this method on the next click prediction problem because when applied to STAMP and NARM, it improves the Recall and MRR at 20 on two publicly available real-world datasets.

Figures

Figures reproduced from arXiv: 1908.08284 by the authors.

Figure 1
Figure 1. ‡e product page o‚ers similar items to the anchor. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. ‡e model architecture: ‡e Candidate Generator [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Our approach with and without Candidate Rank [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Recall@5 of the bests approaches on each dataset [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 9 canonical work pages

  1. [1]

    Fabio Aiolli. 2013. A Preliminary Study on a Recommender System for the Million Songs Dataset Challenge. In IIR

    work page 2013

  2. [2]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ’16). ACM, New York, NY, USA, 191–198. h/t_tps: //doi.org/10.1145/2959100.2959190

    arXiv 2016

  3. [3]

    Gabriel de Souza Pereira Moreira, Felipe Ferreira, and Adilson Marques da Cunha

    work page

  4. [4]

    Robin Devooght and Hugues Bersini. 2016. Collaborative Filtering with Recurrent Neural Networks. CoRR abs/1608.07400 (2016). arXiv:1608.07400 h/t_tp://arxiv. org/abs/1608.07400

    work page Pith review arXiv 2016

  5. [5]

    Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the /T_hirteenth International Conference on Arti/f_icial Intelligence and Statistics (Proceedings of Machine Learning Research), Yee Whye Teh and Mike Ti/t_terington (Eds.), Vol. 9. PMLR, Chia Laguna Resort, Sardinia, Italy, 2...

    work page 2010

  6. [6]

    Bal´a zs Hidasi and Alexandros Karatzoglou. 2018. Recurrent Neural Networks with Top-k Gains for Session-based Recommendations. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM ’18). ACM, New York, NY, USA, 843–852. h/t_tps://doi.org/10.1145/3269206. 3271761

    doi:10.1145/3269206 2018

  7. [7]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). arXiv:1412.6980 h/t_tp://arxiv.org/abs/ 1412.6980

    arXiv 2014

  8. [8]

    Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (Aug. 2009), 30–37. h/t_tps://doi.org/10.1109/MC.2009.263

    doi:10.1109/mc.2009.263 2009

Show all 20 references
  1. [9]

    Rajeev Kumar, BK Verma, and Shyam Sunder Rastogi. 2014. Social popularity based SVD++ recommender system. International Journal of Computer Applica- tions 87 (Jan. 2014)

    2014

  2. [10]

    Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural A/t_tentive Session-based Recommendation. InProceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM ’17). ACM, New York, NY, USA, 1419–1428. h/t_tps://doi.org/10...

    2017

  3. [11]

    Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018. STAMP: Short- Term A/t_tention/Memory Priority Model for Session-based Recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’18). ACM, New York,...

    2018

  4. [12]

    Malte Ludewig and Dietmar Jannach. 2018. Evaluation of Session-based Rec- ommendation Algorithms. CoRR abs/1803.09587 (2018). arXiv:1803.09587 h/t_tp://arxiv.org/abs/1803.09587

    2018 arXiv

  5. [13]

    Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt- /T_hieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the Twenty-Fi/f_th Conference on Uncertainty in Arti/f_icial In- telligence (UAI ’09). AUAI Press, Arlington, Vi...

    2009

  6. [14]

    Vasiliy Rubtsov, Mikhail Kamenshchikov, Ilya Valyaev, Vasiliy Leksin, and Dmitry I. Ignatov. 2018. A Hybrid Two-stage Recommender System for Au- tomatic Playlist Continuation. In Proceedings of the ACM Recommender Systems Challenge 2018 (RecSys Challenge ’18) . ACM, New York, ...

    2018

  7. [15]

    Yong Kiam Tan, Xinxing Xu, and Yong Liu. 2016. Improved Recurrent Neural Networks for Session-based Recommendations. InProceedings of the 1st Workshop on Deep Learning for Recommender Systems (DLRS 2016) . ACM, New York, NY, USA, 17–22. h/t_tps://doi.org/10.1145/2988450.2988452

    2016

  8. [16]

    Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019. Session-based Recommendation with Graph Neural Networks. In Proceedings of /T_he Twenty-/T_hird AAAI Conference on Arti/f_icial Intelligence (AAAI ’19). h/t_tp: //arxiv.org/abs/1811.00855

    2019 arXiv

  9. [17]

    Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016. A Dynamic Recurrent Model for Next Basket Recommendation. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’16). ACM, New York, NY, USA, 729–7...

    2016

  10. [18]

    Bal ´a zs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk

  11. [2015]

    CoRR abs/1511.06939 (2015)

    Session-based Recommendations with Recurrent Neural Networks. CoRR abs/1511.06939 (2015). arXiv:1511.06939 h/t_tp://arxiv.org/abs/1511.06939

    2015 arXiv

  12. [2018]

    In Proceedings of the 3rd Workshop on Deep Learning for Recommender Systems (DLRS 2018)

    News Session-Based Recommendations Using Deep Neural Networks. In Proceedings of the 3rd Workshop on Deep Learning for Recommender Systems (DLRS 2018). ACM, New York, NY, USA, 15–23. h/t_tps://doi.org/10.1145/3270323.3270328

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.