Pith. sign in

REVIEW 4 major objections 4 minor 26 references

Pretrained graph-network item embeddings help only when training data is scarce; at large scale, end-to-end ID embeddings win at lower cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:18 UTC pith:35AGBAUW

load-bearing objection Useful, citable industrial comparison, but the headline 'pretraining not worth it at scale' is undercut by a coverage mismatch in the Market experiment and by the absence of uncertainty estimates. the 4 major comments →

arxiv 2607.26365 v1 pith:35AGBAUW submitted 2026-07-29 cs.IR

Embedding Items at Scale: Comparing GNN-Based and ID-Based Item Embeddings in the Yandex Ecosystem

Sergei Makeev , Artem Matveev , Vladimir Baikalov , Kirill Khrylchenko This is my paper
classification cs.IR
keywords sequential recommendationitem embeddingsgraph neural networkstransformer rankerlarge-scale recommender systemspretrainingID embeddingshashing trick
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to settle a practical question: should a large recommender system spend compute on a separate graph-neural-network stage to pretrain item embeddings, or just learn item IDs end-to-end with the transformer? The authors run the same two-tower transformer ranker with both embedding strategies on three Yandex services. On two large-scale services, end-to-end ID embeddings rank better and cost less; on a small low-resource service, pretrained GNN embeddings win. The paper concludes that pretraining helps in data-scarce settings but provides no worthwhile benefit at scale.

Core claim

On its own terms, the paper's central discovery is that the value of a graph-pretraining stage for item embeddings depends on how much training data the recommender has. With millions of items and billions of interactions, a transformer ranker that learns ID embeddings from scratch outperforms one that uses frozen TwHIN or MultiBiSage GNN embeddings, and fine-tuning the pretrained embeddings does not close the gap. With a small dataset of 15 million interactions, the pretrained TwHIN embeddings consistently outperform ID embeddings. The paper treats this as evidence that pretraining is a low-resource strategy, not a large-scale necessity.

What carries the argument

The central object is the item tower of a two-tower transformer ranking model. The same architecture is trained in the same pretrain-fine-tune pipeline, with only the item embedding source changed: pretrained graph embeddings (TwHIN, a transductive model, or MultiBiSage, an inductive model) versus multihash ID embeddings learned end-to-end. This plug-in comparison, evaluated by offline nDCG and pair accuracy with Wilcoxon-significant folds, is what isolates the embedding strategy.

Load-bearing premise

The large-scale conclusion assumes the GNN embeddings were not handicapped by being trained only for a subset of popular items, with all other items sharing a single vector; if full-catalog GNN embeddings were used, the relative performance could change.

What would settle it

Rerun the same transformer ranker on a large-scale dataset with GNN embeddings computed for the full item catalog rather than only the popular subset. If full-coverage GNN embeddings produce a ranking gain large enough to justify their training cost, the paper's central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Large-scale transformer recommenders can drop the GNN pretraining stage and rely on multihash ID embeddings without sacrificing ranking quality, saving hours of GPU training and infrastructure.
  • Teams with small interaction logs should consider a graph pretraining stage; on the released low-resource dataset it gives consistent nDCG gains.
  • Combining pretrained GNN and ID embeddings adds only marginal gains in the large-scale e-commerce test, not enough to justify pretraining cost.
  • Fine-tuning pretrained GNN embeddings in the transformer does not make them competitive with ID embeddings at scale.
  • The published low-resource dataset and code allow other teams to reproduce the comparison and test whether the conclusion transfers to their own data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's large-scale comparison covers only the most popular items (17% at Yandex Market, 10^5 tracks at Yandex Music); whether full-catalog GNN embeddings would change the ranking is not tested, so the conclusion may not hold for long-tail-heavy catalogs.
  • The reported cost comparison excludes the compute used to train the GNNs and the RAM-GPU transfer overhead; in a full cost-accounting, end-to-end ID embeddings would look even more favorable than the paper's headline numbers.
  • A natural next experiment is to test the same trade-off with cold-start items, where a GNN's ability to use side information might outweigh the ID embedding's advantage.
  • The low-resource result suggests that as training data grows, the marginal value of pretrained embeddings decays; identifying the crossover point (in interactions per item) would give teams a practical rule for when to pretrain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents an industrial case study comparing two item embedding strategies inside transformer-based sequential ranking models at Yandex: pretrained graph neural network embeddings (TwHIN and MultiBiSage) versus multihash item-ID embeddings trained end-to-end with the transformer. Experiments are run on three platforms: Yandex Market, Yandex Music, and a low-resource Yandex Lavka sample for which data and code are publicly released. The headline result is that GNN pretraining helps in low-resource settings, while in large-scale settings pretrained embeddings do not provide benefits that justify their additional cost. The paper also examines fine-tuning pretrained embeddings and combining both embedding types.

Significance. If the conclusions hold, the paper gives practitioners a useful data point on when to invest in graph-based pretraining versus simpler end-to-end ID embeddings. The strengths include a production-scale evaluation on two mature systems, a consistent timestamp-based train/test split, 32-fold CatBoost evaluation with Wilcoxon tests, and the public release of the Lavka dataset and code. However, the large-scale conclusion is currently bounded by item-coverage restrictions in the GNN setup and by incomplete cost accounting. The paper is a credible case study rather than a general proof, and its practical value depends on whether the authors can close or explicitly scope those gaps.

major comments (4)
  1. [§3.1, Table 1; §3 preamble] The paper states that 'we obtain them for the same set of items,' but in the Yandex Market experiment the GNN embeddings are computed only for the top 17% most popular items, with all other items sharing a single vector, while ID embeddings use a multihash table with O(10^6) entries. This gives the ID model distinct representations for tail items by construction, whereas the GNN variant cannot distinguish them. The observed ID advantage over TwHIN/MultiBiSage on retargeting (+1.486% vs +0.943% and +0.651%) could therefore reflect coverage rather than the absence of pretraining value. The Lavka experiment, where all 25,833 items are embedded, shows the opposite result, which is consistent with coverage being the driving factor. The authors should either run the Market comparison with full-catalog GNN embeddings, or add a controlled experiment on the tail in which both methods see the same
  2. [§3.2, Table 2; §4] The Music experiment restricts both embedding types to the same O(10^5) most popular tracks, covering more than 90% of interactions. This makes the comparison internally fair, but it cannot support conclusions about the full catalog. The conclusion in §4 — that 'in large-scale settings pretrained embeddings do not provide benefits' — is broader than the evidence. At minimum, the paper should revise the wording to 'head-of-catalog large-scale models' and either add tail-coverage experiments or explicitly acknowledge that full-catalog conclusions are not established.
  3. [§3.1.1, §4] The cost-related wording is not supported by measured end-to-end cost. Section 3.1.1 reports pretraining times (TwHIN 5 hours, MultiBiSage 52 hours, ID main training 40–70 hours) but explicitly states that 'we do not report training time changes from using GNN embeddings.' The conclusion that pretrained embeddings 'do not provide benefits that justify their additional cost' therefore relies on unmeasured serving and training overhead. Either the end-to-end cost should be reported (including GPU/RAM transfer, SSD reads, and latency) or the conclusion should be softened to 'no quality benefit was observed in our head-of-catalog large-scale settings.'
  4. [§2, §3.3] The paper reports only point estimates with a p<0.01 significance threshold, without confidence intervals or standard deviations. Since the practical conclusion is about magnitudes ('worthwhile'), effect-size uncertainty matters. The Lavka Table 3 reports 'averaged over 10 runs' but gives no standard deviations; the differences at nDCG@20 are within 0.003 and may not be distinguishable from noise. Adding confidence intervals or error bars would materially strengthen the comparison.
minor comments (4)
  1. [§1, TwHIN equation] The TwHIN encoder equation contains a formatting error: 'ENC(·)=𝜃 ·∈R 𝑑' is not well-formed. The decoder expression would also benefit from explicit transposition notation. Please fix the typesetting.
  2. [§3.2] The sentence 'We train embeddings for the sameO(10^5)most popular tracks' is ambiguous about whether this is the item set for both GNN and ID embeddings in the Music experiment. The later ID paragraph suggests that both types use the same 10^5 items; this should be stated explicitly to avoid confusion.
  3. [§3.1.1] The detail about renting GPUs from a third-party provider is not relevant to the scientific content and could be removed or moved to an acknowledgments section.
  4. [Table 3] Please include standard deviations or error bars for the 10-run averages. Also consider reporting the underlying raw metric values rather than only three-decimal averages, especially since the differences are small.

Circularity Check

0 steps flagged

No circularity: the paper is an empirical comparison with an externally testable outcome, not a derivation that reduces to its inputs.

full rationale

No significant circularity. The paper compares pretrained GNN item embeddings (TwHIN, MultiBiSage) with end-to-end ID embeddings by training the same transformer-rank architecture on the same held-out split and measuring ranking metrics. Neither embedding type is defined in terms of the other, and no prediction is a fitted parameter renamed as an independent result. The GNN training objectives (sampled-softmax / binary edge classification) are stated independently of the ranker's fine-tuning objective, and the ranker is evaluated on timestamp-split test data with a Wilcoxon significance test. The self-citations [11] and [12] supply the production ranking setup and production transformer used as the experimental testbed; the paper's comparative claim is not derived from those citations, and the low-resource Lavka experiment is code- and data-released, so it is externally checkable. The asymmetric item coverage at Market (17% of items embedded) and Music (10^5 tracks) is a possible validity threat about whether the large-scale comparison is fair, but it is a confound, not a circularity: it does not make the reported outcome equal to the input by construction. The paper's conclusion is an empirical generalization, and its central measurement is not forced by any self-referential equation or imported uniqueness claim.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No new theoretical entities are introduced. The free parameters listed are design choices that gate the comparison; the axioms are the assumptions needed for the empirical conclusions to generalize beyond Yandex.

free parameters (3)
  • Market GNN item coverage cutoff = 17% most popular items
    Chosen by hand; tail items share a single embedding, which may handicap GNN embeddings in the large-scale comparison (§3.1).
  • Music GNN track coverage cutoff = 10^5 most popular tracks
    Covers more than 90% of interactions but may disadvantage GNN embeddings on tail tracks (§3.2).
  • MultiBiSage neighbor sample count = 50 neighbors per node
    Taken from Gurukar et al. [9] without re-tuning; affects GNN embedding quality (§3.1).
axioms (3)
  • domain assumption Timestamp-based splitting of training and test data prevents leakage and gives unbiased evaluation.
    Used for all experiments; if user interactions are not independent over time, the split could favor one embedding type (§3).
  • domain assumption The CatBoost production ranker with and without the transformer score is an unbiased measurement of embedding quality.
    The conclusion relies on the relative nDCG differences computed through this pipeline (§2, §3.1).
  • standard math The Wilcoxon signed-rank test over 32 folds is an appropriate significance test for the reported metric differences.
    A standard non-parametric test; however, the 32 folds may not be strictly independent because they come from the same underlying dataset.

pith-pipeline@v1.3.0-daily-deepseek · 7417 in / 9917 out tokens · 90272 ms · 2026-08-01T17:18:49.212958+00:00 · methodology

0 comments
read the original abstract

Transformer-based sequential recommendation models, which process sequences of user-item interactions, rely heavily on the item embedding strategy. Existing approaches either use pretrained item embeddings or learn them end-to-end with the transformer. To the best of our knowledge, no prior work has compared these options from both cost and quality perspectives in a large-scale industrial setting. This paper is a case study that compares pretrained industrial graph neural network item embeddings with end-to-end trainable item embeddings across two mature production recommendation systems at Yandex: Yandex Market and Yandex Music. We additionally evaluate both approaches on a low-resource dataset sampled from Yandex Lavka production logs, for which both the data and code are publicly available for demonstration purposes. Our results show that a separate pretraining stage helps when training data is limited, but provides no worthwhile benefit for large-scale models trained on extensive datasets.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 5 linked inside Pith

  1. [1]

    Prabhat Agarwal, Minhazul Islam Sk, Nikil Pancha, Kurchi Subhra Hazra, Jiajing Xu, and Chuck Rosenberg. 2024. OmniSearchSage: Multi-Task Multi-Entity Embeddings for Pinterest Search. InCompanion Proceedings of the ACM on Web Conference 2024. 121–130

    2024

  2. [2]

    Aijun Bai, Rolf Jagerman, Zhen Qin, Le Yan, Pratyush Kar, Bing-Rong Lin, Xuan- hui Wang, Michael Bendersky, and Marc Najork. 2023. Regression compatible listwise objectives for calibrated ranking with binary relevance. InProceedings of the 32nd ACM International Conference on Information and Knowledge Man- agement. 4502–4508

    2023

  3. [3]

    Marjan Celikik, Ana Peleteiro Ramallo, and Jacek Wasilewski. 2022. Reusable self- attention recommender systems in fashion industry applications. InProceedings of the 16th ACM Conference on Recommender Systems. 448–451

    2022

  4. [4]

    Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al . 2025. Longer: Scaling up long sequence modeling in industrial recommenders. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 247–256

    2025

  5. [5]

    Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior sequence transformer for e-commerce recommendation in alibaba. InProceedings of the 1st international workshop on deep learning practice for high-dimensional sparse data. 1–4

    2019

  6. [6]

    Xiangyi Chen, Kousik Rajesh, Matthew Lawhon, Zelun Wang, Hanyu Li, Haomiao Li, Saurabh Vishwas Joshi, Pong Eksombatchai, Jaewon Yang, Yi-Ping Hsu, et al

  7. [7]

    Benjamin Coleman, Wang-Cheng Kang, Matthew Fahrbach, Ruoxi Wang, Lichan Hong, Ed Chi, and Derek Cheng. 2023. Unified Embedding: Battle-tested feature representations for web-scale ML systems.Advances in Neural Information Processing Systems36 (2023), 56234–56255

    2023

  8. [8]

    Ahmed El-Kishky, Thomas Markovich, Serim Park, Chetan Verma, Baekjin Kim, Ramy Eskander, Yury Malkov, Frank Portman, Sofía Samaniego, Ying Xiao, et al

  9. [9]

    Saket Gurukar, Nikil Pancha, Andrew Zhai, Eric Kim, Samson Hu, Srinivasan Parthasarathy, Charles Rosenberg, and Jure Leskovec. 2022. Multibisage: A web- scale recommendation system using multiple bipartite graphs at pinterest.arXiv preprint arXiv:2205.10666(2022)

    Pith/arXiv arXiv 2022

  10. [10]

    William L Hamilton, Rex Ying, and Jure Leskovec. 2017. Representation learning on graphs: Methods and applications.arXiv preprint arXiv:1709.05584(2017)

    Pith/arXiv arXiv 2017

  11. [11]

    Kirill Khrylchenko and Alexander Fritzler. 2023. Personalized transformer-based ranking for e-commerce at yandex.arXiv preprint arXiv:2310.03481(2023). Comparing GNN-Based and ID-Based Item Embeddings in the Yandex Ecosystem

    Pith/arXiv arXiv 2023

  12. [12]

    Kirill Khrylchenko, Artem Matveev, Sergei Makeev, and Vladimir Baikalov. 2025. Scaling recommender transformers to one billion parameters.arXiv preprint arXiv:2507.15994(2025)

    arXiv 2025

  13. [13]

    Chi Liu, Jiangxia Cao, Rui Huang, Kai Zheng, Qiang Luo, Kun Gai, and Guorui Zhou. 2024. KuaiFormer: Transformer-Based Retrieval at Kuaishou.arXiv preprint arXiv:2411.10057(2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinner- former: Sequence modeling for user representation at pinterest. InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 3702–3712

    2022

  15. [15]

    Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Doro- gush, and Andrey Gulin. 2018. CatBoost: unbiased boosting with categorical features.Advances in neural information processing systems31 (2018)

    2018

  16. [16]

    Dan Tito Svenstrup, Jonas Hansen, and Ole Winther. 2017. Hash embeddings for efficient word representations.Advances in neural information processing systems30 (2017)

    2017

  17. [17]

    Tian Wang, Yuri M Brovman, and Sriganesh Madhvanath. 2021. Personal- ized embedding-based e-commerce recommendations at ebay.arXiv preprint arXiv:2102.06156(2021)

    Pith/arXiv arXiv 2021

  18. [18]

    Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. 2009. Feature hashing for large scale multitask learning. InProceedings of the 26th annual international conference on machine learning. 1113–1120

    2009

  19. [19]

    Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and Andrew Zhai. 2023. Transact: Transformer-based realtime user action model for recommendation at pinterest. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5249–5259

    2023

  20. [20]

    Xue Xia, Saurabh Joshi, Kousik Rajesh, Kangnan Li, Yangyi Lu, Nikil Pancha, Dhruvil Badani, Jiajing Xu, and Pong Eksombatchai. 2025. TransAct V2: Lifelong User Action Sequence Modeling on Pinterest Recommendation. InProceedings of the 34th ACM International Conference on Information and Knowledge Manage- ment. 6881–6882

    2025

  21. [21]

    Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaom- ing Wang, Taibai Xu, and Ed H Chi. 2020. Mixed negative sampling for learning two-tower neural networks in recommendations. InCompanion proceedings of the web conference 2020. 441–447

    2020

  22. [22]

    Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. 2019. Sampling-bias-corrected neural modeling for large corpus item recommendations. InProceedings of the 13th ACM conference on recommender systems. 269–277

    2019

  23. [23]

    Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018. Graph convolutional neural networks for web-scale recommender systems. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 974–983

    2018

  24. [24]

    Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, and Aixin Sun. 2025. OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender.arXiv preprint arXiv:2510.26104(2025)

    arXiv 2025

  25. [2022]

    InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining

    Twhin: Embedding the twitter heterogeneous information network for personalized recommendation. InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 2842–2850

  26. [2025]

    InProceedings of the Nineteenth ACM Conference on Recommender Systems

    Pinfm: foundation model for user activity sequences at a billion-scale visual discovery platform. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 381–390