REVIEW 3 major objections 5 minor 2 cited by
Yambda-5B -- A Large-Scale Multi-modal Dataset for Ranking And Retrieval
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Yambda-5B, a public dataset of 4.79 billion music interactions, labels every action as organic or recommendation-driven.
desk verdict A genuinely useful industrial-scale music dataset, but its headline is_organic flag lacks any operational definition or validation, and the results section contradicts itself. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing artifact is the dataset itself, and within it the per-event is_organic binary flag, a label that marks each interaction as organic (initiated by the user independently) or recommendation-driven (surfaced by the platform's algorithms). The flag is what the paper calls a critical aspect of Yambda, because it is meant to let researchers disentangle platform logging policies from recommendation-influenced behavior and to enable study of algorithmic feedback loops. Around the flag, the paper builds a Global Temporal Split evaluation protocol: 300 days of training data, a 30-minute gap that mimics the latency between model training and deployment, and one test day with all model parameters and user states frozen. The audio embeddings for most tracks, generated by a convolutional neural network trained in a contrastive manner, supply the multimodal component that supports content-based and semantic-ID-based recommendation approaches.
What would settle it
Take a random sample of events and check the is_organic flag against the service's internal logs of which screen or recommendation list produced each event. If a noticeable share of events labeled organic actually came from recommendation surfaces, or labeled recommendation-driven came from search or library browsing, the flag's accuracy is refuted and the dataset's main claimed advantage fails.
Extended reading notes
Core claim
The paper presents Yambda-5B as a large-scale multimodal dataset for ranking and retrieval, containing 4.79 billion events across five interaction types — listen, like, dislike, unlike, and undislike — logged from 1,000,000 users and 9,390,623 tracks over roughly 11 months. The authors claim this is one of the largest open datasets of music listening interactions, with a median user history of about 3,076 listens, a length comparable to the context window of modern large language models. Its three distinguishing features are high-fidelity audio embeddings produced by a contrastively trained convolutional network, a binary is_organic flag on every event, and a Global Temporal Split evaluation protocol that preserves temporal ordering to prevent data leakage. Benchmark results on the Listen+ and Like feedback setups show that ItemKNN and SASRec lead ranking metrics for implicit feedback across dataset scales, DecayPop leads the explicit Like scenario, and ItemKNN and SANSA become computationally intractable at the full 5-billion-event scale. The intended contribution is a ready-to-use, industrial-scale benchmark that lets academic research validate scaling behavior, extreme sparsity, and sequential modeling under conditions closer to production.
Load-bearing premise
The paper's most distinctive feature, the is_organic flag, comes with no explanation of how it was assigned and no check of whether it is correct, so the dataset's central claimed advantage rests on the assumption that this binary label truly separates user-initiated actions from recommendation-driven ones.
Editorial extensions
If this is right
- Researchers can train recommender models on billions of interactions and millions of items, testing scaling behavior and extreme sparsity that small academic datasets cannot support.
- The is_organic flag allows training and evaluation to be separated by interaction origin, enabling direct comparisons of models on organic versus recommendation-driven behavior.
- The Global Temporal Split protocol with a 30-minute gap gives a leakage-safe evaluation scheme that approximates daily offline model refreshes in production systems.
- The benchmark tables indicate that sequence-aware models like SASRec and neighborhood methods like ItemKNN dominate implicit-feedback ranking, while a time-decayed popularity baseline leads explicit-feedback ranking.
- The three release sizes (50M, 500M, 5B) let researchers trade computational cost against scale and observe how baseline performance changes as data grows.
Reading between the lines
- If the flag is trustworthy, the dataset becomes a testbed for measuring recommendation influence: researchers could compare engagement decay, consumption diversity, and repeat behavior between organic and recommended tracks, as the paper's future-work section anticipates.
- A separate direction would be to train a classifier to predict is_organic from user and item features; high predictability would suggest the flag encodes systematic behavioral differences, while low predictability would point to label noise.
- The 5-second timestamp rounding and 1-percent played-ratio granularity are fine for sequence models, but studies of short-duration skip behavior would need to confirm that rounding does not destroy the relevant signal.
- The reported intractability of ItemKNN and SANSA at full scale gives a concrete computational baseline for judging whether new approximate algorithms actually scale to billions of events.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces Yambda-5B, a large-scale dataset of music listening interactions from the Yandex Music platform, containing 4.79 billion events from 1 million users and 9.39 million tracks over an 11-month period. In addition to implicit feedback (listens) and explicit feedback (likes, dislikes, unlikes, undislikes), the dataset provides a binary is_organic flag claimed to distinguish organic from recommendation-driven events, audio embeddings for tracks, and timestamps at 5-second granularity. The paper also proposes a Global Temporal Split evaluation protocol (300 days train, 30-minute gap, 1-day test) and reports benchmark results for several baselines (MostPop, DecayPop, ItemKNN, iALS, BPR, SANSA, SASRec) on two tasks (Listen+ and Like) at three dataset scales. The dataset is publicly released on Hugging Face in flat and sequential formats, with subsampled variants.
Significance. If the is_organic flag is reliable, Yambda-5B would be a unique community resource: it is among the largest publicly available interaction datasets, includes both implicit and explicit feedback, provides a temporal split that respects causality, and ships code and audio embeddings. The paper's main strengths are the scale of the data, the explicit release artifacts, and the simple GTS protocol that others can adopt. However, the central novelty—the is_organic flag—is currently ungrounded, and the benchmark claims contain a verifiable inconsistency. The dataset may still be useful without the flag, but the paper's stated contribution and future research directions (e.g., feedback-loop analysis) depend crucially on its validity.
major comments (3)
- [Sec. 3.1 / Table 2] The is_organic flag, described as 'a key distinguishing feature' and 'a critical aspect of Yambda,' is never defined. The paper does not state the logging rule, the UI surface or attribution method, any manual audit, or any error analysis. Table 2 reports recommendation-driven ratios (e.g., 48.74% of listens, 5.01% of unlikes) as if they were ground truth, but without a definition the reader cannot judge whether these numbers reflect exposure to a recommender surface or some other heuristic. Because this flag is the primary differentiator from existing datasets and the basis for the planned organic-vs-algorithmic analyses in Section 5, the authors need to document how it was computed and provide at least a basic validation (e.g., agreement with known platform behavior, audit of a sample). Without this, the dataset's central claim is unsubstantiated.
- [Sec. 4.3 / Table 6] The text claims that 'the top-2 algorithms by ranking metrics in the Listen+ scenario consistently included ItemKNN and SASRec across all dataset scales.' This is contradicted by Table 6, where ItemKNN results are absent for Yambda-5B ('—'). At Yambda-5B, the top-2 by NDCG@10 are SASRec (0.0647) and BPR (0.0408), with ItemKNN unavailable. The statement needs to be corrected or rephrased to reflect the actual rows reported; as written, it gives an inaccurate summary of the results.
- [Sec. 4.1 / Sec. 4.3] The benchmark conclusions are based on a single test day (with a 30-minute gap and frozen user states), and the paper draws general claims such as 'conventional collaborative filtering methods exhibit degraded performance when applied to scenarios requiring real-time interaction processing.' With no variance estimation or multiple test periods, this is a strong generalization from a single realization. The authors should either run the evaluation over several test days (and report mean/standard deviation), or temper the conclusions to acknowledge that the observed rankings may not be stable. This is especially important because the GTS protocol is one of the paper's contributions, and its reliability is not demonstrated.
minor comments (5)
- [Sec. 3.1] The sentence 'Yambda provided neural embeddings' should be 'Yambda provides neural embeddings'; also the paper does not specify the embedding dimensionality, the model architecture (beyond 'CNN trained in contrastive manner'), or the fraction of tracks for which embeddings are available ('most tracks' is unquantified).
- [Sec. 3.2] The code repository is mentioned as 'our GitHub repository' but no URL is provided; the dataset URL appears only in a footnote. Please make both links explicit in Section 3.2.
- [Sec. 2] In the list of 'three critical dataset features,' the third item is 'Global Temporal Split,' which is an evaluation protocol rather than a dataset feature. This conflates dataset properties with benchmarking methodology; consider rephrasing as, for example, 'high-resolution timestamps enabling temporal splits.'
- [Sec. 4.3] The sentence 'Metrics for ItemKNN and SANSA are unavailable at larger dataset scales' is vague because Table 6 shows ItemKNN results for Yambda-500M; it should specify that they are unavailable at the Yambda-5B scale only. The phrase 'due to their computational intractability within practical time constraints' would also benefit from a concrete description of the compute environment.
- [Abstract / Sec. 5] The paper calls Yambda a 'multi-modal' dataset, but the only content modality is audio embeddings (plus structured metadata). Please clarify whether this constitutes multi-modality in the intended sense, or consider using 'multimodal' only if additional modalities (e.g., text, images) are actually provided.
Circularity Check
No circularity: Yambda-5B is a resource paper whose claims are descriptive statistics and a fixed evaluation protocol, with no fitted input renamed as a prediction.
full rationale
The paper's central claims are descriptive: interaction counts, item/user statistics, and an is_organic flag included as a dataset field. No quantity is derived from the flag, and no prediction is presented as if it were derived from first principles. The benchmark section uses a fixed Global Temporal Split (300-day train, 30-minute gap, 1-day test) that was not constructed to make any particular baseline win; hyperparameters are tuned on a held-out validation day, which is standard model selection rather than fitting the target result. The reported numbers in Tables 6 and 7 are empirical measurements on the released data. The unvalidated is_organic flag is a legitimate data-quality/validity concern, but it is not a circular step because the paper does not use the flag to predict itself or to justify any derived quantity. There is also no load-bearing self-citation: the authors cite no prior work by themselves as evidence for the dataset's utility. One internal inconsistency (Section 4.3 claims ItemKNN and SASRec were top-2 across all scales, while Table 6 omits ItemKNN at Yambda-5B) is a reporting error, not circularity.
Assumptions & free parameters
free parameters (4)
- GTS evaluation schedule (300d train / 30m gap / 1d test) =
n/a (design choice)
- Listen+ positive threshold (50% of track duration) =
50%
- User activity filter (>=10 actions in first 10 months, >=1 in the 11th) =
n/a (design choice)
- Timestamp rounding to 5 seconds =
5s
assumptions (4)
- domain assumption The is_organic flag reliably reflects whether an interaction was initiated by the user or driven by a recommendation.
- domain assumption The user and item identifiers, after anonymization, are consistent across the observation period.
- domain assumption Audio embeddings are available for 'most' tracks and are informative for content-based recommendation.
- ad hoc to paper The 50% listening threshold is a valid binarization of implicit feedback.
Cite this review
Pith. "Pith review of Yambda-5B -- A Large-Scale Multi-modal Dataset for Ranking And Retrieval." pith.science (2026). https://pith.science/paper/HTMZWNJN
@misc{pith2026250522238,
author = {Pith},
title = {Pith review of: Yambda-5B -- A Large-Scale Multi-modal Dataset for Ranking And Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/HTMZWNJN}},
note = {Machine review of arXiv:2505.22238}
}
read the original abstract
We present Yambda-5B, a large-scale open dataset sourced from the Yandex Music streaming platform. Yambda-5B contains 4.79 billion user-item interactions from 1 million users across 9.39 million tracks. The dataset includes two primary types of interactions: implicit feedback (listening events) and explicit feedback (likes, dislikes, unlikes and undislikes). In addition, we provide audio embeddings for most tracks, generated by a convolutional neural network trained on audio spectrograms. A key distinguishing feature of Yambda-5B is the inclusion of the is_organic flag, which separates organic user actions from recommendation-driven events. This distinction is critical for developing and evaluating machine learning algorithms, as Yandex Music relies on recommender systems to personalize track selection for users. To support rigorous benchmarking, we introduce an evaluation protocol based on a Global Temporal Split, allowing recommendation algorithms to be assessed in conditions that closely mirror real-world use. We report benchmark results for standard baselines (ItemKNN, iALS) and advanced models (SANSA, SASRec) using a variety of evaluation metrics. By releasing Yambda-5B to the community, we aim to provide a readily accessible, industrial-scale resource to advance research, foster innovation, and promote reproducible results in recommender systems.
Figures
Forward citations
Cited by 2 Pith papers
-
Session-Level Optimization for Large-Scale Retrieval using REINFORCE with Multi-Step Off-Policy Correction
Off-policy REINFORCE with up to 10 importance-weight factors raises estimated discounted session reward over next-item and positive-only baselines in offline evaluation on the Yambda-5B dataset.
-
A Survey of Real-World Recommender Systems: Challenges, Constraints, and Industrial Perspectives
A survey of A/B-validated industrial recommender systems, split into transaction-oriented and content-oriented categories, with a discussion of the academia-industry gap.
Reference graph
Works this paper leans on
-
[1]
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization frame- work. InProceedings of the 25th ACM SIGKDD international conference on knowl- edge discovery & data mining. 2623–2631
2019
-
[2]
Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. 2022. Understanding scaling laws for recommendation models. arXiv preprint arXiv:2208.08489(2022)
arXiv 2022
-
[3]
James Bennett and Stan Lanning. 2007. The netflix prize. (2007)
2007
-
[4]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901
2020
-
[5]
Ching-Wei Chen, Paul Lamere, Markus Schedl, and Hamed Zamani. 2018. Recsys challenge 2018: Automatic music playlist continuation. InProceedings of the 12th ACM Conference on Recommender Systems. 527–528
2018
-
[6]
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555(2014)
arXiv 2014
-
[7]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929(2020)
arXiv 2020
-
[8]
GroupLens. [n. d.]. MovieLens. https://grouplens.org/datasets/movielens/. Ac- cessed: (2025-05-28)
work page 2025
Show all 42 references
-
[9]
F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context.Acm transactions on interactive intelligent systems (tiis)5, 4 (2015), 1–19
2015
-
[10]
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk
-
[11]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural computation9, 8 (1997), 1735–1780
1997
-
[12]
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Jo- hannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556(2022)
2022 arXiv
-
[13]
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley
-
[14]
Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In2008 Eighth IEEE international conference on data mining. Ieee, 263–272
2008
-
[15]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. InProceedings of the 22nd ACM international conference on Information & Knowledge Management. 2333–2338
2013
-
[16]
Yitong Ji, Aixin Sun, Jie Zhang, and Chenliang Li. 2020. A re-visit of the popularity baseline in recommender systems. InProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1749– 1752
2020
-
[17]
Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In2018 IEEE international conference on data mining (ICDM). IEEE, 197–206
2018
-
[18]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020)
2020 arXiv
-
[19]
Criteo AI Lab. [n. d.]. Criteo 1TB Click Logs Dataset. https://ailab.criteo.com/ download-criteo-1tb-click-logs-dataset. Accessed: (2025-05-12)
2025
-
[20]
Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. InProceedings of the 7th ACM conference on Recommender systems. 165–172
2013
-
[21]
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel
-
[22]
Marta Moscati, Emilia Parada-Cabaleiro, Yashar Deldjoo, Eva Zangerle, and Markus Schedl. 2022. Music4All-Onion–A Large-Scale Multi-faceted Content- Centric Music Recommendation Dataset. InProceedings of the 31st ACM Interna- tional Conference on Information & Knowledge Managem...
2022
-
[23]
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural...
2019
-
[24]
InProceedings of the 38th international ACM SIGIR conference on research and development in information retrieval
Image-based recommendations on styles and substitutes. InProceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 43–52
-
[25]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt- Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback.arXiv preprint arXiv:1205.2618(2012)
2012 arXiv
-
[26]
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. InProceedings of the 10th international conference on World Wide Web. 285–295
2001
-
[27]
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al
-
[28]
Markus Schedl, Stefan Brandl, Oleg Lesota, Emilia Parada-Cabaleiro, David Penz, and Navid Rekabsaz. 2022. LFM-2b: A dataset of enriched music listening events for recommender systems research and fairness analysis. InProceedings of the 2022 Conference on Human Information Inte...
2022
-
[29]
Janne Spijkervet and John Ashley Burgoyne. 2021. Contrastive learning of musical representations.arXiv preprint arXiv:2103.09410(2021)
2021 arXiv
-
[30]
Martin Spišák, Radek Bartyzal, Antonín Hoskovec, Ladislav Peska, and Miroslav T˘uma. 2023. Scalable approximate nonsymmetric autoencoder for collaborative filtering. InProceedings of the 17th ACM Conference on Recommender Systems. 763–770
2023
-
[31]
Markus Schedl. 2016. The lfm-1b dataset for music retrieval and recommendation. InProceedings of the 2016 ACM on international conference on multimedia retrieval. 103–110
2016
-
[32]
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang
-
[33]
Gabriel Vigliensoni and Ichiro Fujinaga. 2017. The music listening histories dataset.. InISMIR. 96–102
2017
-
[34]
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. InProceedings of the ADKDD’17. 1–7
2017
-
[35]
Harald Steck. 2019. Embarrassingly shallow autoencoders for sparse data. InThe World Wide Web Conference. 3251–3257
2019
-
[36]
Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, et al. 2024. Wukong: Towards a scaling law for large-scale recommendation.arXiv preprint arXiv:2403.02545(2024)
2024 arXiv
-
[37]
Gaowei Zhang, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji- Rong Wen. 2024. Scaling law of large sequential recommendation models. In Proceedings of the 18th ACM Conference on Recommender Systems. 444–453
2024
-
[40]
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. InProceedings of the web conference 2021. 1785–1797
2021
-
[2015]
Session-based recommendations with recurrent neural networks.arXiv preprint arXiv:1511.06939(2015)
2015 arXiv
-
[2019]
InProceedings of the 28th ACM international conference on information and knowledge management
BERT4Rec: Sequential recommendation with bidirectional encoder rep- resentations from transformer. InProceedings of the 28th ACM international conference on information and knowledge management. 1441–1450
-
[2023]
Recommender systems with generative retrieval.Advances in Neural Information Processing Systems36 (2023), 10299–10315
2023
-
[2024]
Bridging Language and Items for Retrieval and Recommendation.arXiv preprint arXiv:2403.03952(2024)
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.