REVIEW 4 major objections 5 minor 26 references
Contextualizing Spotify's Audiobook List Recommendations with Descriptive Shelves
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Descriptive shelves generated from LLM-enriched metadata improve audiobook engagement and discovery in A/B tests.
desk verdict A plausible and honest industrial pipeline for descriptive shelves, but Table 1's gains are confounded by concurrent UI changes and lack statistical support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the descriptive-shelf pipeline. It uses a hand-built taxonomy of ten descriptor types to keep LLM output structured and human-readable; each catalog item is enriched once, so no per-user LLM requests are needed at serving time. Descriptor ranking scores each distinct descriptor by predicted user affinity, and a greedy diversification step removes descriptors whose content embeddings are too similar, so the displayed shelf titles cover different topics. Item ranking and filtering then places under each shelf only candidate audiobooks that carry the matching descriptors, ordered by the recommendation model's scores. Handcrafted templates combine descriptor types—for example mood plus genre into 'Emotional Romance'—to avoid titles that are too vague or too narrow.
What would settle it
A decisive test holds all interface factors fixed—same home-page section, same 'Audiobooks for you' label, same number of shelves, same candidate recommender—and swaps only the shelf titles: LLM-generated descriptive titles versus editor-chosen titles. If the 86.96% streams-per-impression gain and the 627.27% unique-audiobooks-shown gain shrink to near zero, the central claim is refuted; if they persist, the descriptive-shelf pipeline is the driver.
Extended reading notes
Core claim
At the paper's center is the claim that descriptive shelves generated from LLM-enriched metadata outperform editor-curated audiobook shelves when users have already signaled audiobook intent. The pipeline works in four stages: a large language model (LLM) extracts ten descriptor types—genres, themes, characters, moods, settings, personal situations, story tropes, target audiences, objective-based descriptors, and named entities—grounded in the item's own title, authors, description, and genre labels; a descriptor-ranking and diversification step chooses shelf titles by predicted affinity and embedding similarity; an item-ranking and filtering step keeps candidate items that match each shelf's descriptors and orders them by recommender score; and a decoration step displays several shelves under an 'Audiobooks for you' label. In the second A/B test, this presentation raised clicks per impression by 35.25%, streams per impression by 86.96%, unique audiobooks shown by 627.27%, and unique audiobooks interacted with by 804.56% over editor-curated shelves. The earlier single-shelf test on the main surface had improved discovery but reduced engagement, a result the authors use to explain why the later test showed multiple shelves in an audiobook-intent context. The authors take the second test as evidence that thematic shelf titles let users choose which topic to explore, and that broader exposure helps content creators reach a more diverse audience.
Load-bearing premise
The load-bearing premise is that the second test's large gains come from the descriptive-shelf method itself, not from the concurrent changes—adding the 'Audiobooks for you' label, showing several shelves in a home-page section for users who selected the audiobook filter, and comparing against editor-curated shelves instead of the earlier generic row.
Editorial extensions
If this is right
- The method removes a manual bottleneck: shelf titles can be generated and personalized at catalog scale instead of being written by editors.
- Discovery widens: because shelves are topic-diverse and personalized, users are exposed to and interact with far more distinct audiobooks than with editor-curated rows.
- The approach is viable in metadata-cold-start domains, since the LLM enrichment relies only on title, author, description, and genre labels.
- A multi-shelf presentation with a personalization cue appears to be the condition under which descriptive shelves beat the baseline; a single unlabeled shelf does not.
- The same candidate recommender can be reused unchanged; the gains come from how its output is packaged and described.
Reading between the lines
- Editorial inference: because the second A/B test changed the label, the surface, the number of shelves, and the control condition at once, the reported gains are best read as the joint effect of the new presentation; a decomposition experiment is the obvious next step.
- Editorial inference: the same descriptor-enrichment and shelf-ranking stages should transfer to other catalogs with sparse metadata, such as podcasts or short-form video, as long as a domain taxonomy is rebuilt from that domain's search behavior.
- Editorial inference: the 627% jump in unique audiobooks shown suggests the effect may be concentrated in long-tail discovery; a future analysis could check whether niche titles account for most of the new impressions.
- Editorial inference: if a similar presentation moved to a general surface without audiobook intent, the first test implies engagement might drop; intent filtering could be a necessary part of the recipe rather than a convenience.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a pipeline for generating contextualized audiobook list recommendations called 'descriptive shelves.' The pipeline uses LLMs to enrich sparse audiobook metadata with descriptors from a manually constructed ten-type taxonomy (genres, themes, moods, settings, etc.), ranks and diversifies these descriptors for each user via a greedy similarity-based approach, then populates the resulting shelves with items from a two-tower recommender's candidate set. The authors report two production A/B tests. The first, run on the main home surface with a fixed candidate set, showed increased discovery but decreased engagement for descriptive shelves relative to the existing 'Audiobooks for you' shelf. The second, run in an audiobook-intent subfeed with an added personalization label and multiple shelves, is reported as showing large relative gains over editor-curated shelves: impression-to-click +35.25%, impression-to-stream +86.96%, distinct audiobooks impressed +627.27%, and distinct audiobooks interacted +804.56%. The central claim is that this pipeline improves user engagement and audiobook discovery.
Significance. If the reported second A/B test result were causally attributable to the descriptive-shelf pipeline, the work would be a useful industrial demonstration that LLM-enriched metadata can support interpretable, diverse list recommendations in a cold-start domain, with clear benefits for catalog exploration. The paper's strengths include grounding the taxonomy in real user search behavior and forum requests, using LLM extraction that is grounded in item metadata, and reporting production A/B results including the honest account of the first test's engagement decrease. The paper is also transparent about the pipeline's free parameters (number of shelves, descriptor types, templates, ranking function, and diversification threshold), and it does not claim to derive outcomes from a fitted model. However, the empirical evidence as currently presented does not establish that the descriptive-shelf method itself drives the reported gains, and the absence of statistical detail makes the magnitude of the claims difficult to assess.
major comments (4)
- [Section 4, Second A/B test] The reported Table 1 gains cannot be attributed to the descriptive-shelf method because the second A/B test changed multiple variables simultaneously relative to the first test: an added 'Audiobooks for you' label, a move to the audiobook-intent subfeed with a pre-filtered user population, the display of multiple shelves instead of a single shelf, and a switch of the control condition from a generic algorithmic row to editor-curated shelves across 17 fixed categories. Any of these changes could explain the increases in i2c, i2s, and distinct-audiobook counts. To support the paper's central claim, the authors need an experiment that isolates the descriptive-shelf pipeline from these layout, label, placement, and control-condition changes, or an analysis demonstrating that the gains are robust across conditions that vary these factors.
- [Section 4, First A/B test] The first A/B test contradicts the engagement claim: with the candidate set held constant and only the shelf title generation changed, descriptive shelves underperformed the control on engagement metrics. The paper offers plausible hypotheses (lack of a personalization cue, single-shelf limitation, cold-start users), but provides no test or analysis supporting these hypotheses. The unresolved contradiction means the second test's engagement gains cannot be cleanly interpreted as evidence for the method; the authors should either provide supporting analyses for their hypotheses or report a re-analysis of the first test that reconciles the two outcomes.
- [Section 4, Table 1] The statistical evidence is under-specified: no confidence intervals, significance tests, sample sizes, or user-level variance are reported for the four metrics, and only relative percentage changes are given without absolute baseline values. Large relative changes (e.g., +804.56%) can be driven by small absolute counts, and no information is provided about how many users, impressions, or interactions underlie them. The paper should report absolute metric values, uncertainty estimates, and significance tests, and should state whether the metrics were evaluated at the user or session level.
- [Section 3.1, Descriptor Generation] The claim that 'manual and automatic evaluations showed high accuracy in the task' is not supported by any experimental detail: no evaluation set, no metric, no baseline, and no error analysis are provided. Since descriptor quality is the upstream input to shelf generation, the paper should include at least a summary of the evaluation methodology and results, or explicitly mark this component as a design choice rather than a validated contribution.
minor comments (5)
- [Abstract] The abstract states that 'A/B tests show improvements in user engagement and audiobook discovery metrics,' which is misleading because the first A/B test reported in Section 4 showed an engagement decrease; the abstract should specify that the improvements are from the second A/B test.
- [Section 3.2, Figure 1] The caption of Figure 1 mentions 'N=2,' but the number of displayed shelves N is never formally defined in the text, and the choice of N is not discussed or varied in the experiments; please clarify how N is set in the deployed tests.
- [Section 4, Table 1 caption] The caption defines i2c and i2s but does not state the unit of analysis (e.g., user-level or impression-level aggregates), nor does it say whether '# impressed' and '# interacted' are per-user averages or total counts; please specify.
- [Section 3.1] The phrase 'characters of a book might serve as good descriptors' contains a minor typo ('characters' should be 'characters' in some contexts, but here it is actually intended as 'character descriptions'); more substantively, the taxonomy types could be reformatted as a numbered list for clarity.
- [Section 2] The reference to the candidate recommender as 'the two-tower model described in [4]' cites the authors' own prior paper; while this is acceptable, the paper should clarify whether the candidate recommender is fixed for both A/B tests and whether its outputs were held constant within each test.
Circularity Check
No circular derivation: the A/B tests measure external engagement against a control, and the only self-citation (the two-tower recommender from ref. [4]) is a non-load-bearing input.
full rationale
The paper's claimed result is an empirical A/B comparison: descriptive shelves vs. an editor-curated control, with engagement and discovery metrics measured on users. The shelf-generation pipeline (descriptor extraction via LLM, descriptor ranking/diversification, item ranking/filtering) is not fitted to the outcome metrics; the shelf titles are derived from LLM-enriched metadata and the candidate recommender, and the evaluation metrics (i2c, i2s, #impressed, #interacted) are external user behaviors. No equation in the paper defines the outcome in terms of the method's own outputs, and no parameter is fit to a subset of the A/B data and then 'predicted' on the same data. The only notable self-reference is the candidate recommender, cited as the two-tower model in [4], which has overlapping authorship with the current paper; however, that model is a fixed input to the shelf-generation pipeline rather than the thing being evaluated or derived, so it is not load-bearing for the central claim. The more serious concern is a causal-attribution confound: the second A/B test changed the 'Audiobooks for you' label, moved to an audiobook-intent subfeed, displayed multiple shelves, and switched the control to editor-curated shelves, so the reported gains (+35.25% i2c, +86.96% i2s, +627.27% #impressed, +804.56% #interacted) may be driven by interface/placement or control choice. That is a genuine experimental-design limitation, but it is not circularity: the outcome is still an externally measured behavior, not a quantity defined by construction. Accordingly, the circularity score is minimal, reflecting only the presence of a minor non-load-bearing self-citation.
Assumptions & free parameters
free parameters (5)
- Number of displayed shelves N =
2 (from Figure 1 caption)
- Descriptor extraction set (10 taxonomy types) =
10 types
- Descriptor combination templates =
handcrafted, e.g., <Mood>+<Genre>
- Descriptor ranking function =
not specified
- Greedy diversification similarity threshold =
not specified
assumptions (5)
- domain assumption The taxonomy derived from Spotify search queries and Reddit /r/booksuggestions is representative of audiobook users' interests.
- domain assumption LLM descriptor extraction is sufficiently accurate and grounded in the available metadata.
- domain assumption The candidate recommendation set from the two-tower model is a reasonable pool for shelf generation.
- domain assumption Content-embedding similarity is a valid proxy for descriptor similarity during diversification.
- domain assumption A/B test assignment and control conditions isolate the effect of descriptive shelves.
Cite this review
Pith. "Pith review of Contextualizing Spotify's Audiobook List Recommendations with Descriptive Shelves." pith.science (2026). https://pith.science/paper/TZPAYXGZ
@misc{pith2026250413572,
author = {Pith},
title = {Pith review of: Contextualizing Spotify's Audiobook List Recommendations with Descriptive Shelves},
year = {2026},
howpublished = {\url{https://pith.science/paper/TZPAYXGZ}},
note = {Machine review of arXiv:2504.13572}
}
read the original abstract
In this paper, we propose a pipeline to generate contextualized list recommendations with descriptive shelves in the domain of audiobooks. By creating several shelves for topics the user has an affinity to, e.g. Uplifting Women's Fiction, we can help them explore their recommendations according to their interests and at the same time recommend a diverse set of items. To do so, we use Large Language Models (LLMs) to enrich each item's metadata based on a taxonomy created for this domain. Then we create diverse descriptive shelves for each user. A/B tests show improvements in user engagement and audiobook discovery metrics, demonstrating benefits for users and content creators.
Figures
Reference graph
Works this paper leans on
-
[1]
Blanco, R., Ceccarelli, D., Lucchese, C., Perego, R., Silvestri, F.: You should read this! let me explain you why: explaining news recommendations to users. In: Pro- 6 Penha et al. ceedings of the 21st ACM international conference on Information and knowledge management. pp. 1995–1999 (2012)
work page 2012
-
[2]
arXiv preprint arXiv:2005.14165 (2020)
Brown, T.B.: Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)
arXiv 2020
-
[3]
arXiv preprint arXiv:2101.03392 (2021)
Chen, H., Chen, X., Shi, S., Zhang, Y.: Generate natural language explanations for recommendation. arXiv preprint arXiv:2101.03392 (2021)
arXiv 2021
-
[4]
In: Companion Proceedings of the ACM on Web Conference 2024
De Nadai, M., Fabbri, F., Gigioli, P., Wang, A., Li, A., Silvestri, F., Kim, L., Lin, S., Radosavljevic, V., Ghael, S., et al.: Personalized audiobook recommendations at spotify through graph neural networks. In: Companion Proceedings of the ACM on Web Conference 2024. pp. 403–412 (2024)
work page 2024
-
[5]
In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Deldjoo, Y., He, Z., McAuley, J., Korikov, A., Sanner, S., Ramisa, A., Vidal, R., Sathiamoorthy, M., Kasirzadeh, A., Milano, S.: A review of modern recommender systems using generative models (gen-recsys). In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 6448–6458 (2024)
2024
-
[6]
arXiv preprint arXiv:1810.04805 (2018)
Devlin, J.: Bert: Pre-training of deep bidirectional transformers for language un- derstanding. arXiv preprint arXiv:1810.04805 (2018)
arXiv 2018
-
[7]
arXiv preprint arXiv:2407.21783 (2024)
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)
arXiv 2024
-
[8]
In: Proceedings of the 16th ACM Conference on Recommender Systems
Geng, S., Liu, S., Fu, Z., Ge, Y., Zhang, Y.: Recommendation as language pro- cessing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In: Proceedings of the 16th ACM Conference on Recommender Systems. pp. 299–315 (2022)
work page 2022
Show all 26 references
-
[9]
In: Proceedings of the 44th International ACM SIGIR Con- ference on Research and Development in Information Retrieval
Hada, D.V., Shevade, S.K.: Rexplug: Explainable recommendation using plug-and- play language model. In: Proceedings of the 44th International ACM SIGIR Con- ference on Research and Development in Information Retrieval. pp. 81–91 (2021)
2021
-
[10]
In: European Conference on Information Retrieval
Hou, Y., Zhang, J., Lin, Z., Lu, H., Xie, R., McAuley, J., Zhao, W.X.: Large language models are zero-shot rankers for recommender systems. In: European Conference on Information Retrieval. pp. 364–381. Springer (2024)
2024
-
[11]
Data Min- ing and Knowledge Discovery37(2), 833–872 (2023)
Iferroudjene, M., Lonjarret, C., Robardet, C., Plantevit, M., Atzmueller, M.: Meth- ods for explaining top-n recommendations through subgroup discovery. Data Min- ing and Knowledge Discovery37(2), 833–872 (2023)
2023
-
[12]
In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management
Li, L., Zhang, Y., Chen, L.: Generate neural template explanations for recommen- dation. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. pp. 755–764 (2020)
2020
-
[13]
In: Proceedings of the 44th International ACM SIGIR conference on Research and Development in Information Retrieval
Li, L., Zhang, Y., Chen, L.: Extra: Explanation ranking datasets for explainable recommendation. In: Proceedings of the 44th International ACM SIGIR conference on Research and Development in Information Retrieval. pp. 2463–2469 (2021)
2021
-
[14]
ACM Transactions on Recommender Systems (2024)
Liu, D., Greene, D., Li, I., Jiang, X., Dong, R.: Topic-centric explanations for news recommendation. ACM Transactions on Recommender Systems (2024)
2024
-
[15]
In: Proceedings of the 17th ACM Conference on Recommender Systems
Mysore, S., McCallum, A., Zamani, H.: Large language model augmented narra- tive driven recommendations. In: Proceedings of the 17th ACM Conference on Recommender Systems. pp. 777–783 (2023)
2023
-
[16]
In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)
Ni, J., Li, J., McAuley, J.: Justifying recommendations using distantly-labeled re- views and fine-grained aspects. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language process...
2019
-
[17]
In: Proceedings of the 14th ACM conference on recommender systems
Penha, G., Hauff, C.: What does bert know about books, movies and music? prob- ing bert for conversational recommendation. In: Proceedings of the 14th ACM conference on recommender systems. pp. 388–397 (2020) Contextualizing Recommendations with Descriptive Shelves 7
2020
-
[18]
In: Proceedings of the 2022 Conference on Human Information Interaction and Retrieval
Penha, G., Krikon, E., Murdock, V.: Pairwise review-based explanations for voice product search. In: Proceedings of the 2022 Conference on Human Information Interaction and Retrieval. pp. 300–304 (2022)
2022
-
[19]
Journal of machine learning research21(140), 1–67 (2020)
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research21(140), 1–67 (2020)
2020
-
[20]
Advances in Neural Information Processing Systems 36, 10299–10315 (2023)
Rajput, S., Mehta, N., Singh, A., Hulikal Keshavan, R., Vu, T., Heldt, L., Hong, L., Tay, Y., Tran, V., Samost, J., et al.: Recommender systems with generative retrieval. Advances in Neural Information Processing Systems 36, 10299–10315 (2023)
2023
-
[21]
In: Recommender Systems Handbook, pp
Tintarev, N., Masthoff, J.: Beyond explaining single item recommendations. In: Recommender Systems Handbook, pp. 711–756. Springer (2012)
2012
-
[22]
In: Recommender systems handbook, pp
Tintarev, N., Masthoff, J.: Explaining recommendations: Design and evaluation. In: Recommender systems handbook, pp. 353–382. Springer (2015)
2015
-
[23]
arXiv preprint arXiv:2402.18590 (2024)
Vats, A., Jain, V., Raja, R., Chadha, A.: Exploring the impact of large lan- guage models on recommender systems: An extensive review. arXiv preprint arXiv:2402.18590 (2024)
2024 arXiv
-
[24]
In: Proceedings of the 14th international conference on Intelligent user interfaces
Vig, J., Sen, S., Riedl, J.: Tagsplanations: explaining recommendations using tags. In: Proceedings of the 14th international conference on Intelligent user interfaces. pp. 47–56 (2009)
2009
-
[25]
In: IntRS@ RecSys
Wibowo, A.T., Siddharthan, A., Masthoff, J., Lin, C.: Understanding how to ex- plain package recommendations in the clothes domain. In: IntRS@ RecSys. pp. 74–78 (2018)
2018
-
[26]
arXiv preprint arXiv:2307.02046 (2023)
Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y., Wen, Z., Wang, F., Zhao, X., Tang, J., et al.: Recommender systems in the era of large language models (llms). arXiv preprint arXiv:2307.02046 (2023)
2023 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.