Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

AudioBoost: Increasing Audiobook Retrievability in Spotify Search with Synthetic Query Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Machine-written exploratory queries, injected into both autocomplete and the retrieval index, make cold-start audiobooks measurably more discoverable in production search.

desk verdict A credible industry paper with a small but real A/B effect; the offline simulation is honest but partly mechanical, and the online test bundles both channels. read the letter →

arxiv 2509.06452 v1 pith:ZRHBVCE7 submitted 2025-09-08 cs.IR cs.SD

Enrico Palumbo , Gustavo Penha , Alva Liu , Marcus Eltscheminov , Jefferson Carvalho dos Santos , Alice Wang , Hugues Bouchard , Humberto Jesús Corona Pampin
show 1 more author
Michelle Tran Luu
This is my paper
classification cs.IRcs.SD
keywords syntheticquerygenerationcold-startretrievalaudiobooksearchautocompletedocumentaugmentationLLMretrievabilityBM25
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a search system can fight cold-start bias with machine-written queries. When audiobooks are new to a catalog, few users search for them by topic, so retrieval rarely surfaces them. AudioBoost uses a large language model to turn each audiobook's metadata into synthetic exploratory queries, such as "dark romance audiobooks" or "books set in Venice," and feeds those queries into both the query-autocomplete source and the sparse retrieval index. Offline, the combination raises the audiobook share of retrievable entities from about 23% to about 57% in a simulation; online over three weeks it yields +0.7% audiobook impressions per results page, +1.22% clicks, and +1.82% exploratory completions, with neutral guardrails. If true, this is a practical recipe for making any newly introduced content type findable without waiting for user interaction data to accumulate.

What carries the argument

The load-bearing objects are (1) an LLM prompt with a ten-category descriptor taxonomy and chain-of-thought reasoning, producing queries and compound queries like "Stephen King supernatural fiction audiobooks"; (2) a Query AutoComplete source that scores synthetic completions by the median popularity of associated audiobooks times a broadness factor; and (3) BM25 document augmentation, where synthetic descriptors are concatenated to each audiobook's title-author-description-genres field. The taxonomy supplies the exploratory vocabulary, the broadness-weighted score steers users toward topic-level queries, and the document expansion makes those same queries match audiobook content in sparse r

What would settle it

Decompose the production A/B test into two arms: one with only document augmentation and one with only synthetic QAC suggestions, and measure audiobook impressions, clicks, and click-through on the synthetic completions themselves. If the combined system's lift is not larger than either single channel, or if users click almost none of the synthetic completions, the offline retrievability gains, which assume those completions are clicked, overstate the contribution of the query-formulation channel.

Watch

Extended reading notes

Core claim

At its core, AudioBoost claims that retrievability of cold-start items can be increased in production by generating synthetic queries conditioned on item metadata and placing those queries at both ends of the search funnel: query formulation and retrieval. The generated queries are not just indexed into documents and not just offered as autocomplete suggestions; the system does both, and the offline retrievability simulation shows the two interventions compound. Adding synthetic queries only to the query set lifts the audiobook share from 23.36% to 34.49%, adding them only to documents lifts it to 26.20%, and doing both reaches 56.97%. Online, the combined system increases audiobook impressi

Load-bearing premise

The offline retrievability simulation assumes users would click the synthetic autocomplete suggestions; if they do not, the large simulated gains collapse and only the modest online lift remains, mostly from document augmentation.

Editorial extensions

If this is right

  • Cold-start content becomes discoverable by topic, mood, trope, and setting instead of only by exact title or author.
  • The two interventions compound: document augmentation alone and query suggestion alone both help, but the full effect comes from doing both together.
  • The approach adds no latency to the live search path because generation and indexing run offline.
  • Exploratory behavior increases most among the measured metrics, suggesting users do engage with broader topic-level suggestions when shown them.
  • Neutral guardrail metrics mean the audiobook gains do not come at the expense of overall search or autocomplete engagement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports only the combined online system; an A/B ablation separating the QAC source from document expansion would show which channel carries the online lift.
  • The same pipeline should transfer to other cold-start content classes, but the ten-category taxonomy would need to be re-derived for each domain.
  • If users adopt the exploratory completions, search logs will gradually accumulate topic-level audiobook queries, making popularity signals less biased and possibly reducing the need for synthetic intervention over time.
  • The quality and relevance scores come from an LLM-as-a-judge; a small human-rated sample would be the natural check on those figures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper describes AudioBoost, a production system at Spotify that uses LLMs to generate synthetic audiobook queries from metadata, indexes them in Query AutoComplete and as document augmentation for BM25, and evaluates the approach offline and online. Offline, the authors simulate retrievability under four configurations and use LLM-as-a-judge to assess query quality. Online, a three-week A/B test reports +0.7% audiobook impressions per SERP, +1.22% audiobook clicks, +1.82% exploratory audiobook query completions, with neutral guardrails and p<0.01. The paper argues that both query formulation and retrieval augmentation are needed.

Significance. If the causal claims were fully supported, the work would be a valuable demonstration of LLM-based synthetic query generation for cold-start content in a deployed search system. The online A/B test is a genuine strength: it is large-scale, runs for three weeks, and reports statistically significant improvements on the target metrics with neutral guardrails. The proposed pipeline is also operationally plausible, as generation and indexing are offline. However, the offline retrievability simulation is partially circular, and the online experiment bundles the two interventions, so the current evidence does not establish the dual-channel mechanism or the quality of synthetic queries as strongly as the paper claims.

major comments (3)
  1. [Sec 3.1, Table 1, Fig 3] The simulation's configuration 4 gain is partly guaranteed by construction. Configurations 3 and 4 insert 20k synthetic queries into the query set Q, and configuration 4 simultaneously indexes these same synthetic queries into the audiobook documents (Sec 2.3). Because each synthetic query is generated from a particular audiobook's own metadata, the query-document match is by design; the jump from 23.36% to 56.97% therefore reflects the engineered overlap, not measured user behavior. The paper explicitly assumes 'such synthetic query suggestions made by the system would be clicked by at least one user' (Sec 3.1), and all queries are weighted equally in r(e). If the production click-through rate on synthetic completions is low, the QAC channel's contribution is much smaller than the offline table suggests. Please report the CTR/acceptance rate of synthetic completions, or separate the two
  2. [Sec 3.2] The LLM-as-a-judge relevancy metric is circular: the evaluator is asked whether a query is relevant to 'the audiobook metadata that was provided as input'—the same metadata used by the generator. This measures self-consistency, not external relevance. The paper cites correlation with human evaluation for general benchmarks, but no human or behavioral validation is reported for this task. A small human-judged sample or a behavioral signal (e.g., click-through of suggested queries) would be needed to support the 97.2% relevancy claim.
  3. [Sec 4] The online A/B test bundles the additional QAC source and document augmentation into a single treatment. While the reported effects are statistically significant, the effects are small (+0.7% impressions, +1.22% clicks), and the +0.03% coverage change is negligible. The experiment cannot identify whether the improvement comes from query formulation, document augmentation, or their interaction. To support the 'both steps need to be done' claim, please provide an analysis of how many SERPs used a synthetic completion, the synthetic-completion CTR, and ideally a partial-rollout or holdout experiment separating the channels. Confidence intervals and the denominator for the percentage changes would also help.
minor comments (6)
  1. [Abstract] Typo: 'retrievabiliy' should be 'retrievability'.
  2. [Sec 2.1] Typo: 'chain-of-though' should be 'chain-of-thought'.
  3. [Sec 2.1] Hyperparameters for LLM generation (model, temperature, number of queries per audiobook) are not reported; this limits reproducibility and the assessment of cost claims.
  4. [Sec 3.1] The choice of c=100 and the broadness term log(|A|+1) in Eq. (1) are not sensitivity-tested. Given their role in the metric and the global score, at least a brief robustness check would be useful.
  5. [Sec 4] The phrase 'p-value of 1%' is ambiguous; please state whether p<0.01 and provide confidence intervals for the reported percentage changes.
  6. [References] Minor formatting issues: 'Pyterrier' should be 'PyTerrier' in [4], and some preprint references lack venue details.

Circularity Check

2 steps flagged · score 4.0 of 10

Offline retrievability gains are partly guaranteed by injecting the same synthetic queries into both the query set and the documents; LLM-judge relevancy is self-referential, but the online A/B test independently validates the combined system.

  1. self definitional [Sec 3.1, Retrievability Simulation (Table 1, Fig. 3)]
    "Adding a sample of the synthetic queries (also 20k queries) for the query set Q used to calculate the retrievability (configurations 3 and 4) in this simulation assumes that such synthetic query suggestions made by the system would be clicked by at least one user. ... Table 1 shows the results of this simulation, for each configuration considering that the entire set of synthetic queries suggested would receive clicks."

    Configurations 3/4 insert the generated synthetic queries into Q, while configurations 2/4 concatenate the same synthetic descriptors and queries to the audiobook documents (augmented_document = 'title - author - description - genres - descriptors - synthetic queries'). The retrievability metric r(e)=sum_{q in Q} o_q f(k_eq,c) then counts exact query-document string matches. Since every synthetic query is generated from audiobook metadata and is audiobook-oriented, both the query-mix shift and the BM25 exact-match effect mechanically raise the audiobook share (23.36% to 56.97%). The 'increase in retrievability' is therefore a consequence of the simulation setup under the stated all-clicks assumption, not an independent measurement of retrieval quality or user behavior.

  2. self definitional [Sec 3.2, LLM evaluation (relevancy dimension)]
    "relevancy: queries need to be relevant to the audiobook metadata that was provided as input"

    The synthetic queries are generated from the audiobook title/author/description/BISAC genres, and the evaluator judges 'relevancy' with respect to 'the audiobook metadata that was provided as input' — i.e., the same metadata used for generation. A high relevancy score (97.2%) thus verifies that the generator followed its prompt (self-consistency), not that the queries are useful, clickable, or externally appropriate. The LLM-as-judge methodology has prior correlation evidence, but the specific relevancy check does not break the loop between generation input and evaluation target.

full rationale

AudioBoost's production claim rests on a three-week A/B test that is genuinely external: +0.7% impressions, +1.22% clicks, +1.82% exploratory completions, with neutral guardrails and significance at p=0.01. That measurement is not forced by the construction of the system. The circularity is confined to the offline evidence. In Sec 3.1 the same 20k synthetic queries are added to the query set Q (configs 3/4) and the same synthetic descriptors/queries are appended to audiobook documents (configs 2/4). Because the retrievability metric sums over Q and BM25 matches the exact augmented document strings, the jump to 56.97% audiobook share is a mechanical consequence of the evaluation design once the all-clicks assumption is granted. In Sec 3.2 the LLM judge scores 'relevancy' against the very metadata used to generate the queries, so the 97.2% relevancy is prompt fidelity, not evidence of user value. These are self-consistency checks, not independent validations. They do not invalidate the online result, but they mean the paper's offline 'prediction' of large gains is partly guaranteed by construction. No other load-bearing circularity (e.g., uniqueness theorems) appears. Score 4: partial circularity in supporting evidence, central claim independently grounded.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger is deliberately short: the paper is an empirical systems validation rather than a derivation, so there are no fitted constants in the physics sense. What the central claim rests on is a set of modeling proxies: BM25 for the production ranker, LLM-as-a-judge for human quality perception, the stated click assumption in the offline simulation, and the hand-built taxonomy. The unstated generation configuration and the hand-chosen broadness scoring are free parameters because they directly shape the QAC supply and the augmented documents. No new entities are postulated; the synthetic queries are data artifacts.

free parameters (3)
  • LLM generation configuration (model, temperature, number of queries per audiobook)
    Sec 2.1 never names the LLM, sampling parameters, or how many synthetic queries and descriptors are generated per audiobook. These choices set the supply of QAC candidates and the amount of document augmentation, and they are unstated.
  • Broadness term in global score (Eq. 1) = log(|A|+1)
    The QAC global score is median popularity times log(|A|+1). The log form and the multiplicative combination are hand-chosen without an ablation, and they directly determine which synthetic queries surface in autocomplete.
  • Retrievability cutoff c = 100
    The retrievability indicator in Sec 3.1 uses c=100 (entity counted as retrieved if ranked above 100). Reported shares depend on this arbitrary cutoff; no sensitivity analysis is given.
assumptions (4)
  • domain assumption Pyterrier BM25 with default hyperparameters approximates the production sparse retrieval system that AudioBoost augments
    Sec 3.1 evaluates retrievability offline with BM25 defaults, while Sec 2.3 says the production sparse retrieval system is 'based on BM25'. The mapping from offline BM25 behavior to production ranking behavior is assumed.
  • domain assumption LLM-as-a-judge ratings proxy human judgments of quality, relevancy, diversity, and broadness for audiobook queries
    Sec 3.2 asserts correlation with human evaluation from public benchmarks [9,12] and 'internal assessments on similar tasks'. No human labels for audiobook queries are shown in this paper.
  • ad hoc to paper Suggested synthetic query completions would be clicked by users (offline simulation premise)
    Sec 3.1: 'this simulation assumes that such synthetic query suggestions made by the system would be clicked by at least one user.' This premise drives the query-distribution shift in configurations 3 and 4, without which the simulated gain collapses.
  • domain assumption The manually built 10-category taxonomy spans audiobook search intents
    Sec 2.1 derives the taxonomy from internal search logs and the r/booksuggestions forum. If real exploratory audiobook queries fall outside these categories, both the QAC suggestions and the document augmentation will miss them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AudioBoost: Increasing Audiobook Retrievability in Spotify Search with Synthetic Query Generation." pith.science (2026). https://pith.science/paper/ZRHBVCE7

@misc{pith2026250906452,
  author       = {Pith},
  title        = {Pith review of: AudioBoost: Increasing Audiobook Retrievability in Spotify Search with Synthetic Query Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRHBVCE7}},
  note         = {Machine review of arXiv:2509.06452}
}
read the original abstract

Spotify has recently introduced audiobooks as part of its catalog, complementing its music and podcast offering. Search is often the first entry point for users to access new items, and an important goal for Spotify is to support users in the exploration of the audiobook catalog. More specifically, we would like to enable users without a specific item in mind to broadly search by topic, genre, story tropes, decade, and discover audiobooks, authors and publishers they may like. To do this, we need to 1) inspire users to type more exploratory queries for audiobooks and 2) augment our retrieval systems to better deal with exploratory audiobook queries. This is challenging in a cold-start scenario, where we have a retrievabiliy bias due to the little amount of user interactions with audiobooks compared to previously available items such as music and podcast content. To address this, we propose AudioBoost, a system to boost audiobook retrievability in Spotify's Search via synthetic query generation. AudioBoost leverages Large Language Models (LLMs) to generate synthetic queries conditioned on audiobook metadata. The synthetic queries are indexed both in the Query AutoComplete (QAC) and in the Search Retrieval engine to improve query formulation and retrieval at the same time. We show through offline evaluation that synthetic queries increase retrievability and are of high quality. Moreover, results from an online A/B test show that AudioBoost leads to a +0.7% in audiobook impressions, +1.22% in audiobook clicks, and +1.82% in audiobook exploratory query completions.

Figures

Figures reproduced from arXiv: 2509.06452 by the authors.

Figure 1
Figure 1. Illustrative example - Synthetic queries generated [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. AudioBoost pipeline a) We collect metadata about a given audiobook b) we use a pre-trained LLM with chain-of-thought [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Offline simulation showing the impact on retriev [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LLM-Based Re-Ranking for Real Estate Search

    cs.IR 2026-07 conditional novelty 5.0 of 10

    An LLM pointwise re-ranker using user profiles and candidate-set statistics improved ranking quality in a real-estate marketplace, with statistically significant production gains of +5.3% CTR and +4.8% scheduled visits.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Leif Azzopardi and Vishwa Vinay. 2008. Retrievability: An evaluation measure for higher order information access tasks. InProceedings of the 17th ACM conference on Information and knowledge management. 561–570

    work page 2008

  2. [2]

    Thomas, and Jean Garcia-Gathright

    Ang Li, Jennifer Thom, Praveen Chandar, Christine Hosey, Brian St. Thomas, and Jean Garcia-Gathright. 2019. Search Mindsets: Understanding Focused and Non- Focused Information Seeking in Music Search. InThe World Wide Web Conference (San Francisco, CA, USA)(WWW ’19). Association for Computing Machinery, New York, NY, USA, 2971–2977. https://doi.org/10.114...

    work page arXiv 2019

  3. [3]

    Henrik Lindstrom, Humberto Jesus Corona Pampin, Enrico Palumbo, and Alva Liu. 2024. Encouraging Exploration in Spotify Search through Query Recom- mendations. InProceedings of the 18th ACM Conference on Recommender Systems (Bari, Italy)(RecSys ’24). Association for Computing Machinery, New York, NY, USA, 775–777. https://doi.org/10.1145/3640457.3688035

    work page arXiv 2024

  4. [4]

    Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. 2021. PyTerrier: Declarative experimentation in Python from BM25 to dense retrieval. InProceedings of the 30th acm international conference on information & knowledge management. 4526–4533

    2021

  5. [5]

    Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019. From doc2query to docTTTTTquery.Online preprint6 (2019)

    work page 2019

  6. [6]

    Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction.arXiv preprint arXiv:1904.08375(2019)

    arXiv 2019

  7. [7]

    Enrico Palumbo, Andreas Damianou, Alice Wang, Alva Liu, Ghazal Fazelnia, Francesco Fabbri, Rui Ferreira, Fabrizio Silvestri, Hugues Bouchard, Claudia Hauff, Mounia Lalmas, Ben Carterette, Praveen Chandar, and David Nyhan. 2023. Graph Learning for Exploratory Query Suggestions in an Instant Search System. InProceedings of the 32nd ACM International Confere...

    work page arXiv 2023

  8. [8]

    Gustavo Penha, Enrico Palumbo, Maryam Aziz, Alice Wang, and Hugues Bouchard. 2023. Improving Content Retrievability in Search with Controllable Query Generation. InProceedings of the ACM Web Conference 2023(Austin, TX, USA)(WWW ’23). Association for Computing Machinery, New York, NY, USA, 3182–3192. https://doi.org/10.1145/3543507.3583261

    work page arXiv 2023

Show all 12 references
  1. [9]

    Hossein A Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles LA Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggioli. 2024. LLMJudge: LLMs for Relevance Judgments.arXiv preprint arXiv:2408.08896(2024)

    2024 arXiv

  2. [10]

    Stephen Robertson, Hugo Zaragoza, et al . 2009. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends®in Information Retrieval 3, 4 (2009), 333–389

    2009

  3. [11]

    Federico Tomasi, Rishabh Mehrotra, Aasish Pappu, Judith Bütepage, Brian Brost, Hugo Galvão, and Mounia Lalmas. 2020. Query Understanding for Surfacing Under-served Music Content. InProceedings of the 29th ACM International Con- ference on Information & Knowledge Management(Vir...

    2020

  4. [12]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems36 (2023), 46595–46623

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.