{"id":"70bdfd91-c817-45de-a77f-4fd81b3366e7","arxiv_id":"2501.11034","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GBS applies generative retrieval to book search by augmenting training data with hierarchical book identifiers and pseudo-queries, and encoding books with outline-based bi-level positions and retentive attention, reporting gains over RIPOR on BBS and WhatsThatBook.","lead":"Researchers propose GBS, a generative retrieval system that uses book outlines and augmented training data to return book identifiers from search queries. The method reports gains over strong baselines on a proprietary Baidu book dataset and a public benchmark, but the main evaluation uses synthetic queries generated from the same books used in training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The BBS 40K headline gain may be inflated because the test pseudo-queries are generated from the same book chapters used in GBS's indexing training, so the 9.8% MRR@20 advantage over RIPOR needs verification on real user queries.","rationale":"The reader's weakest-assumption analysis identified the BBS pseudo-query test set as the main concern; I agree that this is the most load-bearing issue. I would sharpen it: the test queries are generated from chapters whose full text is itself a training input for the indexing task, so the evaluation conflates retrieval with recognition of memorized source content. This is a stronger failure mode than ordinary train/test distribution shift, and it directly affects the headline 9.8% number. The WhatsThatBook result, based on real queries, is genuine independent support and explains why I do not call for rejection: it shows a smaller but real gain (6% Hits@10) and suggests the method has value. However, the BBS claim should be re-verified on real queries or with query generation separated from the training source text. Because the reader's CONDITIONAL verdict already encodes exactly this caveat, my read does not change the verdict; it only makes the justification more precise. I did not find a separate internal inconsistency in the method description itself; the weakness is in the evaluation's external validity, not in the derivations.","tokens_in":17980,"tokens_out":7844,"duration_ms":78473,"concrete_test":"Use a held-out set of real user queries from Baidu book search logs for the same 40K books and rerun the BBS 40K comparison between GBS_P and RIPOR with identical training data and hyperparameters, reporting MRR@20 and Hits@10. If GBS_P's advantage over RIPOR drops to the WhatsThatBook level (about 6% Hits@10) or becomes statistically non-significant across multiple seeds, the BBS headline is an artifact of pseudo-query construction rather than a generalizable retrieval gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GBS_P beats RIPOR by 9.8% MRR@20 on BBS 40K and by 6% Hits@10 on WhatsThatBook (Section 4.1). For the BBS result to support the claim, the test queries must represent realistic information needs. Section 3 states: 'We construct pseudo-queries for each book for training and evaluation using the method described in Section 2.3,' i.e., the same LLM prompts generate both training and test queries, and those prompts take chapter texts as input. The same chapter texts appear in GBS's indexing training pairs: whole text, chapter text, section text, keywords, and summaries are all mapped to hierarchical identifiers (Section 2.3.1). Thus, at test time, a pseudo-query that summarizes or paraphrases a chapter can be answered by recognizing content the model has already memorized and tied to a book id, rather than by generalizing to a new information need. RIPOR and the other GR baselines are trained on split book segments without the same whole-book, multi-level identifier augmentation, so part of the reported gap may measure this evaluation-construction advantage rather than retrieval quality. The WhatsThatBook experiment uses real user queries and shows a smaller 6% gain, consistent with the concern. The paper's limitation paragraph (Section 6) mentions training cost and model capacity but does not flag the synthetic, source-overlapping BBS test set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GBS, a generative retrieval (GR) framework for book search. GBS combines two main ideas: (i) data augmentation, which includes coverage-promoting book identifier augmentation (multiple content forms mapped to hierarchical book/chapter/section identifiers) and diversity-enhanced query augmentation (LLM-generated pseudo-queries of single-chapter and multiple-chapter types); and (ii) outline-oriented book encoding, which applies bi-level positional encoding and a retentive attention memory to handle long book texts. Training uses multi-task MLE over indexing and retrieval pairs (Eq. 4-6), and inference uses constrained decoding with book- and chapter-level prefix trees and parallel or serial aggregation (Section 2.6). Experiments on a proprietary Baidu BBS dataset (10K/20K/40K books) and the public WhatsThatBook dataset report improvements over the strongest GR baseline RIPOR, e.g., 9.8% MRR@20 on BBS 40K and 6% Hits@10 on WhatsThatBook (Table 2).","tokens_in":18183,"tokens_out":4366,"duration_ms":43055,"significance":"If the reported results are valid, GBS would be a meaningful advance in applying GR to long structured documents and a practical contribution to book search. The paper is clearly written, the method is well motivated, and the ablations (Tables 3 and 4) show that each component contributes. A notable strength is the evaluation on WhatsThatBook, which uses real user queries and provides external grounding beyond the proprietary dataset. However, the headline BBS result is undermined by the evaluation protocol: the BBS test queries are generated with the same LLM prompts and from the same book content used for training, so the 9.8% MRR@20 gain over RIPOR may reflect memorization of the training distribution rather than retrieval generalization. The public WhatsThatBook result shows a smaller gain (6% Hits@10) and is more credible as evidence of real-world effectiveness.","major_comments":[{"comment":"The BBS test set is constructed with the same procedure as the training pseudo-queries: Section 3 states that pseudo-queries are generated 'for each book for training and evaluation using the method described in Section 2.3,' i.e., from the same chapter texts and with the same LLM prompts. Because the indexing task (Eq. 4) already trains the model to map whole text, chapter text, section text, keywords, and summaries to identifiers, a test pseudo-query that paraphrases or summarizes a chapter can be answered by recognizing content the model has seen in training and tied to a book-id. This creates a circularity that likely inflates the reported 9.8% MRR@20 improvement over RIPOR on BBS 40K. The WhatsThatBook experiment uses real user queries and shows a smaller 6% Hits@10 gain, consistent with this concern. The manuscript should either re-evaluate BBS with held-out books/queries or explicitly present BBS as a synthetic stress test and base the central claim on the public dataset.","section":"Section 3 (Datasets) and Section 2.3.2"},{"comment":"The book-id format is title#author#publisher, and the single-chapter query prompt asks the LLM to generate queries 'focus on key themes, events, characters, and any specific details provided in the chapter.' The case study in Table 5 includes the query 'Who is the author of the book “The Heart of a Boy”?', which already contains the exact title and author, making the correct book-id trivially recoverable from the query itself without any book understanding. This suggests the BBS test set contains queries that are easy for GBS's identifier design and may contribute to the reported performance gap. The authors should report the distribution of query difficulty (e.g., how many test queries contain the title or author) and ideally filter or separately analyze such cases.","section":"Section 2.3.1 and Table 5 (case study)"},{"comment":"The BBS dataset is proprietary and not released, so the headline results cannot be independently reproduced. Given that the test queries are synthetically generated and the dataset is unavailable, the central quantitative claim rests on a black-box setting. At minimum, the authors should release the exact prompt templates, the generated test queries, and the code for the full data-generation and evaluation pipeline, so that reviewers and the community can assess and reproduce the BBS results.","section":"Section 3, Table 1"}],"minor_comments":[{"comment":"There is a typo: 'During inference We employ the Adam optimizer' should be 'During inference, we employ the Adam optimizer...' (missing comma and capital letter).","section":"Section 3.1"},{"comment":"For the multiple-chapter answerable query 'Introducing Enrico', GBS_P ranks the correct book second (An Introduction to the Basics of Reliability and Risk Analysis), yet the text says 'GBS_P also performs well' without noting that it did not achieve the top rank; this should be stated precisely.","section":"Section 4.5, Table 5"},{"comment":"The caption reads 'Hit@10' but the metric elsewhere is 'Hits@10'; please make the terminology consistent.","section":"Figure 4 caption"},{"comment":"The text refers to 'our proposed method GBR' but the method is named GBS; this is likely a typo and should be corrected.","section":"Introduction, paragraph 5"},{"comment":"The dagger symbol (†) indicates p≤0.05 for GBS_P versus RIPOR, but the paper does not specify which statistical test was used, how many runs or folds, or whether the comparison is paired. This should be documented in the experimental settings.","section":"Table 2"},{"comment":"The parallel decoding aggregation multiplies book- and chapter-level scores while serial decoding uses a weighted sum; the text cites [17] for the parallel form but gives no justification for the differing aggregation strategies. Please clarify the rationale.","section":"Section 2.6"}],"recommendation":"major_revision","confidential_remarks":"The BBS dataset is proprietary, and the synthetic construction of both training and test queries from the same book content makes the headline 9.8% MRR@20 improvement over RIPOR unverifiable and potentially inflated. The WhatsThatBook result, which uses real user queries and shows a 6% Hits@10 gain, provides partial support but is smaller and on a different domain distribution. I recommend major revision: the authors should either obtain a realistic BBS test set (e.g., from real Baidu queries) or restructure the claims so that the public dataset is the primary evidence. Given the authors' track record and the availability of the public WhatsThatBook evaluation, this is fixable within the scope of the paper and does not warrant rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a sensible application of generative retrieval to whole-book search, and the outline-aware encoding is worth knowing about, but the headline BBS number is weaker than it looks because the test queries come from the same LLM prompt distribution and the same book content used to train the index.\n\nWhat's actually new: the paper is the first serious attempt to apply GR to whole books, using the outline to build hierarchical identifiers and to guide long-context encoding. The bi-level positional encoding plus retentive attention is a reasonable adaptation to long structured documents, and the coverage-promoting identifier augmentation gives the model multiple entry points into a book. The WhatsThatBook experiment uses real user queries from GoodReads, which is the right kind of external check and provides independent evidence that something in the method works.\n\nThe soft spot is the BBS evaluation. Section 3 says test pseudo-queries are generated with the same method as training (Section 2.3), from the same chapter texts that feed the indexing pairs (whole text, chapter text, section text). At test time the model is essentially being asked to map a paraphrase of a chapter back to a book id it has already memorized during indexing. That measures fit to the pseudo-query distribution more than retrieval generalization. It is not surprising that the gain over RIPOR on BBS (9.8% MRR@20) is larger than the gain on real queries (6% Hits@10). I would not call the BBS result meaningless, but it should be labeled as a distribution-fit test, not a proxy for real user search.\n\nA second, related issue: the baselines are not trained with the same augmented data. GBS gets whole-book, chapter-level, and section-level content-identifier pairs; the GR baselines are trained on split book segments and, in most cases, their own pseudo-queries. Part of the gap could come from having richer and more varied indexing pairs rather than from the outline-oriented encoding per se. An ablation that gives RIPOR a comparable identifier augmentation would separate those factors.\n\nMinor issues: no error bars (the significance marker is described in one line without test details), the proprietary dataset is not released, and the limitation section only mentions training cost and model capacity, not the test-set overlap.\n\nBottom line: this is a real paper with a real contribution, but the main claim needs a harder test. I would send it to review and ask for the WhatsThatBook analysis to be expanded, the BBS test set released or replaced with held-out real queries, and at least one strong baseline trained with the same augmentation. If they address those, the result becomes credible; until then, treat the 9.8% figure with skepticism.","headline":"Solid industrial GR-for-books paper, but the headline BBS gain is inflated by a same-distribution synthetic test set; the public real-query result is the more honest number.","tokens_in":18836,"tokens_out":2739,"would_cite":true,"duration_ms":30601,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new generative retrieval framework for book search, built on outline-based encoding and LLM-augmented training data, reports gains over the strongest prior generative baseline on both industry and public book datasets.","keywords":["generative retrieval","book search","outline-oriented encoding","pseudo-query augmentation","book identifiers","long-text modeling","retentive attention","positional encoding"],"falsifier":"Build a held-out test set from real user queries to a live book search service, with no LLM-generated queries, and compare GBS against RIPOR on MRR@20 and Hits@10; if the gap shrinks below the reported 9.8% and 6%, the advantage is driven by the pseudo-query distribution.","tokens_in":17688,"feed_emoji":"📚","tokens_out":8990,"duration_ms":86628,"temperature":0.7,"pith_summary":"Book search differs from web search because a book's relevant content is spread across metadata, a hierarchical outline, and a very long main text. This paper argues that generative retrieval, a paradigm that compresses the whole corpus into one model that directly emits document identifiers for a query, can be made to work for books if two problems are solved: the model needs enough training signal to cover a book's many facets, and it needs an encoding that respects the outline's chapter-section structure. The proposed framework, GBS, solves the first with identifier augmentation and pseudo-queries generated by large language models, and the second with outline-oriented positional encodings and a retentive attention memory. On a proprietary 40K-book dataset it reports a 9.8% gain in MRR@20 over the strongest prior generative baseline, and on a public dataset of real book queries it reports a 6% gain in Hits@10. The sympathetic reading is that hierarchy-aware encoding and augmented training pairs, not just scale, are what let generative retrieval handle books.","feed_headline":"Generative book search beats top baseline by 9.8%","feed_subtitle":"Outline encoding plus LLM-made pseudo-queries also raises Hits@10 on real tip-of-the-tongue queries.","key_machinery":"The load-bearing machinery is the pair of augmentation schemes plus the outline-oriented encoder. Coverage-promoting identifier augmentation constructs book-, chapter-, and section-level identifiers and ties each to the content form that matches its granularity, so the model learns the mapping from whole-book and partial views to a shared identifier. Diversity-enhanced query augmentation uses two LLM prompts to generate single-chapter answerable and multiple-chapter answerable pseudo-queries, giving the retrieval task a spread of difficulty. The encoder applies bi-level positional encoding, with absolute positions inside sections and rotary relative positions between sections, and outline-oriented retentive attention, which augments standard multi-head attention with a memory state updated as a linear-attention recurrence. These mechanisms together are what the paper credits for the reported gains over RIPOR.","core_discovery":"The paper's central discovery is that applying generative retrieval to books fails when the book is treated as flat segments and succeeds when each book is represented by multiple content views, multiple identifier levels, and an outline-aware encoder. GBS builds hierarchical identifiers at book, chapter, and section levels from titles, authors, publishers, and cluster-derived semantic numbers, then trains the indexing task by pairing keywords, summaries, section text, chapter text, and whole text with the appropriate identifier level. For the retrieval task it pairs LLM-generated pseudo-queries, of two difficulty types, with book-level identifiers. The encoder combines section-local positional encodings with chapter-level rotary relative encodings, and adds a retentive memory that carries key-value states across long inputs in a linear-attention recurrent update. The reported numbers are MRR@20 of 46.9 versus 42.7 for the RIPOR baseline on the 40K-book industry dataset, and Hits@10 of 70.7 versus 66.7 on WhatsThatBook, which the paper takes as evidence that hierarchical structure is the enabling ingredient for generative book search.","pith_inferences":["Because the main test set is generated by the same LLM pseudo-query procedure used to create training queries, the 9.8% gain measures performance on the distribution the model was trained for; the WhatsThatBook result, built from real user queries, is the more informative transfer test.","The outline-based encoder should transfer to other structured long documents, but the gain would depend on the availability of an explicit hierarchy comparable to a book outline.","A natural stress test is to shift the pseudo-query generator, prompt, or language at test time and measure how much of the advantage survives; the paper does not report this.","The stated training-cost bottleneck suggests an obvious next step: use the outline to allocate augmentation selectively to chapters or sections where retrieval errors are concentrated, rather than generating pseudo-queries uniformly."],"forward_implications":["Each component contributes: removing query augmentation drops Hits@10 from 56.7 to 50.6 on the 40K-book dataset, removing identifier augmentation drops it to 45.3, and removing bi-level positional encoding drops it to 52.8.","Retentive attention provides a smaller but consistent lift, and its value increases with input length up to 128K tokens, after which performance slightly declines.","Parallel decoding with book- and chapter-level prefix trees works slightly better than serial decoding, and both variants outperform every non-generative and generative baseline tested.","If the reported results hold, generative retrieval is a viable retrieval paradigm for book-scale documents, and the same outline-aware recipe can be tested on other long hierarchical texts such as legal or technical corpora."],"supporting_citations":[{"why":"Defines the indexing and retrieval tasks of generative retrieval and the use of semantic structured numbers as identifiers, which GBS extends to three identifier levels.","marker":"[36]"},{"why":"RIPOR is the strongest generative baseline and the main comparison target; GBS claims a 9.8% MRR@20 advantage over it on the 40K-book dataset.","marker":"[46]"},{"why":"Supplies the multiview identifier aggregation strategy that GBS adapts to combine book-level and chapter-level relevance scores at inference.","marker":"[17]"},{"why":"Introduces constrained decoding with prefix trees for generated identifiers, the mechanism GBS uses to keep outputs valid.","marker":"[9]"},{"why":"NCI is a baseline and a source of the prefix-aware decoder and pseudo-query-plus-identifier training design for generative retrieval.","marker":"[39]"},{"why":"DSI-QG provides the pseudo-query generation approach for GR baselines and motivates GBS's diversity-enhanced query augmentation.","marker":"[52]"},{"why":"Rotary position encoding is used as the chapter-level relative positional encoding in GBS's bi-level positional encoding.","marker":"[32]"},{"why":"Supplies the linear-attention recurrence that defines the update and retrieval of the retentive memory in GBS's encoder.","marker":"[15]"},{"why":"WhatsThatBook is the public dataset of real tip-of-the-tongue queries used to test generalization beyond the proprietary corpus.","marker":"[18]"}],"fun_headline_variants":["Book search: outline-aware generative retrieval lifts MRR by 9.8%","Hierarchical generative retrieval beats RIPOR by 9.8% on books","Outline-encoded books: generative retrieval jumps 9.8% MRR","Pseudo-queries plus outline encoding power generative book search","Book GR with hierarchical IDs: 9.8% better MRR@20 than RIPOR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The main benchmark's test queries are generated by the same LLM prompts used to create the training queries, so the reported advantage over RIPOR presupposes that such pseudo-queries stand in for what real book-search users actually type.","fun_headline_variants_meta":{"raw":{"variants":["Book search: outline-aware generative retrieval lifts MRR by 9.8%","Hierarchical generative retrieval beats RIPOR by 9.8% on books","Outline-encoded books: generative retrieval jumps 9.8% MRR","Pseudo-queries plus outline encoding power generative book search","Book GR with hierarchical IDs: 9.8% better MRR@20 than RIPOR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2411,"prompt_tokens":1061,"completion_tokens":1350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":1262}},"tokens_in":677,"tokens_out":1350,"duration_ms":11589,"temperature":1.0,"reasoning_tokens":1262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:41:21.289087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a held-out test set from real user queries to a live book search service, with no LLM-generated queries, and compare GBS against RIPOR on MRR@20 and Hits@10; if the gap shrinks below the reported 9.8% and 6%, the advantage is driven by the pseudo-query distribution.","supporting_citations":[{"cited_title":"Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W","cited_arxiv_id":null,"evidence_quote":"Defines the indexing and retrieval tasks of generative retrieval and the use of semantic structured numbers as identifiers, which GBS extends to three identifier levels."},{"cited_title":"In 2020 IEEE conference on multimedia information processing and retrieval (MIPR)","cited_arxiv_id":null,"evidence_quote":"RIPOR is the strongest generative baseline and the main comparison target; GBS claims a 9.8% MRR@20 advantage over it on the 40K-book dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multiview identifier aggregation strategy that GBS adapts to combine book-level and chapter-level relevance scores at inference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces constrained decoding with prefix trees for generated identifiers, the mechanism GBS uses to keep outputs valid."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NCI is a baseline and a source of the prefix-aware decoder and pseudo-query-plus-identifier training design for generative retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DSI-QG provides the pseudo-query generation approach for GR baselines and motivates GBS's diversity-enhanced query augmentation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rotary position encoding is used as the chapter-level relative positional encoding in GBS's bi-level positional encoding."}],"review_version":1}