{"id":"bc78202f-9c53-4704-9ff3-0342794eb8d7","arxiv_id":"2412.18768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Generative retrieval models lose large amounts of accuracy under out-of-distribution queries and tasks in KILT experiments, showing their robustness needs improvement.","lead":"This paper tests how well AI search systems that generate document IDs directly perform when they meet queries, tasks, or documents they were not trained on. It finds clear weakness on several of those shifts, especially misspelled queries and new question types.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Corpus-expansion DROOD uses the D0-trained model on Q0 as the IID baseline, conflating query difficulty with corpus growth; Table 7 does not support the paper's robustness conclusions for that scenario.","rationale":"The reader's weakest assumption correctly identifies the corpus-expansion operationalization as fragile. This is the single most load-bearing problem because the corpus-expansion scenario is one of four pillars of the paper's OOD taxonomy, and the specific numbers in Table 7 are used to argue that generative models differ qualitatively from dense models (e.g., BART improves while CorpusBrain collapses). The flaw is not merely a missing error bar; the baseline violates the paper's own Eq. (6). Because the other three scenarios (query variations, unseen query types, unseen tasks) are more direct instantiations of the stated OOD definitions, the central claim that generative IR OOD robustness needs improvement remains plausible. However, the paper's conditional acceptance should require a corrected corpus-expansion analysis, including a full-corpus-trained baseline and a pure-new-document subset. The BM25 FC DROOD inconsistency in Table 7 is a red flag that the table has not been checked. I therefore agree with the reader's CONDITIONAL verdict and recommend no change.","tokens_in":14011,"tokens_out":9904,"duration_ms":76802,"concrete_test":"Recompute the corpus-expansion DROOD with the correct IID baseline: fine-tune BART and CorpusBrain on the full KILT training set (both D0 and D1 documents) and evaluate on Q1; compare those numbers to the D0-trained model's Q1 performance. If the DROOD sign or magnitude changes substantially for Dialogue or QA, the paper's corpus-expansion robustness conclusion is an artifact of the split. As a second check, restrict Q1 to queries whose relevant documents are entirely in D1 (no D0 overlap) and report R-precision; if generative models score near zero on this 'pure-new' subset, their apparent corpus-expansion robustness is an artifact of retrieving old documents.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that generative IR models are OOD-fragile rests on four scenarios, and the corpus-expansion scenario (Sec. 5.1, Table 7) is the least secure. Eq. (6) defines DROOD with PIID as the model trained on the training set corresponding to the test set. For the OOD test set Q1 (queries whose provenance includes D1 documents) the corresponding training set is the full corpus training set, not the D0-filtered set. The paper instead takes the D0-trained model's R-precision on Q0 as PIID, so DROOD conflates three effects: (i) missing supervised signal for new documents, (ii) query-difficulty differences between Q0 and Q1, and (iii) corpus growth. Moreover, Q1 is defined as 'the remaining dataset' after excluding queries whose relevant articles are exclusively in D0; queries with some relevant articles in D0 and some in D1 are included, so a model can score R-precision by retrieving only seen D0 documents. Thus BART's apparent +10.1% DROOD on Dialogue and CorpusBrain's -58.5% do not measure the ability to retrieve newly added documents. Even internal consistency fails: BM25 FC in Table 7 reports DROOD -6.0 for a drop from 46.5 to 37.8, which should be -18.7. Correcting this one table invalidates the corpus-expansion comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the out-of-distribution robustness of generative information retrieval models. It proposes a four-perspective taxonomy of OOD scenarios—query variations, unseen query types, unseen tasks, and corpus expansion—and evaluates BM25, DPR, BART, and CorpusBrain on KILT-based constructed datasets. The main empirical claim is that generative IR models, despite strong IID performance, degrade substantially under OOD shifts, especially query variations and unseen tasks, and that their OOD robustness is in need of improvement. The paper also analyzes differences between generative and dense models and attributes some patterns to model architecture and pretraining. The authors release code.","tokens_in":14265,"tokens_out":6945,"duration_ms":61501,"significance":"If the findings hold, the paper makes a useful contribution by showing that IID leaderboard scores on KILT do not reliably indicate robustness for generative retrievers. The four-scenario taxonomy is a reasonable organizing framework, and the comparison across sparse, dense, and generative families is informative. The paper is also honest about its limited model and dataset coverage in the conclusion. However, the empirical rigor is currently uneven: one of the four scenarios (corpus expansion) does not realize the paper's own definition of the OOD drop, and several reported DROOD values contain arithmetic inconsistencies. These issues must be fixed before the headline claim can be fully accepted.","major_comments":[{"comment":"The corpus-expansion experiment does not instantiate Eq. (6). Eq. (6) defines PIID as the performance of the model trained on the training set corresponding to the test set. For the OOD set Q1, the corresponding training set is the training set built on the full corpus, not the D0-filtered training set; the paper instead compares the D0-trained model's R-precision on Q0 with its R-precision on Q1. This conflates corpus growth with query-set difficulty. Moreover, Q1 is defined as the remaining dataset after excluding queries whose relevant articles are exclusively in D0, so queries with some relevant articles in D0 and some in D1 are included; a model can score substantial R-precision by retrieving only the D0 pages in the provenance, making the metric partially insensitive to new-document retrieval. The reported values such as BART +10.1 on Dialogue and CorpusBrain -58.5 therefore do not measure the ability to retrieve newly added documents. The table should be recomputed with a proper IID baseline trained on the full-corpus training set, or the experiment should be redesigned to control query difficulty.","section":"Section 5.1 and Eq. (6), Table 7"},{"comment":"There is a straightforward arithmetic error in the BM25 row for the FC column: R-precision drops from 46.5 to 37.8, which gives DROOD = (37.8 - 46.5)/46.5 = -18.7%, not -6.0%. This error, together with the baseline issue in the previous comment, means Table 7 cannot currently support the paper's corpus-expansion conclusions.","section":"Table 7"},{"comment":"Several DROOD entries for BM25 are inconsistent with the reported R-precision values. For FEV Order, 22.3 vs. 50.1 should be -55.5%, not 0; for NQ Naturality, 25.4 vs. 25.8 should be approximately -1.6%, not -0.02; and for NQ Order, 31.0 vs. 25.8 should be +20.2%, not 0. These discrepancies suggest a unit or computation error in the query-variation table and should be corrected before the results are used to support the paper's claims.","section":"Table 4"},{"comment":"No error bars, number of runs, or significance tests are reported for the unseen query type, unseen task, and corpus expansion experiments. The text uses 'significant' claims (e.g., Section 6.3) and compares small differences, such as CorpusBrain -7.3 vs. BART -12.5 on NQ in Table 5, without any indication of variance. The authors should report multiple seeds or at least statistical tests so readers can judge whether the observed patterns are reliable.","section":"Section 5.3 and Tables 5-7"}],"minor_comments":[{"comment":"The text describes this scenario as 'unseen tasks,' but the sentence immediately after Eq. (4) says 'unseen corpus'; please correct the mismatch.","section":"Section 4.3, Eq. (4)"},{"comment":"The expectation notation in Eq. (5) uses (qt, D, yt) ~ GT, but the scenario concerns corpus-expansion samples (qn_t, Dn, yn_t); the notation should be made consistent.","section":"Section 4.4, Eq. (5)"},{"comment":"The row label 'zxRE' should be 'zsRE' to match the dataset name Zero Shot RE.","section":"Table 1"},{"comment":"The phrase 'every though they focus on the same task' contains a typo; it should read 'even though they focus on the same task.'","section":"Section 4.2"},{"comment":"The BM25 Naturality DROOD values appear to be written as fractions (-0.02) rather than percentages, unlike the rest of the table; please unify the units.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the topic is timely. I do not see circularity: the conclusions are empirical and not derived from fitted parameters. The main problems are the invalid corpus-expansion baseline, the table arithmetic errors, and the lack of statistical detail. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is the first systematic robustness evaluation of generative IR models, and the two strongest experiments are worth publishing. But the corpus-expansion leg has a baseline mismatch and a concrete arithmetic error, so the overall claim should be read with that caveat.\n\nThe new thing here is the four-way OOD taxonomy—query variations, unseen query types, unseen tasks, corpus expansion—applied to generative retrievers rather than to dense retrieval or rankers, which is what earlier robustness studies covered. The experimental design is straightforward: BART and CorpusBrain versus BM25 and DPR on KILT. The query-variation results are internally consistent and clear: generative models collapse under misspellings and word-order changes, and CorpusBrain is more resilient than BART. The unseen-task results are also coherent: all neural models lose substantially when transferred across KILT tasks, with generative models handling slot-filling better than entity linking. These two legs alone justify the paper's abstract claim that generative IR robustness is in need of improvement.\n\nThe soft spots are all in the corpus-expansion scenario. Eq. (6) defines PIID as the model trained on the training set corresponding to the test set. For Q1, that means a model trained on the full corpus, not the D0-filtered set. The paper instead uses the D0-trained model on Q0, so the reported DROOD mixes query difficulty, corpus size, and missing supervised signal. The construction of Q1 makes it worse: queries with relevant documents only partly in D1 are included, so a model can score via D0 documents. Then there is a plain numeric error in Table 7: BM25 on fact checking drops from 46.5 to 37.8, which is −18.7%, not −6.0%. The authors appear to have used the QA baseline by mistake. The text also overstates the generative advantage in this scenario: it says both BART and CorpusBrain beat DPR, but Table 7 shows CorpusBrain's average drop is worse than DPR's.\n\nThe other limitations are minor for a first study: two generative models and one benchmark, no error bars, and a random 60/40 split as a proxy for corpus expansion. The authors should rerun the DROOD calculations with the proper PIID, fix the arithmetic, and soften the narrative. Then the paper would be a solid empirical contribution.\n\nI'd send it to peer review with a request for major revision on the corpus-expansion section. The query-variation and unseen-task results are worth refereeing, and the taxonomy is useful for future work.","headline":"First systematic OOD evaluation of generative IR models, but the corpus-expansion analysis has a baseline mismatch and a numeric error that need fixing before the paper's broader claims.","tokens_in":14805,"tokens_out":6480,"would_cite":false,"duration_ms":79006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generative information retrieval models, despite beating dense and sparse baselines under identical distributions, suffer substantial performance drops on out-of-distribution queries, tasks, and corpora.","keywords":["generative information retrieval","out-of-distribution robustness","query variations","corpus expansion","dense retrieval","KILT benchmark","DROOD"],"falsifier":"Run a controlled corpus-expansion experiment in which new documents about a sampled set of topics are added while keeping the query types and difficulty of the test queries identical to the training queries; if previously trained generative models retrieve the new documents without a significant DROOD drop, the paper's conclusion that generative IR models perform poorly on corpus expansion would not transfer to that setting.","tokens_in":1736,"feed_emoji":"📉","tokens_out":2574,"duration_ms":45732,"temperature":0.7,"pith_summary":"This paper claims that generative information retrieval models, which retrieve documents by directly generating their identifiers, generalize poorly to out-of-distribution (OOD) data even though they perform well when test data resembles training data. The authors define OOD robustness through four shifts: query variations, unseen query types, unseen tasks, and corpus expansion. They evaluate two representative generative models (CorpusBrain and BART) against a dense retriever (DPR) and a sparse retriever (BM25) on the KILT benchmark. Across the four scenarios, generative models show large performance drops, especially on misspellings and word-order changes, indicating that leaderboard scores measured under identical distributions overstate the reliability of these models.","feed_headline":"Generative IR models fail out-of-distribution tests","feed_subtitle":"Strong IID leaderboard scores hide sharp drops under query shifts, new tasks, and corpus growth.","key_machinery":"The analysis is carried by four formal OOD settings defined in Section 4, each expressed as an expectation over a shifted distribution: query variations $\\mathcal{G}(q_t)$, unseen query types from $G'_Q$, unseen tasks from $\\tilde{G}_T$, and corpus expansion $D_n$. The quantitative carrier is DROOD, the percentage change from IID R-precision $P_{IID}$ to OOD R-precision $P_{OOD}$, defined as $\\text{DROOD} = (P_{OOD} - P_{IID}) / P_{IID}$, with R-precision being the ratio of relevant pages retrieved in the top-$R$ results. For corpus expansion, the experimental machinery is a random 60/40 split of Wikipedia pages into an initial corpus and an incremental corpus, with KILT queries filtered so that all relevant documents lie exclusively in one side of the split.","core_discovery":"The paper establishes that, overall, generative IR models are not robust to distribution shifts: on query variations such as misspelling and reordering, R-precision drops by up to 74.0% for BART and 68.0% for CorpusBrain on the FEVER dataset, and on unseen query types and unseen tasks both generative models degrade substantially, often more than BM25. In the corpus-expansion scenario, generative models can lose more than half their effectiveness, with CorpusBrain falling 58.5% on the dialogue task when the corpus grows, although they still keep a higher absolute R-precision than the baselines. The authors attribute this residual advantage to generative decoding over a prefix tree, which lets beam search probe document identifiers near those seen during training. The overall conclusion is that the strong in-distribution performance of generative retrievers does not transfer to out-of-distribution conditions and that this weakness deserves dedicated study.","pith_inferences":["A natural extension the authors do not pursue is to use naturally occurring query reformulations, typos, and new web queries rather than the four synthetic perturbation types; such data would test whether the measured fragility is an artifact of the generators.","The corpus-expansion result raises the possibility that the random 60/40 Wikipedia split confounds document growth with query difficulty, since filtering queries by whether their evidence pages fall in the initial corpus can change the difficulty distribution of the test set itself.","One concrete testable prediction from the prefix-tree explanation is that generative models should degrade gracefully when new documents are added in topics close to existing ones, but sharply on unrelated topics; ranking OOD difficulty by semantic distance to the training corpus would check this.","If robustness becomes a standard reporting requirement, model selection in generative IR is likely to shift from maximizing in-distribution accuracy toward optimizing a trade-off between accuracy and DROOD, similar to the effectiveness-robustness trade-off already studied in neural ranking models."],"forward_implications":["If the paper is right, IID leaderboard performance on KILT cannot be used as evidence that a generative retriever will behave reliably when deployed on new query phrasings, new question types, new tasks, or an expanding corpus.","Generative IR models should be evaluated with OOD-aware metrics such as DROOD alongside standard R-precision; a model that scores high only in-distribution should not be preferred over a more stable one.","The four-scenario taxonomy gives a reusable protocol for auditing future generative IR models, so that robustness claims can be compared across models on the same set of shifts.","Because generative models are particularly fragile to misspelling and word-order changes, improving surface-form robustness of the query-to-docid mapping is a concrete next step suggested directly by the findings.","The observation that generative models retain absolute advantages on corpus expansion, attributed to prefix-tree beam search, implies that the decoding structure itself should be examined as a robustness mechanism rather than treated as a fixed component."],"supporting_citations":[{"why":"Supplies the KILT benchmark with its five tasks and multiple datasets that the paper uses to construct all four OOD scenarios.","marker":"[39]"},{"why":"Provides CorpusBrain, the pre-trained generative IR model whose IID and OOD performance is analyzed throughout the paper.","marker":"[3]"},{"why":"Provides BART, the Seq2Seq backbone fine-tuned as a second representative generative IR model.","marker":"[23]"},{"why":"Provides DPR, the representative dense retrieval model that serves as the main comparison baseline in the OOD experiments.","marker":"[19]"},{"why":"Provides BM25, the sparse retrieval baseline that does not learn from the training data and is used as a reference point in every OOD scenario.","marker":"[42]"},{"why":"Supplies the query variation generators (misspelling, naturality, order, paraphrasing) used to instantiate the query-variations OOD scenario.","marker":"[38]"},{"why":"Supplies the DROOD metric and the general framing of measuring OOD robustness by the drop from in-distribution performance.","marker":"[49]"},{"why":"Defines the generative retrieval paradigm of predicting document identifiers, the object of study the paper subjects to OOD analysis.","marker":"[35]"},{"why":"Supplies the counter-fitted word embeddings used by the paraphrasing variation generator, which is one of the four query perturbation types.","marker":"[36]"}],"fun_headline_variants":["Generative IR models fail out-of-distribution challenges","GenIR models crumble under distribution shifts","OOD stress test exposes generative IR's weakness","New tasks and queries trip up generative retrievers","Generative IR's OOD generalization is a weak link"],"cache_read_input_tokens":16896,"weakest_assumption_plain":"The constructed KILT scenarios in Section 5.1 faithfully realize the Section 4 OOD definitions, particularly the random 60/40 Wikipedia split for corpus expansion, which may entangle document growth with query difficulty.","fun_headline_variants_meta":{"raw":{"variants":["Generative IR models fail out-of-distribution challenges","GenIR models crumble under distribution shifts","OOD stress test exposes generative IR's weakness","New tasks and queries trip up generative retrievers","Generative IR's OOD generalization is a weak link"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1305,"prompt_tokens":897,"completion_tokens":408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":335}},"tokens_in":513,"tokens_out":408,"duration_ms":4436,"temperature":1.0,"reasoning_tokens":335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:29:31.053188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled corpus-expansion experiment in which new documents about a sampled set of topics are added while keeping the query types and difficulty of the test queries identical to the training queries; if previously trained generative models retrieve the new documents without a significant DROOD drop, the paper's conclusion that generative IR models perform poorly on corpus expansion would not transfer to that setting.","supporting_citations":[{"cited_title":", De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., Plachouras, V ., Rocktäschel, T., Riedel, S.: KILT: A benchmark for knowledge intensive la nguage tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the KILT benchmark with its five tasks and multiple datasets that the paper uses to construct all four OOD scenarios."},{"cited_title":"In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp","cited_arxiv_id":null,"evidence_quote":"Provides CorpusBrain, the pre-trained generative IR model whose IID and OOD performance is analyzed throughout the paper."},{"cited_title":"In: Proceedings of the 58th Annual Meeting of the Associatio n for Compu- tational Linguistics, pp","cited_arxiv_id":null,"evidence_quote":"Provides BART, the Seq2Seq backbone fine-tuned as a second representative generative IR model."},{"cited_title":"In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp","cited_arxiv_id":null,"evidence_quote":"Provides DPR, the representative dense retrieval model that serves as the main comparison baseline in the OOD experiments."},{"cited_title":"Foundations and Trends in Information Retr ieval 3(4), 333–389 (2009)","cited_arxiv_id":null,"evidence_quote":"Provides BM25, the sparse retrieval baseline that does not learn from the training data and is used as a reference point in every OOD scenario."},{"cited_title":"In: Advances in Information Re- trieval: 44th European Conference on IR Research, ECIR 2022 , Stavanger, Norway, April 10–14, 2022, Proceedings, Part I, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the query variation generators (misspelling, naturality, order, paraphrasing) used to instantiate the query-variations OOD scenario."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DROOD metric and the general framing of measuring OOD robustness by the drop from in-distribution performance."},{"cited_title":"ACM SIGIR Forum 55(1), 1–27 (2021)","cited_arxiv_id":null,"evidence_quote":"Defines the generative retrieval paradigm of predicting document identifiers, the object of study the paper subjects to OOD analysis."},{"cited_title":"In: Proceedings of the 2 016 Conference of the North American Chapter of the Association for Computati onal Linguis- tics: Human Language Technologies, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the counter-fitted word embeddings used by the paraphrasing variation generator, which is one of the four query perturbation types."}],"review_version":1}