{"id":"ef2b5ef6-1fcc-4c14-b871-213e3b6072ee","arxiv_id":"2501.17981","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-3.5 generated query variants from information-need descriptions that overlap substantially with human variants in retrieved relevant documents, up to 71.1% at pool depth 100.","lead":"This paper tested whether a large language model (GPT-3.5) can write the many different search queries that people type for the same information need. The AI-generated queries retrieved up to 71.1% of the same relevant documents as human queries at pool depth 100, suggesting LLMs could help build search test collections more cheaply.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 71.1% relevant-document overlap is computed over UQV100 judgments that omit up to 37% of GPT-pool documents; until those unjudged docs are judged, the headline number is not a verified overlap.","rationale":"The central claim is an empirical overlap estimate, and its most load-bearing assumption is that the UQV100 qrels are sufficient to determine which documents in the GPT pools are relevant. The paper's own Table 2 shows a large asymmetry in unjudged rates, so the qrels are not neutral ground truth for comparing human and GPT pools. The overlap figure is restricted to judged relevant documents; GPT-only relevant documents are invisible, making the headline number conditional on human-built judgments. The correct fix is a judgment-expansion study, which is exactly the condition the reader's verdict requires. I also considered whether using a single BM25 retriever undermines the pooling conclusion, but the comparison is internal to one retriever, and the broader 'replacement' claim is already conditional; the qrels gap is the more direct threat to the 71.1% number. The paper is otherwise transparent: prompts and generated sets are publicly available, the prompting example topic is excluded, and significance testing is reported. The reader's CONDITIONAL verdict remains appropriate.","tokens_in":8086,"tokens_out":9598,"duration_ms":101793,"concrete_test":"Judge a stratified random sample (e.g., 200 documents) of the previously unjudged documents in the GPT temp=1.0 depth-100 pools, using the original UQV100 judging instructions and including documents retrieved by GPT variants but absent from the qrels; then recompute the relevant-document Jaccard with the expanded judgment set. If the overlap falls materially below 71.1% (say, below 60%) or rises above 80%, the current headline is not robust; if it remains within a few points, the qrels gap is not a serious threat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (Section 3.2, Figure 4) is an average Jaccard overlap between relevant documents retrieved by human and GPT query sets, computed with UQV100 relevance judgments as ground truth. Table 2 shows GPT pools are 31–37% unjudged versus 13% for human pools. A relevant document retrieved only by GPT is absent from the qrels and thus cannot enter the relevant union. The reported 71.1% therefore measures overlap only over already-judged relevant documents, not the true overlap of relevant documents; if missing GPT-only documents are relevant, the true overlap is lower. The authors acknowledge the unjudged portion only as future work ('It would be interesting to further investigate the unjudged portion'), yet the conclusion that GPT variants 'show potential for replacing human query variants' depends on whether GPT queries surface relevant documents that human-built qrels missed. Until those unjudged documents are judged, the central empirical claim is conditional on incomplete ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an exploratory study of whether GPT-3.5 (text-davinci-003) prompted with an information-need backstory can generate query variants comparable to the human query variants in the UQV100 test collection. For RQ1, the authors measure lexical overlap between GPT- and human-generated query sets under progressively relaxed matching conditions, reporting Jaccard values up to 13.5% and coverage of up to 18.7%. For RQ2, they run BM25 with both query sets on ClueWeb12-B and compare the resulting document pools, reporting that when only judged relevant documents are considered, GPT pools overlap with human pools up to 71.1% at depth 100. They conclude that LLM-generated variants show potential for replacing human variants during pool construction.","tokens_in":8264,"tokens_out":9034,"duration_ms":91123,"significance":"The potential contribution is real: if LLM-generated query variants can produce document pools close to human-produced pools, test-collection construction and expansion could become substantially cheaper. The paper has concrete strengths: it uses a public corpus and test collection, gives the prompt template and generated queries, reports multiple complementary measures, and transparently notes limitations of the human reference set. The main technical risks are the interpretation of the 71.1% relevant-document overlap in the presence of incomplete relevance judgments, and the use of a symmetric Jaccard metric when the compared pools have very different sizes. These issues are addressable with additional analysis rather than an outright rejection.","major_comments":[{"comment":"The headline 71.1% relevant-document overlap is computed over UQV100 relevance judgments, but Table 2 shows that at depth 10 the GPT-generated pools contain 31–37% unjudged documents compared with 13% for the human pool; unjudged documents are absent from the qrels and are effectively treated as non-relevant in the Jaccard calculation. The manuscript itself labels the unjudged portion as future work ('It would be interesting to further investigate the unjudged portion'), yet Section 4 concludes that GPT 'shows potential for replacing human query variants' on the basis of this overlap. Because the true overlap could differ materially if GPT-only unjudged documents are relevant, the authors should either augment the qrels by judging a sample of GPT-only pool documents (at least at depth 100), or provide a sensitivity analysis that bounds the result under extreme assumptions about the unjudged documents, and adjust the conclusion accordingly.","section":"§3.2, Figure 4, Table 2"},{"comment":"The relevant-document Jaccard is symmetric, but the human and GPT pools are very different in size: at depth 10 the human pool averages 190.69 documents per topic, while the GPT pools average 93.55–105.21 documents (Table 2). A smaller GPT pool that is mostly a subset of the larger human pool can therefore achieve a high Jaccard while missing a substantial share of the human-discovered relevant documents. To support the claim that GPT variants 'generate similar sets of relevant documents,' the paper should also report asymmetric coverage measures, for example the fraction of human-pool relevant documents retrieved by the GPT pool and the fraction of GPT-pool relevant documents retrieved by the human pool, at the depths used in Figure 4.","section":"§2.3, §3.2, Table 2"},{"comment":"The study does not provide a non-human baseline for the document-overlap results. Without a baseline such as the original TREC topic titles, a random sample of topically related query variants, or the system-pool-only condition, it is difficult to know how much of the 71.1% overlap is attributable to the LLM's query-generation quality and how much would be achieved by any reasonable set of related queries. Adding such a baseline would considerably strengthen the 'potential for replacing human query variants' conclusion.","section":"§3.2, §4"}],"minor_comments":[{"comment":"The axis label '/Q_uery Transformations' appears to contain a typo; it should read 'Query Transformations'.","section":"Figure 3"},{"comment":"The notation 'RBP 0.406 +0.111' is unexplained; if the second value is the residual or the standard deviation, please define it, and also define the 'Relevant' and 'Unjudged' columns (proportions of pool documents that are judged relevant and unjudged, respectively).","section":"Table 2"},{"comment":"The statement that few-shot learning was not used because 'we were limited by the number of available backstories' is unclear, since few-shot prompting does not require additional backstories; please clarify the actual constraint.","section":"§2.1"},{"comment":"The citation [12] is used to support the claim that similar findings are reported for 'queries generated to find additional information in response to document summaries,' but the reference points to the CC-News-En corpus paper, which does not obviously contain such an analysis; please verify the citation target.","section":"§1, References"},{"comment":"The sentence beginning 'While we are interested in measuring the overall overlap...' lacks its contrast clause; rephrase for readability.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"The self-referential element is real but mild: two of the authors are also co-creators of UQV100, so the comparison against the human-generated query set is partly in-house. This does not invalidate the study, but it increases the importance of the requested sensitivity analysis and external validation. In terms of scope, this is a short exploratory SIGIR paper, and my requested changes concern the interpretation of the headline metric rather than a wholesale re-run of the experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid 5-page exploratory note, not a paradigm change. The new thing is the direct comparison of LLM-generated query variants from backstories with human variants for pooling; prior synthetic query work generated queries from documents, not from information-need backstories. The paper is reproducible: prompt template and GPT query sets are on GitHub, and the runs are Anserini BM25 with stated parameters on ClueWeb12-B.\n\nWhat it does well: RQs are crisp, metrics are sensible, and the key result—up to 71.1% relevant-document pool overlap at depth 100—is computed on a public collection with public judgments. The paper is honest about the human set being non-exhaustive and about the GPT pools having more unjudged documents.\n\nSoft spots: the 71.1% is an overlap over already-judged relevant documents. Table 2 shows GPT pools are 31–37% unjudged at depth 10 versus 13% for humans, so any relevant document retrieved only by GPT never enters the numerator or denominator. The paper mentions this in Section 3.2, but the abstract and conclusion still say \"replacing human query variants.\" That is too strong; the evidence supports \"can supplement\" or \"can be used to expand pools,\" not replacement, until the unjudged portion is judged and the overlap recomputed. Judging a sample of GPT-only documents would settle it. Missing confidence intervals around the overlap curves is minor for a 5-page paper but would help. Also, the reference standard UQV100 was built by two of the authors; that is a situational note, not a flaw.\n\nBottom line: the exploratory result is real and worth engaging. The paper deserves a serious referee; an editor should send it out. For the authors: soften the replacement wording, add CIs if possible, and ideally judge a sample of unjudged GPT documents. I would bring it to a reading group focused on IR evaluation.","headline":"A clean, reproducible exploratory study showing GPT-3.5 query variants overlap human variants on relevant documents up to 71.1% at depth 100, with the main caveat that the headline number is computed over incomplete judgments.","tokens_in":8791,"tokens_out":2103,"would_cite":true,"duration_ms":21730,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that one-shot GPT-3.5 prompting generates query variants whose judged relevant document pools overlap human-generated pools by up to 71.1% at depth 100.","keywords":["information retrieval","test collections","query variants","large language models","document pooling","relevance judgments","one-shot learning"],"falsifier":"Judge all (or a random sample of) the documents retrieved only by the GPT-3.5 query sets, then recompute the relevant-document overlap; if a substantial share of those unjudged documents turn out to be relevant, the claimed 71.1% overlap with human pools is not the full picture.","tokens_in":7890,"feed_emoji":"🤖","tokens_out":9267,"duration_ms":85310,"temperature":0.7,"pith_summary":"Information retrieval test collections are normally built by having crowdsourced workers write many query variants for each topic, which is expensive and hard to scale. This paper asks whether a large language model prompted once with a topic backstory can generate query variants that do the same job, using UQV100, a test collection of one hundred backstories with crowdsourced query variants, as the reference. It compares GPT-3.5 variants against the UQV100 human variants over the collection's corpus, first on similarity of the query strings and then on similarity of the document pools they retrieve when passed through a standard BM25 ranker. The central result is that lexical similarity is modest (Jaccard 7.1–13.5%, coverage up to 18.7%), but judged relevant-document overlap reaches 43.7% at depth 10 and 71.1% at depth 100. The authors conclude that LLM-generated variants could serve as a low-cost complement to human query pooling when building test collections.","feed_headline":"LLM query variants match 71% of human relevant documents","feed_subtitle":"One-shot GPT-3.5 prompts could build search test-collection pools without crowdsourcing.","key_machinery":"The key object is the query variant set: alternative phrasings of a single information need, generated here by in-context learning, where GPT-3.5 receives a task description, one worked example backstory with human queries, and a new backstory. The comparison machinery is the document pool: for each query set, the union of BM25 results across a topic's variants is measured at increasing depths, with overlap computed as a Jaccard index on all documents and again on judged-relevant documents only. Rank-biased overlap quantifies within-set consistency, and pool growth curves measure how quickly additional variants add new documents.","core_discovery":"On the paper's own terms, the central discovery is that the gap between human and machine query variants is much smaller at the level of retrieved relevant documents than at the level of query wording. A one-shot prompt to GPT-3.5, given the same one hundred backstories used in UQV100, produced about 27–37 variants per topic compared with 57 for the human set, and the generated query strings were only 7.1–13.5% Jaccard-similar to human queries depending on how strictly matches were required. Yet when the variants were run through BM25 and the judged relevant documents were compared, the pools overlapped up to 71.1% at depth 100. The paper interprets this as evidence that generative LLMs can contribute to document pooling for test collection construction, with the caveat that GPT pools contain a larger share of unjudged documents, so the true relevant overlap may not be fully captured by existing judgments.","pith_inferences":["Human evaluation of whether GPT-generated variants read as natural phrasings of a backstory would separate 'looks human' from 'retrieves like human'; the paper leaves that evaluation to future work.","Temperature 1.0 gave the highest relevant-document overlap and shorter queries, suggesting that more exploratory sampling might approximate human query diversity better; sampling multiple completions per backstory is a cheap way to test this.","Testing the same one-shot template on other LLMs and other topic sets would show whether the 71.1% overlap reflects one-shot prompting in general or something specific to this model and collection."],"forward_implications":["If the 71.1% relevant-document overlap holds, LLM-generated query variants could replace a substantial part of the crowdsourcing effort in building new test collection pools.","TREC-style collections that already have backstory or topic statements could be expanded cheaply by prompting an LLM, since no human query generation would be needed for the new variants.","The lower pool growth and higher within-set consistency of GPT variants imply they sample a narrower diversity of queries, so they are likely a complement rather than a full replacement for human variants.","Because GPT pools contain 31–37% unjudged documents, judging those extras is the direct next step; if they contain relevant documents, the current overlap figure underestimates the GPT set's utility."],"supporting_citations":[{"why":"Supplies the UQV100 human query variants, backstories, and relevance judgments that define the reference set for all comparisons.","marker":"[2]"},{"why":"Establishes the in-context learning paradigm used to prompt GPT-3.5 with one example before generating variants.","marker":"[5]"},{"why":"Provides the pool-size growth method and the argument that query variants widen pools like systems do, motivating the pooling comparison.","marker":"[14]"},{"why":"Supplies the pooling methodology for building test collections, which the paper extends by testing LLM-generated queries as a source of pool documents.","marker":"[19]"},{"why":"Provides the BM25 retrieval implementation used to generate document runs for both human and GPT query sets.","marker":"[21]"}],"fun_headline_variants":["LLM queries: different words, 71% same relevant docs","AI query variants match human relevance up to 71.1%","GPT-3.5 query pools reach 71% human relevant overlap","LLM-generated queries yield 71% overlap in relevant documents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats the UQV100 relevance judgments as the ground truth for relevant documents, so any relevant document retrieved only by a GPT query and never judged is invisible in the 71.1% overlap figure.","fun_headline_variants_meta":{"raw":{"variants":["LLM queries: different words, 71% same relevant docs","AI query variants match human relevance up to 71.1%","GPT-3.5 query pools reach 71% human relevant overlap","LLM-generated queries yield 71% overlap in relevant documents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1498,"prompt_tokens":844,"completion_tokens":654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":579}},"tokens_in":460,"tokens_out":654,"duration_ms":6464,"temperature":1.0,"reasoning_tokens":579,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:25:45.262379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Judge all (or a random sample of) the documents retrieved only by the GPT-3.5 query sets, then recompute the relevant-document overlap; if a substantial share of those unjudged documents turn out to be relevant, the claimed 71.1% overlap with human pools is not the full picture.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the in-context learning paradigm used to prompt GPT-3.5 with one example before generating variants."},{"cited_title":"Voorhees","cited_arxiv_id":null,"evidence_quote":"Supplies the pooling methodology for building test collections, which the paper extends by testing LLM-generated queries as a source of pool documents."}],"review_version":1}