{"id":"8ff38bc9-a76f-4ecd-a725-84aecbfc0959","arxiv_id":"2411.14914","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-generated Boolean queries for systematic reviews are unstable across seeds, and the original ChatGPT results could not be reproduced with the documented setup.","lead":"This study reruns LLM-based Boolean query generation for systematic reviews and finds that results vary widely across runs and cannot reliably reproduce an earlier highly cited study. It also benchmarks open-source models and catalogs practical failure modes, which matters for anyone building or relying on automated literature search.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ1's 'neither reproducible nor reliable' claim rests on pooled variance that may reflect topic heterogeneity, not seed variation; the paper never isolates within-topic seed variance.","rationale":"The paper is transparent about its pipeline, datasets, and limitations, and its documentation of reproducibility problems in prior work is valuable. The RQ1 conclusion, however, is the paper's central empirical claim, and it is supported by standard deviations that pool across topics and seeds. Because topics in Seed and CLEF TAR vary widely in difficulty, a large pooled SD can arise even when seed-to-seed variation for a fixed topic is small. The paper does not report a variance decomposition, so the specific mechanism named in the claim—'rerunning the same prompts with different seeds'—is not isolated. This is an internal methodological gap rather than a disagreement with consensus. The reader's identified weakness about GPT-3.5 inference and prompt adaptation is a valid separate confound for the comparison against Wang et al., which is why my agreement is partial rather than full; but the variance-decomposition issue is more load-bearing because it targets the direct evidence for RQ1. The proposed reanalysis of the released code would settle the issue. Since the finding may survive that reanalysis, and since the reader already issued a conditional verdict, I recommend no change to the verdict.","tokens_in":19023,"tokens_out":4592,"duration_ms":46367,"concrete_test":"Using the released repository (Section 1), recompute results for Seed q1 and q4 for GPT-3.5-1106: for each of the ~40 topics, calculate precision and recall for each of the 10 seeds, then the topic-level SD across seeds. Report the mean of those within-topic SDs against the pooled SD in Table 1. Also rerun 3 topics with the same seed 5 times each to check API-level determinism. If mean within-topic SD is much smaller than pooled SD, the headline RQ1 variance evidence collapses to topic heterogeneity and must be revised; if it remains large, the reliability claim is directly supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central answer to RQ1 (Section 6) is that rerunning the same prompts with different seeds yields high variance and thus unreliable queries. The supporting evidence is the standard deviations in Tables 1–3 and the boxplots in Section 4.2. In every reported table, each model's standard deviation is computed over the pooled set of topics and seeds (e.g., 'average Precision scores with standard deviation for each model'), and Section 4.2 says it compares 'average Recall and Precision scores across topics and random seeds.' Pooling conflates two distinct sources: (a) topics differ intrinsically in difficulty, and (b) seeds cause query-to-query variation for the same topic. If topic difficulty dominates, the large SDs would appear even with a perfectly deterministic generator, and they would not support the claim that 'rerunning the same prompts with different seeds' is unreliable. The authors do not report the variance decomposition (e.g., average within-topic SD across seeds), so the load-bearing empirical assertion is not yet established. The reader's concern about adapted prompts and inferred GPT-3.5 is a separate confound for reproducing Wang et al., but it does not address this internal variance issue. A focused reanalysis of the released code would settle it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a reproducibility and generalizability study of LLM-based Boolean query generation for systematic reviews, reproducing and extending Wang et al. (2023) and Alaniz et al. (2023). The authors implement a pipeline that prompts GPT-3.5, GPT-4, Mistral, and Zephyr models, retrieves PubMed results, and evaluates precision, recall, F1, and F3 on the Seed and CLEF TAR collections across multiple seeds. They report that they cannot reproduce the numeric results of Wang et al., find high variance when re-running the same prompts with different seeds, compare open-source and commercial models, and conduct a failure analysis of malformed queries and dataset issues. The paper concludes that LLM-generated Boolean queries in their current form are not reliable enough for systematic review automation.","tokens_in":19321,"tokens_out":5406,"duration_ms":45468,"significance":"If the central claim holds, the paper provides an important cautionary result for a highly cited work, with practical implications for any pipeline that uses LLMs to generate search strategies. The authors ship code, use repeated seeded runs, report standard deviations, and quantify formatting errors, which are concrete strengths. However, the RQ1 claim is currently under-supported because the reported variability pools topic and seed effects, and the reproduction comparison relies on inferred model versions and adapted prompting. These issues are addressable with reanalysis and a more careful framing, so the study makes a useful contribution despite these gaps.","major_comments":[{"comment":"The standard deviations in Tables 1-3 and the boxplots in Section 4.2 are computed over the pooled set of topics and random seeds (Section 4.2: \"average Recall and Precision scores across topics and random seeds\"). This pooling conflates topic difficulty with seed variability, so the large SDs do not by themselves demonstrate that \"rerunning the same prompts with different seeds\" is unreliable, as claimed in the RQ1 answer in Section 6. A deterministic model that is very good on some topics and poor on others would yield the same pooled SD. Please report a variance decomposition, such as the mean or median of within-topic standard deviations across seeds, or a mixed-effects model with topic as a random factor, to isolate the seed contribution.","section":"Section 4.2, Tables 1-3"},{"comment":"The comparison to Wang et al. rests on two unverified assumptions: that the original ChatGPT experiments used GPT-3.5, inferred from the January 2023 submission deadline (Section 3.3), and that the adapted prompts, JSON return mode, and system/user prompt split preserve the original query generation behavior (Section 3.4). Since Section 5.1 itself states that the original prompt-issuing and query-extraction procedures are not described, the failure to reproduce Wang et al.'s numbers could be caused by these setup differences rather than by inherent LLM unreliability. Please reframe the RQ1 claim as conditional on these assumptions, or add a sensitivity analysis using free-text output without JSON and with the original prompt as a single user message.","section":"Section 3.3-3.4"},{"comment":"The text in Section 3.5 says \"Thus we could not run the relevant example queries q4 and q5 for the CLEF TAR dataset,\" yet Tables 1-3 list q4-HQE and q5-HQE rows for CLEF TAR with values. This is a direct internal contradiction. Please clarify whether the reported CLEF TAR q4/q5 values come from a different example selection or were obtained after the statement was written, and correct the text or the tables accordingly.","section":"Section 3.5, Tables 1-3"},{"comment":"For CLEF TAR, the paper copies the baseline results from Wang et al. rather than reproducing them, because the original queries are not available in a PubMed-compatible format (Section 3.2). As a result, the CLEF TAR comparison cannot independently validate either the baseline or Wang et al.'s query results. Please either obtain and translate the original CLEF TAR queries (e.g., via the Polyglot tool the authors already tried) and report the reproduced baselines, or explicitly label the CLEF TAR comparison as dependent on the original paper's numbers and therefore not a full reproduction.","section":"Section 3.2"},{"comment":"The one-sample t-test against the single reported value from Wang et al. is used to mark results with a dagger throughout Tables 1-3. This test treats Wang et al.'s point estimate as a fixed population value, ignores the fact that Wang et al. also ran only one sample (or a small number) of queries, and does not correct for the large number of comparisons across models and query types. Please report effect sizes or confidence intervals and apply a multiple-comparison correction, or remove the significance annotations and rely on descriptive comparisons.","section":"Section 4.1, Tables 1-3"}],"minor_comments":[{"comment":"There is a typesetting error in the model list (\"Mistral-tiny (Mistral-7B-v0.2) and Mistral-small (Mixtral-8X7B-v0.1)8:.\"), with a stray colon and number after the closing parenthesis.","section":"Section 3.3"},{"comment":"The first model row is labeled \"GPT3.5-turbo,\" which is not among the API models listed in Section 3.3; please clarify which model this row refers to.","section":"Table 4"},{"comment":"The statement \"for six out of eight query types on the Seed dataset, we couldn't reproduce the original results\" is followed later in the same section by \"none of our models reached the scores reported by Wang et al.,\" which is stronger than the six-out-of-eight formulation; please reconcile the two statements.","section":"Section 4.1"},{"comment":"The duplicate-topic analysis would benefit from a small table or list of the identified duplicate pairs (e.g., 43/96, 7/67, 8/112) with their numbers of shared and disjoint relevant documents, since the current prose is dense and hard to verify.","section":"Section 5.3.1"},{"comment":"The phrase \"In this Section, we summarize\" uses an unnecessary capitalization of \"Section\"; please use lowercase.","section":"Section 6"},{"comment":"The word \"influental\" should be \"influential.\"","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of SIGIR-AP as a reproducibility study. The main concern is that the RQ1 claim requires a focused reanalysis of the released code to separate seed variance from topic variance; this is a feasible revision rather than a fundamental flaw. No concerns about citation practices or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a worthwhile replication study with extensions. The broad finding—LLM-generated Boolean queries for systematic reviews are unstable across reruns, and Wang et al.'s exact numbers don't reproduce—is directionally credible. But the specific claim that rerunning the same prompts with different seeds causes high variance is not cleanly supported by the tables. The standard deviations in Tables 1–3 pool across topics and seeds, so they conflate topic difficulty with seed instability. The paper never reports within-topic variance across seeds, which is the load-bearing evidence needed for RQ1. That's a genuine gap, not a fatal one.\n\nWhat's actually new and useful: the open-source model comparison, the error analysis (parentheses, JSON failures, term repetition), the dataset deduplication audit (Seed has 34 unique SLRs, not 40), and the honest documentation of tricky assumptions—the inferred GPT-3.5 version, the adapted prompts, the JSON return mode. Code and data are released, and the cost transparency ($150 total) is a nice touch. The paper does well at cataloging the practical pitfalls of reproducing LLM-based IR experiments.\n\nSoft spots: besides the variance decomposition issue, the reproduction comparison rests on an inferred model version and on prompts that differ from the original in non-trivial ways. The authors acknowledge this, but it means the failure to reproduce could partly reflect setup differences. They also copied CLEF TAR baselines rather than reconstructing them. The t-tests don't correct for multiple comparisons, though that's minor for a descriptive replication study. None of this undermines the core empirical contribution, but it does mean the strong wording 'neither reproducible nor reliable' is a bit ahead of the evidence.\n\nWho this is for: anyone planning to build on Wang et al. or use LLMs for systematic review query generation. It will save people time and wasted API spend. It deserves a serious referee, and the fix is straightforward: reanalyze the released code to produce within-topic seed variance, then restate RQ1's conclusion in proportion to that evidence.","headline":"A useful replication-with-extension study; the seed-variance claim is real but under-supported because the reported standard deviations pool topic and seed variation rather than isolating within-topic seed instability.","tokens_in":19732,"tokens_out":2079,"would_cite":true,"duration_ms":22102,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that Boolean queries generated by large language models for systematic reviews are not reproducible: rerunning identical prompts with different random seeds produces high-variance results, so the original reported scores…","keywords":["systematic reviews","Boolean query generation","large language models","reproducibility","query variability","retrieval evaluation","PubMed retrieval","open-weight LLMs"],"falsifier":"Audit the original experiments to establish the exact model version and prompt-handling procedure, then rerun each of the same prompts ten times with those exact settings on the same two collections. If the per-seed standard deviation of F1 turns out small (say under 0.05) and the original precision and recall values are recovered, the paper's central claim of unreliable, non-reproducible LLM query generation would be falsified.","tokens_in":18869,"feed_emoji":"🔁","tokens_out":6402,"duration_ms":59704,"temperature":0.7,"pith_summary":"This study tests a specific promise of generative AI: that a chatbot can draft the Boolean search queries used in systematic literature reviews. The authors reran a previously published LLM query-generation setup across two standard test collections and seven models, including open-weight alternatives, repeating every prompt with multiple random seeds. Their central finding is that the results are neither reproducible nor reliable: scores fluctuate strongly when the same prompt is rerun with a different seed, and none of the models reliably recovered the original paper's effectiveness numbers. If this finding holds, out-of-the-box LLMs are not yet safe to trust as an automatic component of systematic-review automation, and any single-run evaluation of such queries should be regarded as unverified. The paper also documents malformed outputs and dataset problems that make the original comparisons difficult to reconstruct.","feed_headline":"LLM Boolean queries fail the reproducibility test","feed_subtitle":"Rerunning identical prompts with different seeds gives high-variance results, so single-run scores are not stable.","key_machinery":"The central object is the reproducibility protocol: a fixed pipeline that feeds a systematic-review title to an LLM, parses the returned Boolean query from a structured JSON response, runs it against the PubMed database, and computes precision, recall, F1, and F3 against the collection's relevance judgments. The protocol's load-bearing move is repetition: each prompt is run with several random seeds (ten for the smaller models, five for GPT-3.5-0125 and GPT-4), so the spread of scores across seeds becomes the measured quantity. That spread, not any single score, is what the paper uses to argue that LLM query generation is currently unreliable.","core_discovery":"For systematic reviews, this paper reports that LLM-generated Boolean queries, evaluated by retrieving documents from PubMed and scoring precision, recall, F1, and F3 against known relevant sets, are too variable to reproduce. Re-running the same prompt with different seeds on the same topic produced large spread in every metric; on the main Seed test collection no tested model reached the effectiveness figures of the work being reproduced, while on the CLEF TAR collection the commercial models at times exceeded the reproduced precision values. The guided multi-step prompting approach deteriorated badly for several models, and across 11,200 generated queries 1,771 had incorrectly placed parentheses while a substantial share of guided runs yielded no valid query at all. The authors state plainly that none of the obtained results are reliable enough for the critical task of systematic reviews.","pith_inferences":["Editorial extension: if this variance result generalizes to other structured-generation tasks, then every published comparison of LLMs should report seed-to-seed variance rather than a single generated output.","Editorial extension: the duplicate and near-duplicate topics found in the Seed collection imply that some reported improvements on that benchmark may reflect the model echoing an example query that is effectively the answer key, rather than genuine query-writing skill.","Editorial extension: a focused follow-up could rerun the protocol with JSON-format output enforced for all models; the paper's own error analysis suggests that structured return modes eliminate a large class of extraction failures and may reduce the observed variance."],"forward_implications":["Researchers planning to automate systematic-review search should not yet replace expert-authored Boolean queries with a single LLM-generated query, because one run is effectively a random draw.","Numbers reported by the reproduced study should be treated as unstable benchmarks; future comparisons need repeated-seed means and standard deviations before claiming that one query generator outperforms another.","Open-weight models performed competitively with commercial models on some precision-oriented settings, but the same seed variance applies, so openness alone does not solve the reliability problem.","Detailed reporting of model version, prompt parsing, and dataset cleaning is necessary for any follow-up study; the paper shows that vague documentation made the original setup hard to reconstruct.","The small monetary cost of this repeated-seed study (about 150 USD in API calls) means that reproducibility testing is affordable and should become a standard practice."],"supporting_citations":[{"why":"The original ChatGPT Boolean-query generation study whose numerical results this paper tries to reproduce; it supplies the prompts, baselines, and evaluation approach.","marker":"[54]"},{"why":"The ChatGPT-4 medical query formulation study that this paper extends and critiques for lacking a reliability analysis.","marker":"[1]"},{"why":"The Seed collection used for evaluation, providing review topics, seed studies, expert Boolean queries, and relevance judgments.","marker":"[53]"},{"why":"The CLEF 2017 Technology Assisted Reviews overview, a source of the CLEF TAR collection material.","marker":"[23]"},{"why":"The CLEF 2018 Technology Assisted Reviews overview, the source of the review topics this paper uses.","marker":"[25]"},{"why":"Introduces the Mistral 7B open-weight model used as a local and API-based comparison model.","marker":"[19]"},{"why":"Introduces the Mixtral mixture-of-experts model used as the Mistral-small API model.","marker":"[20]"},{"why":"Sentence-BERT, used in the paper's pipeline to find the relevant example review queries from the Seed collection.","marker":"[36]"}],"fun_headline_variants":["LLM query generation too unreliable for systematic reviews","Study: LLM Boolean queries vary wildly across runs","Reproducing LLM query results fails on PubMed tests","LLM queries for reviews: high variance, low trust","Even ChatGPT can't reliably produce review queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reproduction depends on the assumption that the original study used the model version and prompt format the authors inferred; if the original setup actually differed, the failure to reproduce would reflect a changed setup rather than inherent LLM instability.","fun_headline_variants_meta":{"raw":{"variants":["LLM query generation too unreliable for systematic reviews","Study: LLM Boolean queries vary wildly across runs","Reproducing LLM query results fails on PubMed tests","LLM queries for reviews: high variance, low trust","Even ChatGPT can't reliably produce review queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1406,"prompt_tokens":984,"completion_tokens":422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":600,"tokens_out":422,"duration_ms":4415,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:43:15.640399+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the original experiments to establish the exact model version and prompt-handling procedure, then rerun each of the same prompts ten times with those exact settings on the same two collections. If the per-seed standard deviation of F1 turns out small (say under 0.05) and the original precision and recall values are recovered, the paper's central claim of unreliable, non-reproducible LLM query generation would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The ChatGPT-4 medical query formulation study that this paper extends and critiques for lacking a reliability analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The CLEF 2017 Technology Assisted Reviews overview, a source of the CLEF TAR collection material."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The CLEF 2018 Technology Assisted Reviews overview, the source of the review topics this paper uses."}],"review_version":1}