{"id":"a94c744a-c564-427b-931c-e89e0e1139c1","arxiv_id":"2506.04760","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Exp4Fuse improves sparse retrieval by fusing the ranked lists from the original query and an LLM-expanded query using a modified reciprocal rank fusion.","lead":"This paper presents Exp4Fuse, a search method that runs the same sparse retriever with the original query and with an LLM-expanded query, then merges the two ranked lists. A smart generalist might read it because it offers a low-cost way to improve search accuracy without heavy dense neural models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing controlled baselines: query2doc/LameR are never run on the same sparse retrievers, so the central claim of superiority is not established.","rationale":"The paper has real strengths: it releases code, tests multiple sparse retrievers, includes out-of-domain BEIR evaluation, a LLaMA3 generalizability table, and an ablation showing the QE route alone can hurt. Those do not, however, secure the abstract's comparative claim. The claim 'surpasses existing LLM-based query expansion methods in enhancing sparse retrievers' requires comparing Exp4Fuse against query2doc and LameR on the same sparse retriever, with the same LLM and retrieval settings. Section 4.1 lists BM25+query2doc and BM25+LameR as baselines, but Table 1 omits them; the only query2doc/LameR rows in Table 1 use SimLM, a dense retriever, so they cannot validate a sparse-retrieval claim. The absent rows cannot be assumed favorable to Exp4Fuse: published query2doc results on BM25 are typically much stronger than the +2.3 MRR gain shown for BM25+Exp4Fuse, so the burden is on the authors to supply the direct comparison. In addition, Table 4 shows the expanded route alone underperforms the original query, so the gain comes from fusion; comparing to standard RRF is necessary to know whether the modified weighting contributes anything. The reader's hyperparameter-selection concern remains valid and is part of the same reproducibility gap: with no held-out validation reported for k and lambda, even a fair baseline comparison would not fully prove generalizability. Verdict remains CONDITIONAL pending these controls.","tokens_in":12687,"tokens_out":5847,"duration_ms":67456,"concrete_test":"Re-run the Section 4.1 protocol on MS MARCO dev, DL19, and DL20, adding to Table 1 the missing rows: BM25+query2doc, BM25+LameR, docT5query+query2doc, docT5query+LameR, SPLADE++-v1+query2doc, SPLADE++-v1+LameR, and the same for SPLADE++-v2, using the same GPT4-mini generation and retrieval settings; also fuse the same two ranked lists with standard RRF (Eq. 2 with the n/10 term removed) as a control. If Exp4Fuse does not beat same-backbone query2doc/LameR variants, or if standard RRF matches it, the superiority claim and the modified fusion component lose support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Exp4Fuse 'surpasses existing LLM-based query expansion methods in enhancing sparse retrievers' (Abstract), and Section 4.1 promises comparisons with query2doc and LameR 'under identical experimental conditions.' However, Table 1 and Table 2 contain no rows for BM25+query2doc, BM25+LameR, SPLADE++-v1+query2doc, or SPLADE++-v1+LameR. The only query2doc/LameR rows use SimLM, a dense retriever, so they do not test 'enhancing sparse retrievers.' Without same-backbone controls, the observed advantage of Exp4Fuse+SPLADE++ over SimLM+query2doc/LameR is confounded by retriever strength rather than by the fusion framework. Additionally, Table 4 shows the LLM-expanded route alone degrades SPLADE++ (e.g., 67.8 vs 73.1 nDCG@10 on DL19), making standard RRF the necessary control to attribute gains to the modified fusion rule; this control is also absent. Until these rows are supplied, the strongest_claim is unverified rather than demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Exp4Fuse, a rank fusion framework that aims to improve sparse retrieval by combining two retrieval routes built on the same sparse retriever: one using the original query and one using a zero-shot LLM-generated hypothetical document appended to the original query. The two top-1000 ranked lists are fused using a modified reciprocal rank fusion (RRF) formula, which adds an occurrence-count term and per-list weights to the standard RRF score. The method is evaluated on MS MARCO dev, TREC DL 2019/2020, and seven BEIR datasets, using GPT-4-mini as the default LLM and LLaMA3-8B-Instruct for a generalizability check. The reported results show consistent improvements of the fused output over the original-query baseline across all tested sparse retrievers and datasets, with the largest gains on TREC DL 2019 and low-resource BEIR datasets. The paper also includes ablations on the number of route retrievals and on the necessity of individual routes.","tokens_in":12992,"tokens_out":4206,"duration_ms":45359,"significance":"If the claimed effect is real, Exp4Fuse offers a simple, low-cost way to leverage zero-shot LLM-based query expansion for sparse and learned-sparse retrievers, avoiding the heavy memory and latency costs of dense retriever pipelines. The strengths of the paper include a clear and reproducible framework, released code, use of public checkpoints for several baselines, consistent positive results across ten benchmarks, and a generalizability experiment with an open-source LLM. However, the central comparative claim—that Exp4Fuse 'surpasses existing LLM-based query expansion methods in enhancing sparse retrievers'—is not directly demonstrated, because the existing methods are not evaluated on the same sparse backbones. The absence of a standard-RRF control also leaves open whether the modified fusion formula, rather than fusion itself, is responsible for the gains.","major_comments":[{"comment":"The abstract and Section 1 claim that Exp4Fuse 'surpasses existing LLM-based query expansion methods in enhancing sparse retrievers,' and Section 4.1 states that query2doc and LameR are compared 'under identical experimental conditions.' Yet Tables 1 and 2 contain no sparse-retriever rows for BM25+query2doc, BM25+LameR, SPLADE++-v1+query2doc, or SPLADE++-v1+LameR. The only query2doc/LameR rows use SimLM, a dense retriever. Consequently, the observed advantage of, e.g., SPLADE++-v1+Exp4Fuse over SimLM+query2doc is confounded by retriever strength rather than by the fusion framework, leaving the headline comparative claim unverified. Please add same-backbone runs for query2doc and LameR on at least BM25 and SPLADE++-v1/v2.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The paper never compares the modified reciprocal rank fusion formula against the standard RRF of Cormack et al. [6] or against any other fusion method. This is a load-bearing omission because Table 4 shows that the LLM-expanded route alone degrades learned sparse retrievers substantially (e.g., SPLADE++-v1 DL19 nDCG@10 drops from 73.1 to 67.8), so the gain to 77.6 comes from the fusion rule itself. Without a standard-RRF baseline, the contribution of the 'adaptive weight strategy' (the n/10 term and the wi weights) is not established. Please include standard RRF on the same two lists, and ideally one additional fusion baseline such as CombSUM or Borda count, as a control.","section":"Section 3.2, Eq. (2), and Table 4"},{"comment":"The key hyperparameters—k=60 in Eq. (2), λ=5 in Eq. (1), and w1=w2=1—are described as fixed during a pilot study or set empirically, but the pilot study data, tuning procedure, and any sensitivity analysis are not reported. If k or λ were selected using the evaluation benchmarks (MS MARCO dev, TREC DL 2019/2020), the reported improvements would be partly circular. Since these constants directly affect the fusion scores and the augmented queries, please provide a sensitivity analysis on a validation split that is independent of the test sets, or otherwise justify that the values were not chosen on the evaluation data.","section":"Sections 3.2 and 4.1"},{"comment":"All experiments appear to be single runs with no error bars, per-topic variance, or significance tests. Several reported improvements are small—for example, SLIM++ +Exp4Fuse gains only +0.4 nDCG@10 on DL20 and +0.1 R@1k on DL19, and SPLADE++-v1 +Exp4Fuse gains +0.7 R@1k on MS MARCO dev—yet Section 6 concludes that Exp4Fuse 'significantly improves' performance. Please report standard deviations across the query set or paired significance tests (e.g., two-tailed paired t-test or bootstrap) for the principal comparisons.","section":"Tables 1-4"}],"minor_comments":[{"comment":"The text says the LLaMA3-8B-Instruct results are 'presented in Table 4,' but the table is captioned 'Table 3: Results for Exp4Fuse using LLaMA3-8B-Instruct.' The table numbering is inconsistent and should be corrected.","section":"Section 5, Generalizability"},{"comment":"Metric names are formatted inconsistently: Table 1 uses 'M AP' and 'R@1k,' while Table 3 uses 'R@1K'; the model names 'SPLADE_v1' and 'SPLADE_v2' in Table 3 should match the 'SPLADE++-v1' and 'SPLADE++-v2' used in Table 1.","section":"Tables 1 and 3"},{"comment":"The sentence 'Exp4Fuse uses λ = 5for LLM-based query expansion' contains a missing space before 'for,' and later 'nDCG @10' should be 'nDCG@10' for consistency.","section":"Section 4.1, Implementation Details"},{"comment":"The summation notation '2X' is not standard; it should be written as \\(\\sum_{i=1}^{2}\\). Also, the sentence defining \\(r_i\\) should clarify that \\(i=1\\) corresponds to \\(I_{oq}\\) and \\(i=2\\) to \\(I_{eq}\\), which is only stated in the surrounding text.","section":"Equation (2)"},{"comment":"The dataset name 'Robsut04' is a typo for 'Robust04,' and the inconsistent capitalization of 'Touche2020' versus 'Touche 2020' should be harmonized.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a consistently positive empirical effect, which is genuine and worth reporting. However, the advertised comparison against query2doc and LameR on sparse retrievers is not actually present in the tables, and the missing standard-RRF control is easy to add given the released code. I would ask for those additions and a sensitivity analysis for k and λ before considering acceptance. The margins on some benchmarks are small, so I would also like to see either significance tests or a statement about run-to-run variance. No concerns about research integrity beyond the methodological gaps noted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [colleague],\n\nWhat you should know up front: the core trick works. Running the same sparse retriever twice, once on the original query and once on a zero-shot LLM-expanded query, then fusing the two lists with a small tweak to reciprocal rank fusion, reliably beats the original-query-only baseline across MS MARCO dev, TREC DL 19/20, and several BEIR datasets, for BM25, uniCOIL, SLIM++, and SPLADE++ variants. The effect is consistent, and the authors are honest about the mechanism: Table 4 shows the expanded route alone underperforms the original route, so fusion is doing real work.\n\nThe novelty is modest. Nothing here is conceptually new—LLM expansion for sparse retrieval and multi-query fusion are established—but the specific pipeline (original query repeated lambda times plus a hypothetical document, then a modified RRF with a membership bonus) is a reasonable combination, and the code is public. That is enough to be practically useful.\n\nThe soft spots are real, and one is load-bearing. Section 4.1 promises comparisons with query2doc and LameR 'under identical experimental conditions,' but those methods appear only with SimLM as the backbone, never with BM25 or SPLADE. So the abstract's claim that Exp4Fuse 'surpasses existing LLM-based query expansion methods in enhancing sparse retrievers' is not actually tested. The SOTA claims are cross-paper comparisons. Second, there is no comparison against the original RRF. With w1=w2=1, the only difference from standard RRF is the (1+n/10) multiplier, which rewards documents appearing in both lists. That might help, but we cannot tell from the data. Third, k=60 and lambda=5 come from an unspecified 'pilot study'; if they were tuned on these benchmarks, the reported gains are partly circular. A sensitivity analysis is needed. Finally, there are no error bars or significance tests, and some gains are noisy (e.g., SLIM++ on DL20, +0.1 nDCG@10).\n\nNone of this kills the paper. The central effect—fusion over the single-query baseline—holds up, and the missing controls are fixable in a revision. A serious referee should require same-backbone query2doc/LameR rows, a standard RRF baseline, and sensitivity analysis for k and lambda. I would send this to peer review with those conditions. It is worth a read for anyone working on sparse retrieval or LLM-based query expansion, and I would cite it as a cheap fusion recipe, though not for its head-to-head claims.\n\nBest,\n[Your name]","headline":"A practical fusion recipe for sparse retrieval that works, but the paper's headline claim against query2doc/LameR is not yet demonstrated because those baselines are never run on the same sparse backbones.","tokens_in":13432,"tokens_out":4519,"would_cite":true,"duration_ms":48899,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Exp4Fuse claims that fusing a sparse retriever's ranking of the original query with its ranking of a zero-shot LLM-augmented query consistently improves retrieval and reaches state-of-the-art results on TREC DL 2019 when paired with…","keywords":["rank fusion","query expansion","large language models","sparse retrieval","learned sparse retrievers","reciprocal rank fusion","zero-shot","BEIR"],"falsifier":"Run Exp4Fuse on new, unseen retrieval datasets with $k=60$ and $\\lambda=5$ frozen exactly as reported, and compare the fused ranking against the better of the two individual route lists; if the fused ranking does not consistently beat that single best list, the central fusion claim fails.","tokens_in":12510,"feed_emoji":"🔀","tokens_out":5636,"duration_ms":58501,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models can improve sparse retrieval without the usual fragility of LLM-based query expansion, by never letting the LLM's hypothetical document answer the query on its own. Instead, Exp4Fuse runs the same sparse retriever twice—once on the original query, once on a query augmented with an LLM-generated passage—and fuses the two ranked lists with a modified reciprocal rank fusion. The claim is that this indirect route reliably beats direct LLM query expansion, which often degrades learned sparse retrievers, and yields state-of-the-art results on several web-search benchmarks when paired with SPLADE++-v1 or v2. A sympathetic reader would care because it offers a cheap, retriever-agnostic way to make sparse and learned sparse retrieval competitive with much heavier dense-pipeline systems.","feed_headline":"Sparse retrieval hits SOTA by fusing original and LLM-expanded rankings","feed_subtitle":"Same retriever runs twice—on the original and LLM-augmented query—and the fused list beats heavy dense pipelines.","key_machinery":"The load-bearing object is a two-route retrieval plus a modified reciprocal rank fusion score: $FR_{score} = (w_i + \\frac{n}{10}) \\sum_{i=1}^{2} \\frac{1}{k + r_i}$, with $k=60$ fixed from a pilot study, $w_i=1$ for both routes, and $n \\in \\{1,2\\}$ the number of lists containing the document. The $\\frac{n}{10}$ bonus preferentially promotes documents that both the original query and the LLM-augmented query retrieve, while the $k$ term dampens rank outliers. A second mechanism is the query-balancing weight $\\lambda$: repeating the original query $\\lambda$ times before concatenating the LLM-generated passage prevents the long hypothetical document from drowning out the user's terms in a sparse lexical match.","core_discovery":"Exp4Fuse establishes that the value of zero-shot LLM query expansion for sparse retrieval is realised only when the expanded query is used in parallel with the original query, not as a replacement. The framework generates two ranked lists from one sparse retriever—one for the original query, one for the original query repeated λ times and concatenated with an LLM-generated hypothetical passage—and fuses them with an adaptive reciprocal rank score that up-weights documents appearing in both lists. On MS MARCO dev, TREC DL 2019/2020, and seven BEIR datasets, fusing improves every sparse retriever tested, from BM25 to SPLADE++-v2, across nearly all metrics; SPLADE++-v1 + Exp4Fuse sets state-of-the-art numbers on TREC DL 2019. The paper also shows the fusion stage is necessary: using the LLM-augmented query alone often hurts learned sparse retrievers, whereas fusing recovers and exceeds the original-query baseline.","pith_inferences":["The $\\frac{n}{10}$ bonus in the fusion formula acts as an implicit consensus prior; a testable extension is to let $n$ carry fractional credit so documents present in both lists are treated as more reliable without over-penalising unique relevant documents.","Because the framework is retriever-agnostic, it should transfer to any dense or hybrid retriever that produces ranked lists; fusing original-query dense rankings with LLM-augmented dense rankings could show similar or larger gains, assuming the same failure mode of direct expansion.","The results suggest that LLM hallucination may be less harmful to retrieval when hypothetical documents are used as a second opinion rather than as the query itself; a controlled experiment varying LLM accuracy could test this.","The paper fixes $k=60$ and $\\lambda=5$; a systematic sweep on held-out data could yield larger gains than the reported fixed values, but the fixed values' success across datasets hints at robustness."],"forward_implications":["Any sparse retriever, from BM25 to learned SPLADE variants, can be improved by fusing original and LLM-expanded result lists, with no retraining.","Direct zero-shot LLM expansion alone is unreliable for learned sparse retrievers; the fusion ranking stage is what converts LLM output into consistent gains.","Exp4Fuse with SPLADE++-v1/v2 matches or beats strong dense retrievers and multi-stage retrieval pipelines on several benchmarks, at lower computational cost.","The benefit extends out-of-domain to low-resource BEIR datasets, with the largest gains on short, ambiguous queries like Touche2020 and NQ.","Performance grows with additional LLM-based routes up to three, then plateaus or degrades, so two routes (original + hypothetical document) are the cost-effective choice."],"supporting_citations":[{"why":"Supplies the reciprocal rank fusion method that Exp4Fuse modifies with adaptive weights.","marker":"[6]"},{"why":"The zero-shot LLM query expansion baseline that works for dense but poorly for sparse retrieval, motivating the indirect fusion approach.","marker":"[12]"},{"why":"Few-shot LLM query expansion baseline (query2doc) that Exp4Fuse outperforms on sparse retrieval.","marker":"[31]"},{"why":"Strong prompt-based LLM query expansion baseline (LameR) whose sparse gains are shown to be trivial.","marker":"[28]"},{"why":"Provides the SPLADE++ learned sparse retrievers on which Exp4Fuse achieves state-of-the-art results.","marker":"[9]"},{"why":"Document expansion baseline (docT5query) that Exp4Fuse also improves when combined.","marker":"[23]"},{"why":"Supplies the seven low-resource BEIR datasets used for out-of-domain evaluation.","marker":"[29]"},{"why":"Evidence that LLM query expansion performance is highly prompt-sensitive, motivating Exp4Fuse's avoidance of direct prompting dependence.","marker":"[15]"}],"fun_headline_variants":["Sparse retrieval SOTA via fusing original and LLM-expanded queries","LLM query expansion shines only when fused with original, not alone","Exp4Fuse: dual-route sparse retrieval outranks dense pipelines","Fuse, don't replace: LLM-expanded queries boost sparse retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume the fusion constants $k=60$ and $\\lambda=5$, and the zero-shot prompts, were chosen without tuning on the evaluation test sets; if any of these were selected using MS MARCO dev or TREC DL labels, the improvements would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Sparse retrieval SOTA via fusing original and LLM-expanded queries","LLM query expansion shines only when fused with original, not alone","Exp4Fuse: dual-route sparse retrieval outranks dense pipelines","Fuse, don't replace: LLM-expanded queries boost sparse retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000496,"raw_usage":{"total_tokens":2471,"prompt_tokens":1020,"completion_tokens":1451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1370}},"tokens_in":636,"tokens_out":1451,"duration_ms":12302,"temperature":1.0,"reasoning_tokens":1370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:33:34.421271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Exp4Fuse on new, unseen retrieval datasets with $k=60$ and $\\lambda=5$ frozen exactly as reported, and compare the fused ranking against the better of the two individual route lists; if the fused ranking does not consistently beat that single best list, the central fusion claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reciprocal rank fusion method that Exp4Fuse modifies with adaptive weights."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Strong prompt-based LLM query expansion baseline (LameR) whose sparse gains are shown to be trivial."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SPLADE++ learned sparse retrievers on which Exp4Fuse achieves state-of-the-art results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Document expansion baseline (docT5query) that Exp4Fuse also improves when combined."}],"review_version":1}