{"id":"78d61383-0b8c-4543-9c24-dd4813003f6d","arxiv_id":"2507.00355","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Splitting multi-hop questions into subquestions and reranking the merged retrieval pool improves RAG evidence coverage and answer accuracy on MultiHop-RAG and HotpotQA.","lead":"This paper tests a retrieval-augmented generation pipeline that splits a multi-hop question into subquestions, retrieves passages for each, and reranks the merged results against the original question. On two multi-hop QA benchmarks it reports consistent gains in retrieval quality and answer accuracy over a standard RAG baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation does not control for candidate-pool size: QD+RR reranks up to ~60 merged passages, RR only 20, so the headline retrieval gains may reflect pool size, not decomposition-specific subqueries.","rationale":"The paper is honestly written, with a clear pipeline, internally controlled baselines, released code, and a limitations section that flags prompt/model sensitivity and overhead. Those are genuine positives. The central empirical claim, however, is supported by an ablation that does not isolate decomposition from a larger candidate pool: RR uses 20 candidates and QD+RR uses up to 60, so the reranker has more chances to surface gold passages. The fixed five-subquery behavior in Table 3 makes this pool-size difference systematic rather than adaptive. The HotpotQA supporting-fact and joint metrics, which are lower for QD+RR than for RR, are additional evidence that decomposition does not consistently improve gold-evidence identification. These facts do not refute the paper, but they shift the burden onto a controlled comparison. A paraphrase-expansion control is a direct, low-cost way to test whether decomposition-specific structure matters. The reader's CONDITIONAL verdict remains appropriate; my stress test identifies a sharper condition than the reader's prompt-sensitivity concern, hence partial agreement.","tokens_in":12281,"tokens_out":8348,"duration_ms":89738,"concrete_test":"Run a paraphrase-expansion control on MultiHop-RAG: use the same LLM to generate five independent paraphrases of the original query without decomposition into subquestions, retrieve top-10 per paraphrase plus the original, merge, rerank with bge-reranker-large, and compare Hits@10 and MRR@10 against QD+RR. If the control matches 0.872 and 0.635 within noise, the headline gain is a candidate-pool effect rather than a decomposition effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 (Table 1) compares RAG+RR with QD+RR without holding the candidate-pool size fixed. RR retrieves top-2k=20 passages for the original query, while QD+RR merges top-k=10 passages for the original query plus top-10 for each of up to five subqueries, yielding up to 60 candidates before reranking. Table 3 shows the LLM emits exactly five subqueries in 93-99% of cases, so the merged pool is effectively 3x larger. A reranker over a larger pool can improve Hits@10 and MRR even if the extra candidates are generic paraphrases, so the causal claim that decomposition-specific subqueries assemble complementary evidence is not isolated. HotpotQA results reinforce the concern: in Table 2, QD+RR has lower supporting-fact EM/F1 and joint EM/F1 than RR alone (17.9/11.2 vs 19.6/12.9 and 8.9/5.8 vs 9.5/6.4), so the improved answer accuracy is not accompanied by consistently better evidence identification. Controlling for pool size and reporting significance would settle whether question decomposition itself carries the reported benefit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retrieval-augmented generation pipeline that adds an LLM-based question decomposition (QD) module and an off-the-shelf cross-encoder reranker (RR) to standard dense retrieval. For a multi-hop query, the LLM generates up to five subqueries; top-k passages are retrieved for the original query and each subquery, merged, deduplicated, and reranked against the original query before the top-k passages are passed to the generator. The authors evaluate four system variants (Naive RAG, +QD, +RR, +QD+RR) on MultiHop-RAG and HotpotQA, reporting that QD+RR achieves the highest Hits@10 (87.2%) and MRR@10 (0.635) on MultiHop-RAG and the highest answer EM/F1 on HotpotQA. The paper also reports latency overhead, subquery-count distributions, and a correlation analysis showing no relation between subquery count and gold evidence count.","tokens_in":12498,"tokens_out":3922,"duration_ms":43875,"significance":"If the reported retrieval gains survive a properly controlled comparison, the paper offers a practical and modular enhancement for multi-hop RAG: it requires no fine-tuning, uses only an instruction-tuned LLM for decomposition and a standard reranker, and the authors release their code. The four-way internal ablation is a useful design choice, and using exact FAISS inner-product search removes approximate-search confounds. The central scientific claim, that decomposition-specific subqueries assemble complementary evidence that reranking then distills, is plausible but is not yet cleanly established because the main ablation does not hold the candidate-pool size fixed, and the HotpotQA supporting-fact metrics partially contradict the evidence-selection narrative. The paper is an incremental but potentially useful empirical contribution if these issues are resolved.","major_comments":[{"comment":"The headline comparison between +RR and +QD+RR does not control for candidate-pool size. As defined in Section 4.2, +RR retrieves top-2k = 20 passages for the original query and reranks them, while +QD+RR merges top-10 passages for the original query plus top-10 passages for each subquery. Table 3 shows that the LLM emits exactly five subqueries in 93.3% of MultiHop-RAG cases and 98.6% of HotpotQA cases, so the QD+RR pool is typically 60 candidates, three times larger than the RR pool. A reranker over a larger candidate pool can improve Hits@10 and MRR@10 even if the additional candidates are generic paraphrases of the original query, so the specific contribution of decomposition-generated subqueries is not isolated. Please add a controlled condition, for example reranking top-60 passages retrieved with the original query alone, or reducing the per-subquery cutoff so the merged pool matches 20 candidates, and report significance or bootstrap intervals across multiple seeds.","section":"Section 5.1, Table 1; Section 4.2"},{"comment":"The claim that QD+RR leads to better evidence-grounded answers is weakened by the supporting-fact and joint metrics on HotpotQA. QD+RR has lower supporting-fact EM and F1 than RR alone (17.9/11.2 vs. 19.6/12.9) and lower joint EM and F1 (8.9/5.8 vs. 9.5/6.4), despite having higher supporting-fact precision (46.8 vs. 44.9). Since the paper's narrative is that decomposition improves coverage and reranking restores precision, the evidence-identification outcome on HotpotQA does not consistently support that narrative. The paper should explicitly discuss this pattern, and should report supporting-fact recall, which is listed in Section 4.3 but omitted from Table 2; without recall it is impossible to tell whether the higher precision comes at the cost of coverage.","section":"Section 5.2, Table 2"}],"minor_comments":[{"comment":"The comparison against text-ada-002 and voyage-02 baselines from Tang and Yang (2024) uses different embedding models and retrieval settings, so the sentence 'our QD+RR thus improves Hits@10 by 16.5% and MRR@10 by 8.4%' is a cross-paper comparison rather than a controlled head-to-head gain; please label it accordingly.","section":"Section 5.1"},{"comment":"Algorithm 1 calls DECOMPOSE(q0) while Section 3.1 defines DECOMPOSE(q, p); the argument q0 is not defined. Please align the pseudocode with the notation in the text.","section":"Algorithm 1"},{"comment":"The claim that 'once decomposed, subqueries can be reused ... so that the latency remains identical to the baseline' depends on an unspecified cache-population assumption; on a cold cache the reported 18.9s/query latency applies, and this tradeoff should be stated more carefully.","section":"Section 5.4"},{"comment":"Table 5 labels columns 'Pearson (p)' and 'Spearman (p)' but the values shown appear to be the correlation coefficient with a p-value in parentheses; please rename the columns to avoid ambiguity.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is an incremental empirical contribution that could be suitable for publication after the candidate-pool-size confound is addressed. The lack of significance testing and the HotpotQA supporting-fact inconsistencies are secondary but should also be handled. The work appears methodologically honest: limitations are acknowledged, code is released, and the internal ablation is a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean, honest empirical paper about a drop-in RAG enhancement—LLM-generated subqueries plus an off-the-shelf cross-encoder reranker. The specific pairing is new as a reported configuration, and the internal comparisons are well-controlled: QD+RR beats naive RAG, QD-only, and RR-only on the main MultiHop-RAG retrieval metrics and on HotpotQA answer EM/F1. The paper ships open code, exact FAISS search, fixed hyperparameters, and a limitations section that names the real failure modes. That is worth referee time.\n\nThe soft spots are real but not fatal. The biggest one is the stress-test point: RAG+RR reranks 20 candidates, QD+RR reranks up to 60 (the LLM emits exactly five subqueries 93–99% of the time, so the pool is effectively 60). The retrieval gains are therefore not isolated to decomposition-specific subqueries; a larger pool alone can lift Hits@10 and MRR. The paper never controls for pool size, and on HotpotQA the supporting-fact EM/F1 for QD+RR is actually lower than for RR alone (17.9/11.2 vs 19.6/12.9), which undercuts the claim that decomposition assembles better evidence. They do report precision gains, but the core mechanism is unproven.\n\nAlso: no significance tests or error bars, the external comparison to Tang and Yang uses different embedding models, and there is no comparison to prior decomposition-based systems on the same benchmarks. The ablation showing a fixed subquery budget is honest—they admit it—but it weakens the 'decomposition' framing: it is generate-five-subqueries, not adapt to the question.\n\nFor a reader in the multi-hop RAG area, this is a useful data point and a good baseline. It deserves a serious referee, though the referee should push for a pool-size-controlled ablation and significance testing. I would not cite it as proof that decomposition works; I would cite it as a reproducible configuration with an open question.","headline":"Useful drop-in RAG configuration, honestly reported, but the decomposition-specific benefit is confounded by candidate-pool size.","tokens_in":13074,"tokens_out":2468,"would_cite":true,"duration_ms":25427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairing an off-the-shelf cross-encoder reranker with LLM-driven question decomposition bridges the retrieval gap on multi-hop questions, without extra training or specialized indexing.","keywords":["retrieval-augmented generation","question decomposition","multi-hop question answering","reranking","cross-encoder","dense retrieval","zero-shot RAG"],"falsifier":"On a multi-hop dataset with gold evidence annotations, compare Hits@10 and MRR@10 of (i) single-query retrieval with reranking (retrieve 2k passages, rerank to k) and (ii) question decomposition plus reranking (retrieve k per subquery, merge, rerank to k). If run (i) matches or exceeds run (ii) on the same queries and passage budget, the claim that decomposition expands coverage is falsified, because the reranker alone would then be doing all the work.","tokens_in":12042,"feed_emoji":"🧩","tokens_out":14782,"duration_ms":117071,"temperature":0.7,"pith_summary":"This paper argues that ordinary retrieval-augmented generation (RAG) struggles with multi-hop questions because the facts needed to answer them live in different documents, and a single query embedding rarely lands on all of them. The proposed fix is a pipeline that first has an instruction-tuned LLM decompose the original question into several fact-seeking subquestions, retrieves passages for each, merges and deduplicates the candidates, and then reranks the pool against the original question with an off-the-shelf cross-encoder. On MultiHop-RAG this combination raises Hits@10 from 78.1% to 87.2% and MRR@10 from 0.464 to 0.635, and on HotpotQA it produces the best answer exact-match and F1 scores among the four configurations tested, with no fine-tuning or specialized indexing. If the finding holds, any RAG system facing multi-hop or multi-entity questions can adopt this as a drop-in enhancement.","feed_headline":"Decompose the question, rerank the pool: multi-hop RAG up 36.7%","feed_subtitle":"MRR@10 rises 36.7% and answer F1 rises 11.6%, with no extra training.","key_machinery":"The central mechanism is a two-stage retrieval pipeline built on top of standard dense retrieval. In the first stage, an instruction-tuned LLM with a fixed prompt generates up to five fact-seeking subquestions per original question; dense retrieval is run for each subquestion and for the original query, and the results are merged and deduplicated. In the second stage, a cross-encoder reranker scores every merged candidate against the original question and returns the top-k passages for generation. Question decomposition is the coverage-expanding stage, and reranking is the precision-restoring stage; the original query is always retained in the retrieval set to preserve baseline performance.","core_discovery":"The central claim is that the retrieval gap in multi-hop RAG is bridgeable by pairing LLM-driven question decomposition with a cross-encoder reranker, without any extra training or specialized indexing. Decomposition expands the candidate pool so that each subquestion targets a separate document holding a needed fact; reranking then scores every merged candidate against the original query and keeps only the top-k, restoring precision. The paper's QD+RR system reaches 87.2% Hits@10 and 0.635 MRR@10 on MultiHop-RAG, beating the strongest reported baselines from the dataset paper, and it achieves the best answer EM (28.1) and F1 (35.0) on HotpotQA among naive RAG, QD-only, and reranker-only variants. The paper takes this as evidence that coverage from decomposition and precision from reranking are complementary, not redundant.","pith_inferences":["The near-zero correlation between subquery count and gold evidence count suggests the decomposition LLM is producing a diverse cover rather than counting hops; this predicts that maximizing subquery diversity, rather than number, would be the better objective for the decomposition prompt.","The paper does not test adaptive decomposition, but its own data suggest a fixed five-subquery budget is wasteful on HotpotQA, where most questions need only two supporting facts; a gating rule that decomposes only when the query is predicted to be multi-hop could retain most of the gain at a fraction of the latency.","Caching subqueries per raw query, as the paper briefly proposes, makes the amortized latency of QD+RR close to plain RAG, turning the primary overhead into a one-time storage cost.","The method's gains are demonstrated on English Wikipedia-style corpora; a natural test is whether the same pipeline holds when the reranker is mismatched with the domain, a failure mode the paper itself flags as a limitation."],"forward_implications":["RAG systems operating on multi-hop or multi-entity questions can add this two-module pipeline as a drop-in enhancement, gaining retrieval recall and ranking quality without retraining or reindexing.","Because both modules are off-the-shelf, the method is model-agnostic: any instruction-following LLM can generate subquestions, and any cross-encoder can rerank.","The ablation shows that decomposition alone yields only small gains, so the reranker is the component that turns broader coverage into usable precision.","On MultiHop-RAG, QD+RR outperforms the dataset authors' strongest reranked baselines even with a smaller embedding model, suggesting the method is not tied to a specific retriever.","Because the LLM almost always emits exactly five subquestions, the effective behavior is a fixed subquery budget; changing that budget is a direct lever that trades latency against coverage."],"supporting_citations":[{"why":"Defines the retrieval-augmented generation framework that the pipeline builds upon and extends.","marker":"Lewis et al. (2020)"},{"why":"Supplies the dense passage retrieval dual-encoder method used as the base retriever.","marker":"Karpukhin et al. (2020)"},{"why":"Introduces question decomposition as a strategy for answering complex questions, which the paper adapts to retrieval.","marker":"Perez et al. (2020)"},{"why":"Provides the MultiHop-RAG benchmark, its gold evidence annotations, and the strongest baseline results the paper compares against.","marker":"Tang and Yang (2024)"},{"why":"Provides the HotpotQA benchmark and its official QA and supporting-fact evaluation metrics.","marker":"Yang et al. (2018)"},{"why":"Supplies both the bge-large-en-v1.5 embedding model and the bge-reranker-large cross-encoder used in all experiments.","marker":"Xiao et al. (2023)"},{"why":"Establishes cross-encoder passage reranking, the precision-restoring stage of the pipeline.","marker":"Nogueira and Cho (2020)"}],"fun_headline_variants":["Split questions, rerank passages: multi-hop RAG gains 36.7%","Decompose-then-rerank boosts multi-hop RAG by 36.7%","Multi-hop RAG: decomposing queries plus reranking lifts MRR 36.7%","For multi-hop RAG, decompose and rerank: MRR up 36.7%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method depends on the assumption that a fixed prompt makes an instruction-tuned LLM produce fact-seeking subquestions whose independently retrieved passages, after reranking against the original question, retrieve the gold evidence better than the original query alone.","fun_headline_variants_meta":{"raw":{"variants":["Split questions, rerank passages: multi-hop RAG gains 36.7%","Decompose-then-rerank boosts multi-hop RAG by 36.7%","Multi-hop RAG: decomposing queries plus reranking lifts MRR 36.7%","For multi-hop RAG, decompose and rerank: MRR up 36.7%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000503,"raw_usage":{"total_tokens":2496,"prompt_tokens":1021,"completion_tokens":1475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":1378}},"tokens_in":637,"tokens_out":1475,"duration_ms":10625,"temperature":1.0,"reasoning_tokens":1378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:18:04.177476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a multi-hop dataset with gold evidence annotations, compare Hits@10 and MRR@10 of (i) single-query retrieval with reranking (retrieve 2k passages, rerank to k) and (ii) question decomposition plus reranking (retrieve k per subquery, merge, rerank to k). If run (i) matches or exceeds run (ii) on the same queries and passage budget, the claim that decomposition expands coverage is falsified, because the reranker alone would then be doing all the work.","supporting_citations":[],"review_version":1}