{"id":"861c5090-26e0-463c-8938-78cb41952772","arxiv_id":"2506.00491","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Q-DREAM improves multi-hop retrieval-augmented QA by decomposing questions, rewriting dependent subquestions, and retrieving with cluster-specific LoRA embeddings.","lead":"This paper introduces Q-DREAM, a retrieval-augmented system that breaks hard questions into smaller pieces, rewrites dependent pieces, and retrieves evidence with specialized semantic embeddings per question cluster. On open-domain multi-hop QA benchmarks it reports large accuracy gains and six times faster inference than IRCoT, though the supporting code, statistical uncertainty, and fair baseline training are not provided.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported SOTA is not established because Q-DREAM trains on 2WikiMQA while baselines do not, and k/N are tuned on test-set curves; a fair trained baseline could erode the margin.","rationale":"The paper proposes a plausible pipeline (QDM, SDOM, DPRM) and ablations suggest each module contributes. However, the headline empirical claim is 'significantly outperforms existing RAG methods' in both in-domain and OOD settings. The weakest point is the evaluation protocol: Q-DREAM benefits from supervised training on 2WikiMQA, while the main baselines (IRCoT, SURE, ChatGPT) are zero-shot prompters, and InstructRAG/ChatQA2 are not trained on this domain. This makes the comparison apples-to-oranges. Additionally, the hyperparameters k and N are selected using test-set curves (Figures 3 and 4), which is a form of test-set tuning and inflates reported scores. These issues are more directly load-bearing than the reader's DPRM training/inference assumption, because even if DPRM's cluster transfer works, the SOTA claim would still be unsupported without fair baselines and proper hyperparameter selection. The missing code/data and absence of error bars further prevent verification. I therefore agree with the reader's CONDITIONAL verdict, but for a different primary reason; the DPRM concern is valid but secondary. The concrete test above would settle whether the reported margin is real or an artifact of the evaluation setup.","tokens_in":12971,"tokens_out":9675,"duration_ms":86063,"concrete_test":"Re-run Table 1 with a strong baseline that also uses the 2WikiMQA training data: fine-tune a dense retriever (e.g., Contriever or DPR) on the same 13,363 subquestion-helpful-passage pairs, and use it within IRCoT or SURE, with N and k chosen on a held-out validation split of 2WikiMQA rather than on HotpotQA/IIRC test sets. If Q-DREAM's EM/F1 advantage over this trained baseline drops below the reported margins (e.g., HotpotQA F1 from 17.5 to less than 5), the SOTA claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of state-of-the-art performance in both in-domain and OOD settings is not supported by a fair comparison. In §4.1, Q-DREAM is trained on 13,363 samples from 2WikiMQA; QDM, SDOM, and DPRM are all fine-tuned on this data. Baselines in §4.2 are not: IRCoT and SURE are zero-shot prompting methods, and InstructRAG/ChatQA2 are not trained on 2WikiMQA. Thus the reported gains on HotpotQA and IIRC may reflect the benefit of supervised training on a related domain rather than the proposed architecture. Moreover, Figures 3 and 4 show that N (number of retrieved passages) and k (number of clusters) are selected based on test-set performance, and the Table 1 numbers likely use the best per-dataset N/k, inflating EM/F1. No error bars or code are provided. The DPRM cluster-transfer issue raised by the reader is secondary: even if DPRM works as intended, the empirical comparison remains unfair.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Q-DREAM, a three-module pipeline for retrieval-augmented multi-hop question answering. The Question Decomposition Module (QDM) splits a multi-hop question into subquestions; the Subquestion Dependency Optimizer Module (SDOM) rewrites dependent subquestions using previously retrieved passages; and the Dynamic Passage Retrieval Module (DPRM) clusters subquestions and applies cluster-specific LoRA adapters to an embedding model so that question and helpful-passage embeddings are aligned in separate semantic spaces. The modules are trained on 13,363 samples from 2WikiMQA, with QDM/SDOM labels generated by ChatGPT and DPRM trained with helpful-passage supervision. Experiments on 2WikiMQA, HotpotQA, and IIRC report EM/F1 improvements over InstructRAG, ChatQA2, ChatGPT, SURE, and IRCoT, plus a retrieval efficiency comparison against IRCoT and retrieval accuracy comparisons against BM25, Contriever, and baseline retrievers.","tokens_in":13165,"tokens_out":3188,"duration_ms":29564,"significance":"If the results are reproducible and the comparison is fair, Q-DREAM would be a useful contribution: it addresses a real limitation of one-step RAG for multi-hop QA, the architecture is model-agnostic, the DPRM idea of cluster-specific retrieval spaces is interesting, and the paper includes ablation studies showing that each module contributes positively (Table 2). The efficiency claim, if validated, would also be practically important. However, the empirical claims as presented are not yet established: the main comparison confounds architecture with supervised training on the in-domain dataset, and the hyperparameters k and N appear to be tuned on the test sets. These issues are load-bearing for the central state-of-the-art claim, so the paper needs a substantial revision of the experimental protocol before the results can be accepted.","major_comments":[{"comment":"The central SOTA claim is not supported by a fair comparison. Q-DREAM is trained on 13,363 samples from 2WikiMQA (§4.3), covering QDM, SDOM, and DPRM, whereas the baselines in Table 1 are not described as receiving any 2WikiMQA training: IRCoT and SURE are zero-shot prompting methods, and InstructRAG/ChatQA2 are general-purpose models. The reported gains on 2WikiMQA could therefore reflect supervised training on the test distribution, and the HotpotQA/IIRC gains could reflect transfer from 2WikiMQA supervision rather than the proposed architecture. Please report results for baselines trained on the same 2WikiMQA data, or provide an untrained variant of Q-DREAM's modules, and state clearly which systems use 2WikiMQA supervision.","section":"§4.1–§4.2, Table 1"},{"comment":"The hyperparameters N (number of retrieved passages per subquestion) and k (number of semantic clusters) appear to be selected from test-set accuracy curves in Figures 3 and 4. If the numbers in Table 1 use the per-dataset optimal N and k, then part of the reported EM/F1 is fitted to the evaluation sets, inflating the in-domain and out-of-domain results. Please select these hyperparameters on a held-out validation split disjoint from the reported test sets, or report the selection procedure and show results for fixed N/k across all datasets.","section":"§5.3, Figures 3 and 4"},{"comment":"The retrieval comparison is difficult to interpret because DPRM is trained with the 'helpful passage' supervision in Eq. (3), while BM25, Contriever, and the retrieval components of the RAG baselines are not trained on the same gold-passage labels. It is expected that a retriever trained on helpful-passage pairs would achieve higher precision and recall on 2WikiMQA. Please clarify the retrieval evaluation protocol, including whether the gold helpful passages are used as training labels, evaluation targets, or both, and report a DPRM variant trained only on the question-passage signals available to the IR baselines. This is necessary to support the claim of improved retrieval accuracy.","section":"§5.3, Table 4 and Eq. (3)"},{"comment":"The efficiency claim rests on a single comparison against IRCoT, and the three IRCoT runs vary between 8,887 s and 16,138 s (Table 3), which suggests uncontrolled conditions such as API variance, caching effects, or concurrent load. As reported, the '6× faster' conclusion is not reliable. Please describe the measurement setup in detail, report the number of samples and per-run variance, and compare against at least one additional retrieval-augmented baseline under identical conditions.","section":"§5.3, Table 3"}],"minor_comments":[{"comment":"The abstract contains a typo: 'Whereas,conventional' should be 'Whereas, conventional'; in §1, 'root temperature' should be 'room temperature'.","section":"Abstract and §1"},{"comment":"In Eq. (3), the notation 'E(q,p)∈Ci log ...' is unclear; it should be a sum over the pairs in cluster Ci, e.g., '∑_{(q,p)∈Ci} log ...'.","section":"§3.2, Eq. (3)"},{"comment":"The text refers to 'SDQM' in 'subquestions processed through QDM and SDQM'; this should be 'SDOM'.","section":"§3.2"},{"comment":"The notation 'E_{q_i*} = h^{(M)}_{Last} ← M^{c_{q_i*}}_{DPRM}(q_i*)' is confusing; it mixes an assignment with the extraction of the last hidden state. Please rewrite this line to separate the forward pass from the hidden-state extraction.","section":"Algorithm 1, line 11"},{"comment":"The table reports single EM/F1 values without error bars or significance tests; given the small margins in some comparisons, please provide variance or significance information.","section":"Table 1"},{"comment":"The Limitations section does not mention the evaluation-fairness issues (training on 2WikiMQA, test-set hyperparameter selection) or the reliance on ChatGPT-generated decomposition labels; these should be acknowledged.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.IR and the core idea has merit, but the experimental protocol needs substantial strengthening. The most serious issues are the unfair comparison with untrained baselines on 2WikiMQA and the apparent selection of k and N on test-set curves; both directly affect the SOTA claim. I would encourage the editor to request a revision with retrained baselines, validation-based hyperparameter selection, and a clarified retrieval evaluation protocol, rather than rejecting the paper outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Q-DREAM is a sensible modular pipeline: decompose, rewrite dependent subquestions, and retrieve with cluster-specific LoRA blocks trained on E5-Mistral. The work is readable, the ablations show each module contributes, and the 6x inference speedup over IRCoT is a useful practical point if it holds up. The per-cluster LoRA idea is a genuine extension worth borrowing, even though each piece comes from existing work.\n\nThe problem is the headline claim of state-of-the-art performance. Q-DREAM is trained on 13,363 2WikiMQA samples; the ChatGPT-row baselines (IRCoT, SURE, ChatGPT) are zero-shot prompting methods. So the in-domain comparison is essentially trained-versus-untrained, and even the OOD gains on HotpotQA and IIRC may reflect supervised training on a related task rather than anything the architecture does. The paper never acknowledges this. Without a baseline that is also trained on the same 2WikiMQA split, the margin is not interpretable. Adding to that, Figures 3 and 4 tune N and k on test-set curves, so the Table 1 numbers likely use the best per-dataset hyperparameters. No error bars, no code or data release. These are fixable but currently make the empirical contribution a prototype rather than a verified result.\n\nA secondary concern: DPRM's training objective assumes a known helpful passage for each subquestion, and the paper is silent on how those passages are obtained for the training split. That is a reasonable setup on 2WikiMQA but limits the \"real application\" story. The stress-test note on cluster transfer is real but secondary; the main issue is the comparison.\n\nBottom line: this is a legitimate engineering contribution for multi-hop RAG researchers. It deserves a serious referee, but the referee should insist on trained baselines, error bars, validation-set hyperparameter selection, and released artifacts. If those land, the result could be a solid subfield contribution. As is, the overclaim should not go to press.","headline":"A plausible modular RAG pipeline whose headline SOTA claim is undercut by a trained-versus-zero-shot comparison and test-set tuning of k and N.","tokens_in":13711,"tokens_out":2988,"would_cite":false,"duration_ms":27206,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Q-DREAM claims that splitting multi-hop questions into subquestions, rewriting dependent steps, and retrieving each subquestion in a cluster-specific embedding space produces state-of-the-art accuracy on multi-hop QA benchmarks and…","keywords":["retrieval-augmented generation","multi-hop question answering","question decomposition","semantic matching","cluster-specific retrieval","LoRA","out-of-domain generalization","retrieval efficiency"],"falsifier":"The central claim would be falsified if, on the same QDM and SDOM subquestions, a single shared embedding encoder matched or beat DPRM's retrieval recall on out-of-domain data, since the paper attributes the gain specifically to the cluster-specific spaces; a direct test would compare retrieval recall on HotpotQA and IIRC with and without the per-cluster LoRA blocks.","tokens_in":12739,"feed_emoji":"⚡","tokens_out":12253,"duration_ms":101257,"temperature":0.7,"pith_summary":"Multi-hop questions like \"Who is the child of the director of Mukhyamantri (1996 Film)?\" require several retrieval steps, and conventional RAG often pulls out passages that are semantically close but not actually useful. Q-DREAM addresses this by decomposing the question, rewriting any subquestion that depends on an earlier answer into a self-contained query, and retrieving each subquestion through a separately tuned embedding space chosen by its semantic cluster. The paper claims this pipeline outperforms existing RAG baselines with both ChatGPT and Llama2-7B backbones, generalizes from 2WikiMQA to HotpotQA and IIRC without retraining, and runs about six times faster per question than interleaved retrieval-and-reasoning. If true, it would mean that organizing retrieval by question type can buy both accuracy and speed for multi-step open-domain questions.","feed_headline":"Splitting questions lifts QA accuracy, cuts retrieval time 6x","feed_subtitle":"Decomposing questions, rewriting dependent steps, and giving each topic cluster its own retriever lifts accuracy.","key_machinery":"The load-bearing mechanism is the Dynamic Passage Retrieval Module (DPRM), which replaces one shared retriever with $k$ cluster-specific retrievers. Subquestions are embedded and partitioned with $k$-means; for each cluster $C_i$, a LoRA adapter is fine-tuned to maximize the cosine similarity $\\cos(E_q, E_p)$ between a subquestion $q$ and its labeled helpful passage $p$ (the paper's Eq. 3). At inference, a subquestion is assigned to the nearest cluster, and only that cluster's LoRA block encodes the question and the candidate passages before the highest-scoring passage is selected. The Subquestion Dependency Optimizer (SDOM) supplies DPRM with complete queries by rewriting markers such as '#1#' using the retrieved passage for the earlier subquestion, and the Question Decomposition Module (QDM) provides the initial subquestions; together they convert a multi-hop problem into a sequence of typed single-hop retrievals.","core_discovery":"The paper's central claim is that the bottleneck in multi-hop retrieval-augmented QA is semantic mismatching: questions and their genuinely helpful passages sit in different neighborhoods of a shared embedding space, so a single retriever favors plausible but useless text. Q-DREAM's response is to make retrieval dynamic and typed. The Question Decomposition Module (QDM) splits the original question into subquestions; the Subquestion Dependency Optimizer (SDOM) turns dependent subquestions such as 'Who is the child of the director from #1#?' into complete queries using the earlier subquestion's retrieved passage; and the Dynamic Passage Retrieval Module (DPRM) assigns each subquestion to one of $k$ semantic clusters, each with its own LoRA-tuned embedding space that pulls helpful passages close to questions sharing the same relational pattern. Trained on 2WikiMQA and evaluated with ChatGPT and Llama2-7B, Q-DREAM reports exact-match/F1 of 48.6/62.1 on 2WikiMQA, 48.4/60.9 on HotpotQA, and 28.2/31.9 on IIRC, with retrieval precision 81.8 and recall 85.7 on 2WikiMQA, gains the paper attributes to the cluster-specific embedding spaces rather than to the backbone model.","pith_inferences":["Going beyond the paper: the same cluster-indexed retrieval scheme should transfer to any domain with stable relational patterns and varying entities, such as 'mayor of city' or 'CEO of company', because DPRM learns the pattern rather than the specific entities.","Going beyond the paper: the gold-passage requirement could be relaxed by using an existing retriever to generate pseudo-helpful passages and keeping only those that lead to a correct final answer, which would let DPRM be trained on new domains without annotated passages.","Going beyond the paper: the reported speedup is measured against interleaved retrieval-and-reasoning; comparing against a decomposition-only baseline with a shared encoder would isolate how much of the efficiency gain comes from avoiding round trips versus from the per-cluster retrieval setup."],"forward_implications":["A multi-hop question can be answered in one pass of decomposition, dependency rewriting, and per-subquestion retrieval, so the generator never needs to interleave its own reasoning with retrieval calls.","Retrieval precision and recall rise sharply when each semantic cluster of subquestions has its own LoRA-tuned embedding space: on 2WikiMQA, Q-DREAM reports 81.8 precision and 85.7 recall versus 53.8 and 68.9 for IRCoT.","Because the modules are trained once on 2WikiMQA and then applied without retraining, the same pipeline transfers directly to HotpotQA and IIRC.","The pipeline works with a small generator: Q-DREAM with Llama2-7B beats RAG baselines built on ChatGPT, suggesting the gain comes from retrieval organization rather than model scale.","Inference is about six times faster than IRCoT (4 versus 25 seconds per sample on average), because the framework avoids multiple round trips between retriever and generator."],"supporting_citations":[{"why":"IRCoT is the strongest retrieval-augmented baseline and the source of the subsampled test splits used for all three datasets.","marker":"Trivedi et al., 2023"},{"why":"HotpotQA is the out-of-domain benchmark used to test generalization without retraining.","marker":"Yang et al., 2018"},{"why":"2WikiMQA is the in-domain dataset used for training and evaluation.","marker":"Ho et al., 2020"},{"why":"IIRC is the second out-of-domain benchmark in the evaluation.","marker":"Ferguson et al., 2020"},{"why":"E5-Mistral is the base embedding model on which DPRM's cluster-specific LoRA blocks are fine-tuned.","marker":"Wang et al., 2023"},{"why":"Mistral-7B is the backbone fine-tuned for the QDM and SDOM modules.","marker":"Jiang et al., 2023"},{"why":"InstructRAG is a RAG baseline that denoises retrieved content with self-synthesized rationales.","marker":"Wei et al., 2025"},{"why":"SURE is the strongest baseline Q-DREAM compares against on out-of-domain HotpotQA.","marker":"Kim et al., 2024"},{"why":"ChatQA2 is a long-context RAG baseline used in the comparison.","marker":"Xu et al., 2024"}],"fun_headline_variants":["Per-cluster retrievers on decomposed subquestions boost QA accuracy","Cluster-tuned embedding spaces solve multi-hop QA retrieval mismatches","Dynamic retrieval after question decomposition improves multi-hop QA","Split, rewrite, cluster: dynamic retrieval for multi-hop QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that every training subquestion has a known 'helpful passage' to pull toward, and that clusters learned on 2WikiMQA will assign unseen subquestions to LoRA blocks that align them with genuinely relevant passages.","fun_headline_variants_meta":{"raw":{"variants":["Per-cluster retrievers on decomposed subquestions boost QA accuracy","Cluster-tuned embedding spaces solve multi-hop QA retrieval mismatches","Dynamic retrieval after question decomposition improves multi-hop QA","Split, rewrite, cluster: dynamic retrieval for multi-hop QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00168,"raw_usage":{"total_tokens":6700,"prompt_tokens":1023,"completion_tokens":5677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":5609}},"tokens_in":639,"tokens_out":5677,"duration_ms":36439,"temperature":1.0,"reasoning_tokens":5609,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:04:02.038032+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The central claim would be falsified if, on the same QDM and SDOM subquestions, a single shared embedding encoder matched or beat DPRM's retrieval recall on out-of-domain data, since the paper attributes the gain specifically to the cluster-specific spaces; a direct test would compare retrieval recall on HotpotQA and IIRC with and without the per-cluster LoRA blocks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IRCoT is the strongest retrieval-augmented baseline and the source of the subsampled test splits used for all three datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IIRC is the second out-of-domain benchmark in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"InstructRAG is a RAG baseline that denoises retrieved content with self-synthesized rationales."}],"review_version":1}