{"id":"1a062cf2-bd20-494c-8602-a173b82f6cd1","arxiv_id":"2506.00232","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A modular, verifier-driven RAG pipeline with iterative re-decomposition outperforms fine-tuned and agentic baselines on four multi-hop QA benchmarks.","lead":"ComposeRAG splits retrieval-augmented question answering into separate modules (decomposition, retrieval, verification) orchestrated by a self-reflection loop that re-decomposes the question when the final answer fails verification. The authors report consistent accuracy and grounding gains over RQ-RAG and Search-o1 on four multi-hop QA benchmarks, though several comparisons confound model or retriever changes with the architecture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Grounding gain is not directly measured: the 10% ungrounded-answer reduction relies on an internal verifier and a GPT-4o judge, not on an external groundedness metric.","rationale":"The reader's weakest assumption matches my own: the reliability of the Answer Verification module and the GPT-4o evaluator is load-bearing for the grounding claims. The paper's reported accuracy improvements are plausible given the controlled retrieval conditions and multiple backbone comparisons, but the headline grounding improvement is not directly measured. The ablation studies support modular contributions, though without significance testing. A human-annotated grounding audit would either confirm the 10% reduction or invalidate the central grounding claim, so the conditional verdict is appropriate pending that check.","tokens_in":554,"tokens_out":595,"duration_ms":13402,"concrete_test":"Run a human or external metric grounding audit on a matched sample: take ComposeRAG and Search-o1 outputs on 2WikiMultiHopQA (wiki2018 subset described in Table 7), have annotators label whether each answer is supported by the retrieved context, and compare ungrounded-answer rates. If ComposeRAG's ungrounded rate is not at least 10 percentage points (absolute) lower than Search-o1's, the abstract's grounding claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that ComposeRAG reduces ungrounded answers by over 10% (Abstract, Section 5.2.2) is not established by the evidence in the paper. Section 4.1.3 defines Answer Verification as a binary LLM decision (Verify_theta), and Section 5.1 uses GPT-4o as the semantic evaluator for accuracy; grounding itself is never measured with an external metric. The only direct grounding analysis (Appendix C) is a manual, qualitative categorization of Search-o1 errors, not ComposeRAG's ungrounded-answer rate. The stated claim that Search-o1-only correct answers lack direct support in 18% of cases (Section 5.2.2) is an analysis of baseline failures, not a measured grounding rate for ComposeRAG. If the 10% reduction is instead inferred from verification abstentions, that conflates the verifier's own judgment with an independent grounding measure. Because the pipeline's verifier is the same type of LLM judgment used to evaluate outputs, systematic leniency or style bias would directly inflate both accuracy and grounding claims. Without calibration or human agreement checks on the verifier and evaluator, the grounding advantage remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ComposeRAG, a modular RAG pipeline for multi-hop question answering in which the reasoning process is decomposed into atomic modules such as Question Decomposition, Question Construction, Query Rewriting, Retrieval Decision, Passage Reranking, Answer Generation, Answer Verification, and Final Answering. A self-reflection loop re-decomposes and re-executes the pipeline when final-answer verification fails. The authors evaluate ComposeRAG on HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle, comparing against RQ-RAG and Search-o1, and report accuracy gains of up to 15% over fine-tuning-based methods and up to 5% over reasoning-specialized pipelines, plus reductions in ungrounded answers of over 10% in low-quality retrieval settings. Ablation and module-upgrade experiments are presented to support the claim that the architecture is well-composed. The code is stated to be released.","tokens_in":28799,"tokens_out":5537,"duration_ms":50885,"significance":"If the empirical claims were fully supported, ComposeRAG would be a useful demonstration that modular, prompt-based pipelines can match or exceed task-specialized models while offering transparency and component-level upgradability. The paper's formal criteria for a well-composed system (P1 and P2), the four-dataset evaluation, and the claimed code release are assets. However, the headline comparisons and the grounding claim are not currently established by controlled evidence, so the significance of the contribution is contingent on substantial revision.","major_comments":[{"comment":"The RQ-RAG comparison is not a controlled test of ComposeRAG. The comparison changes the LLM (fine-tuned llama2-7b to Llama3.1-8b), the retriever (e5-base-v2 to Cortex Search), and the corpus preprocessing (FlashRAG wiki2018 chunks vs. the authors' own KILT-based preprocessing) simultaneously. The abstract's claim of up to a 15% accuracy improvement over fine-tuning-based methods therefore cannot be attributed to ComposeRAG's modular architecture. Please provide a same-backbone, same-retriever, same-corpus comparison, or explicitly rephrase the claim as a combined system comparison.","section":"Section 5.2.1, Table 1, Appendix B"},{"comment":"The grounding improvement claim is not directly measured. The abstract states that ComposeRAG reduces ungrounded answers by over 10% in low-quality retrieval settings, but Section 5.2.2 only reports that 18% of Search-o1-only correct answers lack direct support, and Appendix C.2 categorizes Search-o1's failure modes (totaling 10.4% in Table 8). No external groundedness metric, such as NLI-based faithfulness, citation-precision scoring, or human annotation, is applied to ComposeRAG outputs. Please measure grounding and abstention rates directly for ComposeRAG and for each baseline, under both low-quality and standard retrieval conditions.","section":"Section 5.2.2, Appendix C"},{"comment":"There is a potential circularity in the evaluation. Answer Verification is defined in Section 4.1.3 as a binary LLM judgment, and the main accuracy metric in Section 5.1 uses GPT-4o as the LLM evaluator, while GPT-4o and GPT-4o-mini are also backbones of ComposeRAG. No calibration, human-agreement check, or independent metric is reported for either the verifier or the evaluator. This is especially consequential for the grounding claim, because the same verification module decides which answers are retained and which are abstained on.","section":"Section 4.1.3, Section 5.1"},{"comment":"The claim that modules provide distinct and additive contributions is not fully supported by the ablation table. On Hotpot-200, Cover-EM drops from 0.510 with QD+QC+PR to 0.505 when all modules are active, and no significance tests or confidence intervals are provided for the 200-example subsets. The non-monotonic behavior should be reconciled with property P1, and the results should be accompanied by variance estimates or repeated sampling.","section":"Table 3, Section 6.2"},{"comment":"The Search-o1 comparison also confounds the architecture with the choice of backbone: ComposeRAG uses Qwen2.5-72B-Instruct while Search-o1 uses QwQ-32B, even though the retriever and corpus are matched. Moreover, on 2WikiMultiHopQA Search-o1 outperforms ComposeRAG (average 0.789 vs. 0.723). The 'up to 5% gain over reasoning-specialized pipelines' should therefore be characterized as a system-level comparison rather than a property of ComposeRAG alone, and the datasets where ComposeRAG does not win should be discussed.","section":"Table 1, Section 5.2.2"}],"minor_comments":[{"comment":"The phrase '15% accuracy improvement' is ambiguous between absolute percentage-point differences and relative improvements; please state the comparison convention explicitly.","section":"Abstract, Section 1"},{"comment":"The prompt examples contain typos, such as 'captital city' in the Question Decomposition example and 'rertieval' in the Retrieval Decision section; these should be corrected before publication.","section":"Appendix D"},{"comment":"The code release is referenced only as 'Arctic Agentic RAG' without a URL or repository identifier; please provide a complete link or anonymized repository for reproducibility.","section":"Section 1, Code Release"},{"comment":"The comparison with ReARTeR and R1-Searcher uses numbers reported in other papers under potentially different evaluation settings; the caveat in the caption is appropriate, but the main text should avoid drawing strong conclusions from this comparison.","section":"Table 2"},{"comment":"Cover EM rewards verbose predictions that contain a ground-truth span as a contiguous subsequence; consider also reporting exact match or span precision to avoid inflating scores for over-generated answers.","section":"Section 5.1, Cover EM"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an under-review preprint and the code link appears to be a placeholder. The modular formalization and the honest appendix categorization are strengths, but the headline accuracy and grounding claims need stronger, better-controlled evidence. I recommend major revision rather than rejection because the architecture is plausible and the issues appear fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid systems contribution: it takes the familiar pieces of multi-hop RAG—decomposition, rewriting, reranking, verification—and assembles them into a genuinely modular pipeline with a self-reflection loop that re-decomposes the question when final verification fails. That specific integration is new, and the authors do the field a service by describing each module's interface and prompts in enough detail to reimplement. The ablations are the best part: they show each module adds a measurable increment, that upgrading one module at a time improves performance, and that early exit via the Retrieval Decision and Simple QA paths cuts tokens by about a third without hurting accuracy. The code and prompt templates are released, which makes this reproducible in principle.\n\nThe soft spots are in the headline claims, not in the architecture. The 15% gain over RQ-RAG is not a controlled comparison: it changes the LLM (Llama2-7b to Llama3.1-8b), the retriever (e5 to Cortex Search), and possibly the corpus preprocessing at once. The authors explain why in Appendix B—the fine-tuned model can't follow their prompts—but that means the gain is confounded. The grounding reduction claim is weaker: I could not find a direct measurement of ComposeRAG's ungrounded-answer rate. The 'over 10%' figure appears to be inferred from the verifier abstaining, or from a categorization of Search-o1's errors in Appendix C, not from a paired external groundedness metric on ComposeRAG's outputs. The verifier itself is a binary LLM call with no calibration or human-agreement check, and the accuracy evaluator is GPT-4o, the same model family as a backbone. That doesn't sink the paper, but it means the grounding claim should be softened or re-measured.\n\nThe ablations are on 200 examples and have no significance tests; the differences are small (e.g., 0.505 to 0.510 Cover-EM), so I wouldn't hang much on the exact increments. The self-reflection trend in Figure 5 is more convincing.\n\nWho is this for? Anyone building modular RAG systems or trying to compare prompt-based vs fine-tuned vs agentic approaches on multi-hop QA. It deserves a serious referee: the engineering is careful, the code is out, and the modularity claims are testable. I'd recommend the editor send it to review, with a clear request that the authors add an external groundedness measure and either run a controlled RQ-RAG comparison or drop the 15% language.","headline":"A well-engineered modular RAG system with a convincing ablation story, but the headline grounding and accuracy gains are not measured cleanly enough to support the abstract's claims.","tokens_in":29340,"tokens_out":2698,"would_cite":true,"duration_ms":24252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modular, verification-first RAG pipeline with a self-reflection loop claims up to 15% accuracy gains over fine-tuned baselines and over 10% fewer ungrounded answers in low-quality retrieval settings.","keywords":["multi-hop question answering","retrieval-augmented generation","modular pipeline","self-reflection","answer verification","grounding fidelity","question decomposition","query rewriting"],"falsifier":"Take the set of ComposeRAG answers that passed its own Answer Verification, plus the set it abstained on, and have human annotators or an independent non-LLM grounding metric judge them, on the same 500-question subsets used in the paper; if the verification-pass rate is no better than chance at separating grounded from ungrounded answers, or if the abstained answers are mostly correct, then the grounding-fidelity claim collapses.","tokens_in":28361,"feed_emoji":"🧩","tokens_out":6780,"duration_ms":59085,"temperature":0.7,"pith_summary":"The paper argues that monolithic retrieval-augmented generation pipelines, where query reformulation, retrieval, reasoning, and verification are tightly coupled, can be replaced by a set of atomic, composable modules that each act as a parameterized transformation on structured inputs and outputs. It claims that on four multi-hop QA benchmarks, this modular pipeline consistently outperforms fine-tuning-based and reasoning-specialized baselines under matched retrieval conditions, with accuracy gains up to 15% over fine-tuned methods and up to 5% over agentic reasoning pipelines. The load-bearing design is a verification-first loop: every sub-answer and the final answer must be checked against cited passages, and verification failure triggers self-reflection that analyzes the reasoning trace, re-decomposes the question, and re-executes the pipeline up to three times. The paper reports that this design reduces ungrounded answers by over 10% in low-quality retrieval settings and by about 3% even with strong corpora, at the cost of occasionally abstaining when evidence is insufficient. If true, this supports a modular, upgradeable, interpretable alternative to task-specific fine-tuning or monolithic agentic systems.","feed_headline":"Verification-first RAG design beats fine-tuned baselines by up to 15%.","feed_subtitle":"Modular, verification-first pipeline claims up to 15% accuracy gains and 10% fewer ungrounded answers on multi-hop QA.","key_machinery":"The central object is a formal notion of a well-composed system, defined by two properties: ablation, where removing module $M_i$ yields a non-positive performance change ($\\Delta P_{-i} = P(S) - P(S \\setminus M_i) \\ge 0$), and upgrade, where substituting a better module $M'_i$ yields a positive change ($\\Delta P_{+i} = P(S[M_i \\leftarrow M'_i]) - P(S) > 0$). The mechanism that carries the argument is the Answer Verification module, a binary LLM judgment $(y, r) = \\text{Verify}_\\theta(q, a, P)$ that checks both factual correctness and grounding against cited passages, together with the Self-Reflection loop that, on final verification failure, runs an error-analysis prompt to produce guided re-decomposition instructions and re-executes the multi-hop pipeline up to three times. Every other module is defined as a parameterized transformation on structured inputs and outputs, which is what makes independent ablation and independent upgrade meaningful.","core_discovery":"The central claim is that a retrieval-augmented multi-hop QA system can be decomposed into atomic modules—Question Decomposition, Question Construction, Retrieval Decision, Query Rewriting, Passage Reranking, Answer Generation, Answer Verification, and Final Answering—and that this decomposition is not merely cosmetic: each module contributes measurably, and replacing a module with a stronger version improves the whole system. The paper formalizes this as a well-composed system with two properties: removing a module must not improve performance (P1), and upgrading a module to a better version must improve performance (P2). The mechanism that carries the argument is the verification-first loop: every answer must be explicitly supported by cited passages, and when verification fails, the pipeline diagnoses the reasoning trace, generates guided re-decomposition instructions, and re-executes the multi-hop pipeline. The paper reports accuracy gains up to 15% over fine-tuning-based RQ-RAG and up to 5% over reasoning-specialized Search-o1 under identical retrieval conditions, alongside reductions in ungrounded answers of over 10% in low-quality retrieval settings and about 3% with strong corpora.","pith_inferences":["A testable extension the paper does not run: apply ComposeRAG's Answer Verification module as a post-hoc filter on Search-o1's outputs, then measure how much accuracy survives when ungrounded answers are removed; this would isolate how much of the grounding difference is due to verification alone rather than to the full modular pipeline.","Because the modules are defined entirely by prompts and structured input/output formats, the same decomposition, verification, and self-reflection loop should transfer to other retrieval corpora or tool-augmented settings, such as question answering over private documents; the paper only evaluates Wikipedia-based open-domain benchmarks.","An honest reading should treat the reported module-level effect sizes as coarse: the ablations use 200-example subsets and a single GPT-4o evaluator, so the stable signal is likely the ordering and general direction of each module's contribution rather than the exact percentage points."],"forward_implications":["If the claim holds, task-specific fine-tuning of retrieval modules becomes unnecessary: a general instruction-tuned LLM in a modular prompt pipeline can match or beat a fine-tuned system across all four tested benchmarks.","If verification-first abstention is the source of the grounding gains, then adding a verification loop to existing monolithic RAG systems may reproduce part of the improvement without redesigning the whole pipeline.","If P1 and P2 hold, then upgrading a single module (for example, swapping GPT-4o-mini for GPT-4o in Question Decomposition) yields measurable, additive gains, so progress in individual LLMs transfers directly to the full system.","If self-reflection's benefit is monotone in the number of reflection steps (Cover-EM rising from 0.340 to 0.375 and LLM Eval from 0.375 to 0.415 on the MuSiQue-200 subset), then bounded re-decomposition is a cheap robustness mechanism.","If the Simple QA pipeline and Retrieval Decision maintain accuracy while cutting average token use by about 32%, then modular routing can improve efficiency without sacrificing answer quality."],"supporting_citations":[{"why":"Supplies the fine-tuning-based baseline (RQ-RAG) whose reported accuracy ComposeRAG claims to beat by up to 15%.","marker":"[5]"},{"why":"Supplies the agentic, reasoning-specialized baseline (Search-o1) compared under identical retriever and corpus settings.","marker":"[18]"},{"why":"HotpotQA benchmark used for primary multi-hop QA evaluation.","marker":"[32]"},{"why":"2WikiMultiHopQA benchmark used for evaluation.","marker":"[10]"},{"why":"MuSiQue benchmark used for evaluation.","marker":"[25]"},{"why":"Bamboogle benchmark (full 125-question test set) used for evaluation.","marker":"[20]"},{"why":"FlashRAG provides the truncated 2018 Wikipedia corpus and the reproduction setup for the RQ-RAG baseline.","marker":"[14]"},{"why":"KILT supplies the preprocessed 2019 Wikipedia corpus used for the ComposeRAG versus Search-o1 comparisons.","marker":"[19]"},{"why":"Justifies the use of GPT-4o as an automatic evaluator for semantic alignment between predicted and ground-truth answers.","marker":"[8]"}],"fun_headline_variants":["Modular RAG decomposes pipelines for 15% accuracy boost","Verification-first loop cuts ungrounded answers by 10%","Composable modules give RAG a 15% edge on multi-hop QA","Self-correcting RAG pipeline improves accuracy and grounding","Modular RAG with verification boosts multi-hop QA by 15%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Answer Verification module and the GPT-4o evaluator produce reliable binary judgments of correctness and grounding; if these LLM verdicts are systematically lenient or biased toward ComposeRAG's own output style, the reported accuracy and grounding improvements are not actually measured.","fun_headline_variants_meta":{"raw":{"variants":["Modular RAG decomposes pipelines for 15% accuracy boost","Verification-first loop cuts ungrounded answers by 10%","Composable modules give RAG a 15% edge on multi-hop QA","Self-correcting RAG pipeline improves accuracy and grounding","Modular RAG with verification boosts multi-hop QA by 15%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3241,"prompt_tokens":1078,"completion_tokens":2163,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":2071}},"tokens_in":694,"tokens_out":2163,"duration_ms":13824,"temperature":1.0,"reasoning_tokens":2071,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:09:13.231700+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the set of ComposeRAG answers that passed its own Answer Verification, plus the set it abstained on, and have human annotators or an independent non-LLM grounding metric judge them, on the same 500-question subsets used in the paper; if the verification-pass rate is no better than chance at separating grounded from ungrounded answers, or if the abstained answers are mostly correct, then the grounding-fidelity claim collapses.","supporting_citations":[{"cited_title":"♪ musique: Multihop questions via single-hop question composition","cited_arxiv_id":null,"evidence_quote":"MuSiQue benchmark used for evaluation."}],"review_version":1}