{"id":"f9143bc1-538c-415a-9121-847481b408b4","arxiv_id":"2606.21553","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Ablation on local 7B agentic RAG shows fixed hybrid retrieval outperforms adaptive routing and two iterations capture nearly all gains from five on HotpotQA.","lead":"The paper ablates components of an agentic RAG system for multi-hop QA using a local 7B model on 5000 HotpotQA questions. It finds that fixed hybrid retrieval and short loops deliver most gains while adaptive routing and extra iterations add little.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Conclusion that fixed hybrid beats adaptive routing rests on one weak rule-based heuristic","rationale":"The load-bearing concern is identical to the reader's weakest_assumption. No stronger internal inconsistency or methodological flaw is evident from the reported design; the abstract-only limitation noted by the reader is the source of the low confidence, but the identified assumption remains the precise point that would alter the interpretation if violated.","tokens_in":1857,"tokens_out":308,"duration_ms":15740,"concrete_test":"Replace the rule-based router with a learned one: fine-tune or prompt the same 7B model to output a binary routing decision (BM25 vs. dense) on a held-out set of 500 HotpotQA sub-questions, then re-run the full ablation pipeline and compare EM/F1 against the fixed RRF baseline; if the learned router matches or exceeds fixed hybrid, the headline ranking reverses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes the +1.8 EM advantage of fixed RRF hybrid over adaptive routing to the specific heuristic over-routing to BM25 whenever named entities appear (common in nearly all HotpotQA sub-questions). This shows only that this particular rule-based router underperforms; it does not demonstrate that adaptive routing in general is inferior. The paper's broader interpretation ('simpler and fixed choices turn out to be competitive with or better than their adaptive versions') therefore depends on the untested assumption that no stronger adaptive router exists.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper reports a controlled ablation study of an agentic RAG pipeline for multi-hop QA on 5,000 HotpotQA distractor dev questions using a local Qwen2.5-7B-Instruct model. The full pipeline reaches EM=53.2% and F1=61.6% versus a single-pass dense baseline of 43.1%/54.0%. Across eight conditions it finds that fixed hybrid retrieval (RRF) outperforms a rule-based adaptive router by +1.8 EM / +1.9 F1 because the router over-routes to BM25 on named entities; two retrieval iterations capture 95% of the gain from five; and query decomposition plus cross-encoder reranking each add smaller but statistically significant improvements (p<0.01, p<0.001). The authors conclude that, on a fixed local-model budget, simpler fixed choices are competitive with or better than adaptive versions.","tokens_in":1949,"tokens_out":550,"duration_ms":22275,"significance":"If the empirical comparisons hold, the work supplies concrete, reproducible evidence on component contributions in resource-constrained agentic RAG, showing that most gains derive from a short retrieval loop rather than adaptive routing or deeper iteration. The large sample, reported p-values, and exclusive use of local models without proprietary APIs are clear strengths that support reproducibility and practical relevance.","major_comments":[{"comment":"Abstract and final paragraph: the claim that 'simpler and fixed choices turn out to be competitive with or better than their adaptive versions' is load-bearing for the central interpretation, yet rests on a single rule-based routing heuristic whose failure mode (over-routing to BM25 on named entities common to nearly all sub-questions) is explicitly diagnosed. The result demonstrates only that this particular heuristic underperforms fixed RRF; it does not test whether a stronger learned router would reverse the ranking. This assumption is stated as the weakest in the evaluation and directly limits the broader conclusion.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract states p<0.01 and p<0.001 for decomposition and reranking but does not name the statistical test or correction method; adding this detail would clarify the significance claims.","section":"Abstract"},{"comment":"The 5,000-question subset is drawn from the HotpotQA distractor dev set; the paper should explicitly state the sampling procedure and confirm that the same questions are used across all ablation conditions to ensure comparability.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for highlighting the scope of our claims on adaptive routing. We address the major comment below and will revise the manuscript accordingly.","responses":[{"response":"We agree that the load-bearing claim generalizes from the specific rule-based heuristic tested. The manuscript already diagnoses the over-routing failure mode on named entities. To align the interpretation with the evidence, we will revise the abstract and final paragraph to state that the tested rule-based adaptive routing underperforms fixed RRF, rather than referring to adaptive versions more broadly. This revision will be incorporated in the next version. Evaluating whether a learned router could reverse the result would require additional training data and compute outside the current ablation of off-the-shelf local-model components.","revision_made":"yes","referee_comment":"[Abstract] Abstract and final paragraph: the claim that 'simpler and fixed choices turn out to be competitive with or better than their adaptive versions' is load-bearing for the central interpretation, yet rests on a single rule-based routing heuristic whose failure mode (over-routing to BM25 on named entities common to nearly all sub-questions) is explicitly diagnosed. The result demonstrates only that this particular heuristic underperforms fixed RRF; it does not test whether a stronger learned router would reverse the ranking. This assumption is stated as the weakest in the evaluation and directly limits the broader conclusion."}],"tokens_in":1557,"tokens_out":298,"duration_ms":18686,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The key things to know are that a fixed reciprocal rank fusion hybrid beats the tested rule-based adaptive router by 1.8 EM and 1.9 F1, and that two retrieval iterations on decomposed sub-questions get 95% of the lift from five. The full pipeline hits 53.2 EM / 61.6 F1 against a 43.1 / 54.0 single-pass baseline on 5000 HotpotQA distractor questions with Qwen2.5-7B-Instruct.\n\nThe work does a clean job of running controlled ablations across eight conditions and reporting concrete deltas plus p-values. Query decomposition and cross-encoder reranking each add statistically detectable but smaller gains. The mechanistic note that the router over-fires on named entities (common in nearly every sub-question) explains why adaptive underperformed here. All of this is done with only local models and no proprietary APIs, which matches the resource-constrained setting they target.\n\nThe soft spot is exactly the one the stress-test flags: the adaptive comparison rests on a single rule-based heuristic. That shows this particular router is not helpful, but it leaves open whether a learned router could reverse the ranking. The paper's broader takeaway that simpler fixed choices are competitive or better therefore depends on an untested assumption about the space of adaptive methods. The evaluation details (exact prompts, data splits, implementation of the router) are not visible in the abstract, so replication would require the code.\n\nThis is useful for people running multi-hop RAG on local 7B-class models who need guidance on which knobs actually matter. It is not a theoretical advance and the citation pattern is light, but the empirical measurements are internally consistent and falsifiable. It deserves peer review because the ablation design is straightforward and the reported numbers give practitioners something concrete to test against.","headline":"The ablation shows fixed hybrid retrieval beats this one rule-based adaptive router by 1.8 EM on HotpotQA with a 7B model, and two iterations capture most gains, but the adaptive result is tied to a weak heuristic.","tokens_in":2413,"tokens_out":467,"would_cite":false,"duration_ms":14159,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Fixed hybrid retrieval via reciprocal rank fusion outperforms rule-based adaptive routing in agentic RAG for multi-hop QA with a local 7B model.","keywords":["agentic RAG","multi-hop QA","ablation study","local language models","retrieval routing","query decomposition","HotpotQA"],"falsifier":"Replace the rule-based router with a learned router and re-run the comparison to check whether adaptive routing then outperforms fixed hybrid retrieval.","tokens_in":2747,"feed_emoji":"🔬","tokens_out":608,"duration_ms":21602,"temperature":0.7,"pith_summary":"The paper runs a controlled ablation of an agentic RAG pipeline on 5,000 HotpotQA questions using only a local 7B model. It compares eight conditions that vary retrieval routing, iteration depth, query decomposition, and reranking. The results show fixed hybrid retrieval beats the adaptive router, two iterations capture nearly all the benefit of five, and decomposition plus reranking add smaller gains. This indicates that on a fixed local-model budget simpler fixed components often match or exceed more elaborate adaptive designs.","feed_headline":"Fixed hybrid retrieval beats adaptive routing in 7B RAG","feed_subtitle":"Ablations on 5000 HotpotQA questions show two iterations capture 95% of gains from deeper loops.","key_machinery":"Controlled ablation across eight conditions of the agentic RAG pipeline, isolating effects of fixed versus adaptive retrieval routing, retrieval loop depth, query decomposition, and reranking.","core_discovery":"The full pipeline reaches 53.2% EM and 61.6% F1. Fixed hybrid retrieval via reciprocal rank fusion consistently outperforms rule-based adaptive routing by 1.8 EM and 1.9 F1 points because the routing heuristic over-routes to BM25 on named entities present in nearly all sub-questions. Two retrieval iterations over decomposed sub-questions capture 95% of the gains from five iterations, while query decomposition and cross-encoder reranking each contribute statistically significant but smaller gains.","pith_inferences":["A stronger learned router might reverse the ranking between fixed and adaptive approaches.","The same pattern of short loops sufficing may appear on other multi-hop QA datasets.","Reducing iteration depth could lower latency and compute with little accuracy loss."],"forward_implications":["Most of the performance gain comes from running a short retrieval loop rather than adaptive routing or many iterations.","On a fixed local-model budget, simpler and fixed choices turn out to be competitive with or better than their adaptive versions.","Query decomposition and cross-encoder reranking each add statistically significant but smaller gains."],"fun_headline_variants":["Fixed hybrid tops adaptive in 7B RAG","Two loops capture 95% gains in 7B multi-hop QA","Adaptive routing over-routes on entities in 7B RAG","Ablations favor fixed choices in 7B local RAG","Deep loops add little beyond two iterations in 7B RAG"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The particular rule-based adaptive routing heuristic tested is representative of adaptive routing methods in general.","fun_headline_variants_meta":{"raw":{"variants":["Fixed hybrid tops adaptive in 7B RAG","Two loops capture 95% gains in 7B multi-hop QA","Adaptive routing over-routes on entities in 7B RAG","Ablations favor fixed choices in 7B local RAG","Deep loops add little beyond two iterations in 7B RAG"]},"model":"grok-4.3","cost_usd":0.006156,"raw_usage":{"total_tokens":2979,"prompt_tokens":818,"num_sources_used":0,"completion_tokens":87,"cost_in_usd_ticks":61562000,"prompt_tokens_details":{"text_tokens":818,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2074,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":818,"tokens_out":87,"duration_ms":15506,"temperature":1.0,"reasoning_tokens":2074,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T14:11:26.830136+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Replace the rule-based router with a learned router and re-run the comparison to check whether adaptive routing then outperforms fixed hybrid retrieval.","supporting_citations":[],"review_version":1}