{"id":"21a21a96-b8f8-47c3-8244-c9d317fb14b0","arxiv_id":"2506.00054","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of retrieval-augmented generation methods, organized by architecture, with a comparative analysis of published QA results.","lead":"This paper surveys dozens of systems that let large language models look up external documents before answering, and groups them into four architecture families. It also builds comparison tables of reported performance on question answering benchmarks, showing where different designs gain or lose.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 relative-gain normalization is internally inconsistent with Appendix Tables 5–6; several headline narratives contradict their own source rows.","rationale":"I read the paper as a survey whose central deliverables are a taxonomy and a comparative performance analysis. The taxonomy is coherent, and the paper deserves credit for providing appendix source tables and for flagging incomplete metric coverage and dash entries. However, the strongest claim—that retriever-centric designs yield the largest relative gains on multi-hop QA while hybrid designs are more variable—depends entirely on Table 2's normalization. The reader identified the general risk that published numbers may not be directly comparable and that no protocol standardization or uncertainty quantification is provided. My concern is more specific and more load-bearing: the paper's own appendix does not consistently reproduce Table 2, regardless of which normalization convention one adopts. This is a concrete, internally checkable correctness risk rather than a disagreement with external consensus or an ad hominem issue. It does not undermine the survey's organizational value, but it does undermine the empirical ranking that the strongest claim asserts. The natural remedy is to recompute Table 2 from the appendix with one explicit formula and to correct or qualify any row that fails; hence the verdict remains CONDITIONAL, matching the reader's verdict but with a sharper and more concrete justification. My agreement is 'partial' because the reader pointed to the same table and the general issue of comparability, but did not identify the internal inconsistency in the normalization itself.","tokens_in":35618,"tokens_out":2036,"duration_ms":23210,"concrete_test":"Reproduce every non-dash cell of Table 2 from Tables 5–6 using one explicit, uniformly applied formula—either relative gain = framework/raw − 1 (ratio) or framework/raw − 1 expressed as a percentage—and flag each cell that deviates from the printed value by more than 0.5 percentage points (or 0.005 in ratio terms). If more than 10% of cells fail, the comparative analysis is not internally reproducible and the Section 5 conclusions about which architectural families yield the largest gains should be restated or removed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparative claim in Section 5 rests on the normalized relative gains in Table 2, and the paper invites traceability by providing source scores in Appendix Tables 5–6. Spot-checking reveals that Table 2 mixes inconsistent normalization conventions. For RQ-RAG on HotpotQA, Table 2 reports 8.485/2.749 (B/B+R), but Table 6 gives Raw LLM F1 = 6.6 and LLM+Retrieval F1 = 16.7, which implies 848.5% and 274.9%—i.e., Table 2's value is a percentage, not the multiplicative ratio stated in Section 5.1. Conversely, SELF-RAG LLaMA2-13B on PopQA is listed as 2.796/0.221 (Acc); Table 5 gives 55.8/14.7, which implies 280% (a ratio of 3.8), not 2.796. Similarly, xRAG Mixtral-8x7B on NQ is listed as 0.126/0.047 (EM); Table 5 gives 47.28/41.99 and 47.28/45.15, which imply 12.6% and 4.7%, not 0.126 and 0.047. SimRAG's ARC entries (0.145/- and 0.034/-) are printed with no denominator, while Table 5 shows LLM+Retrieval values of 71.08 and 85.75; treating 0.145 and 0.034 as percentages yields 14.5% and 3.4% gains, which do not match 81.4/71.08 and 88.65/85.75. The Section 5 text then interprets some values as percentages ('over 270% improvement', 'over 800% improvement'), which only works for the percentage convention, not the ratio convention stated in the table caption. Because the table cannot be reproducibly derived from the paper's own appendix, the claim that retrieval-based systems consistently yield the largest gains is not internally supported; a single transcription error or convention ambiguity could change the ranking of framework families.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey proposes a taxonomy of retrieval-augmented generation (RAG) architectures—retriever-centric, generator-centric, hybrid, and robustness-oriented—and reviews enhancements in retrieval, filtering, efficiency, robustness, and reranking. The paper presents comparative performance analyses of representative RAG frameworks on short-form and multi-hop QA, a robustness comparison, and a review of evaluation benchmarks. The central evidence for the comparative claims is Table 2 (relative gains over raw LLM and LLM+retrieval baselines) and Table 3 (robustness gains), with Appendix Tables 5–7 providing the original reported scores.","tokens_in":98,"tokens_out":12379,"duration_ms":195117,"significance":"If the comparative analysis were reliable, the survey would provide a useful map of the RAG design space and a starting point for model selection. The proposed taxonomy is reasonable and the appendix's inclusion of raw reported scores is a welcome transparency measure. However, the central tables contain several transcription, labeling, and convention errors that currently prevent the paper's main comparative claims from being verified from its own appendix. The paper's contribution is therefore weakened until these issues are resolved. The survey also covers a broad and current set of systems, which is valuable for the community, but the presence of self-inconsistencies in the empirical core makes the 'comprehensive' and 'comparative' claims premature.","major_comments":[{"comment":"The central comparative table (Table 2) is not reproducible from the appendix for at least two rows. In the R2AG/LLaMA2-7B row, the 2Wiki entry '34.52/4.445' mixes an absolute F1 score (34.52, from Table 6) with a relative-gain value (4.445 = (34.52-6.34)/6.34). Since Table 2 promises relative improvements in both columns, this entry would be read as a 3452% gain rather than the correct 444.5% gain. In the same table, the SimRAG/ARC entries (0.145 and 0.034) are placed in the 'B' (raw-LLM) column even though Appendix Table 5 supplies no raw-LLM score for SimRAG and the values correspond to (81.4-71.08)/71.08 and (88.65-85.75)/85.75, i.e., gains over LLM+retrieval. These errors directly affect the Section 5.1 and 5.2 narratives that attribute large gains to retriever-based designs.","section":"Table 2 / Appendix Table 6"},{"comment":"Section 5.3 and Table 3 contain several label/value mismatches that undermine the robustness comparison. (i) For FILCO/FEVER, Table 3 reports 3.25 in a column of relative improvements, where Flare-Direct's 0.216 is interpreted by the text as 21.6%; read consistently, 3.25 would be a 325% relative gain, but the text says '+3.25%'. (ii) Table 3 lists two 'SELF-RAG LLaMA2-7B ASQA' rows with different values; the second row's values match the LLaMA2-13B scores in Table 7. (iii) The sentence 'Self-RAG and CRAG, reporting +0.372 and +0.252 gains on the same dataset' is unsupported: no Self-RAG/Biography row exists in either Table 3 or Table 7. These issues cloud the claimed ordering of robustness gains.","section":"Section 5.3 / Table 3 / Table 7"},{"comment":"The claimed empirical basis of Section 5.1 is incomplete: RAAT is discussed as showing a 116% raw-baseline and 27% retrieval-baseline improvement on RAG-Bench, yet RAAT is absent from Table 2, which is the table the section opens by referencing. The values are present in Appendix Table 5, so this is fixable, but as written the headline claim is not traceable to the table that the section says it summarizes.","section":"Section 5.1 / Table 2 / Table 5"},{"comment":"The relative-gain comparisons are computed from highly heterogeneous raw scores, which can make the rankings fragile; for example, SELF-RAG's PopQA gains are computed from a raw-LLM accuracy of 14.7, while RQ-RAG's HotpotQA gain is computed from a raw F1 of 6.6. The paper should provide explicit protocol-compatibility caveats, confidence intervals, or error bars, or restrict claims to qualitative ordering. Without this, a single outlier such as the R2AG 2Wiki entry can materially change the conclusion that retriever-based systems are the most consistent winners.","section":"Section 5 / Table 2 / Table 6"},{"comment":"The metric labels in Tables 2 and 3 need to be reconciled with the appendix. Stochastic RAG/HotpotQA is labeled F1 in Table 2 but EM in Table 6; Re2G KGI1/TriviaQA Precision in Table 3 has no corresponding appendix row (Table 7 lists KGI0 rows and a TriviaQA recall, not precision). These mismatches prevent a reader from verifying the reported gains and should be corrected systematically.","section":"Tables 2 and 3 / Appendix Tables 6 and 7"}],"minor_comments":[{"comment":"The text states that RAAT improves F1/EM by 20–30%, but Appendix Table 5 shows a 116% improvement over the raw baseline and 27% over the retrieval baseline; please clarify which baseline the 20–30% figure refers to.","section":"Section 4.4 / Table 5"},{"comment":"The sentence 'xRAG achieves 10–29% improvements over raw LLM baselines' seems to exclude the Mixtral-8x7B TriviaQA entry, which is 0.043 (4.3%); the range should be 4–29%.","section":"Section 5.1 / Table 2"},{"comment":"The claim that TA-ARE reduces redundant retrievals by 14.9% is not supported by an appendix row or a specific citation in the text; please provide the source or add an appendix entry.","section":"Section 4.1 / Table 1"},{"comment":"Table 1 reports a 20–50% TTFT reduction for Speculative Pipelining, while the text says 20–30%; please reconcile these numbers and report the actual range from the cited paper.","section":"Section 4.3 / Table 1"},{"comment":"The Table 1 entry for RAGCache says it 'eliminates recomputation,' while the text correctly limits this to high-throughput workloads; please soften the table description to match the text.","section":"Section 4.3 / Table 1"},{"comment":"Equation (2) presents the truncated sum as an approximation of Equation (1), but it is not a normalized approximation since P(d|x) is not renormalized over the top-k set; please clarify the nature of the approximation.","section":"Section 2.2 / Equation (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a broad survey submitted to TOIS, a top IR journal; the taxonomy and coverage are relevant, but the empirical core currently contains enough internal inconsistencies that the comparative claims cannot be used as-is. I would suggest the editor require a careful re-derivation of Tables 2 and 3 from the appendix, with column semantics made uniform and every row traced to the cited source. The paper also overlaps substantially with prior RAG surveys (e.g., reference [19]); the authors should clarify what new synthesis or analysis the manuscript adds beyond existing surveys."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent, well-organized survey of recent RAG work, and it is more internally consistent than the stress-test note suggests. It is not a breakthrough—the taxonomy reorganizes existing categories and the comparative analysis is arithmetic on published numbers—but as a map of the current design space it has real value.\n\nThe strongest parts are the organization and the traceability. The four-way split (retriever-centric, generator-centric, hybrid, robustness) is clean and captures most of the recent systems I care about. Appendix Tables 5–7 give source scores, and I spot-checked a good chunk of Table 2 against them: the relative-gain convention is applied consistently. For example, RQ-RAG on HotpotQA: (62.6−6.6)/6.6 = 8.485, and (62.6−16.7)/16.7 = 2.749, matching the table. SELF-RAG on PopQA: (55.8−14.7)/14.7 = 2.796, (55.8−45.7)/45.7 = 0.221, matching. The stress-test's specific accusations on those rows do not hold up. Section 5.1's statement that 2.7 means 270% is consistent with this convention.\n\nThe real soft spots: R2AG's 2Wiki row in Table 2 lists 34.52/4.445, but the appendix gives raw=6.34, framework=34.52, so the left number should be (34.52−6.34)/6.34 = 4.445, not 34.52. That looks like a transcription error. SimRAG's ARC row reports 0.145/- and 0.034/-; the appendix only reports LLM+Retrieval baselines, so those gains are relative to B+R, not B, and the column label is misleading. More generally, the paper uses relative gains without stating the formula explicitly and without any uncertainty or protocol standardization. Since some baselines are tiny (MuSiQue raw=3.0), a modest absolute gain becomes a large relative number. The 'comprehensive' claim also overreaches: there is no systematic search or explicit positioning against prior taxonomies, and some text claims (e.g., TA-ARE's 28×) are not in Table 2.\n\nBottom line: the paper is useful for a reader who wants a structured overview and a quick comparative sense of RAG families, but the comparative numbers should be treated as indicative, not precise. It deserves refereeing; with the error fixed, the labeling corrected, and a candid caveat about comparability, it would be a solid, citable survey. I would not cite it myself in the next year, but I'd happily point a student to it.","headline":"A solid, incremental RAG survey with mostly reproducible tables; one real transcription error and a weak comparability story, but worth refereeing.","tokens_in":36511,"tokens_out":7837,"would_cite":false,"duration_ms":60202,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A survey of retrieval-augmented generation organizes the field into four architectural families and shows which family delivers the largest gains on question answering.","keywords":["Retrieval-Augmented Generation","architecture taxonomy","query reformulation","context filtering","reranking","multi-hop reasoning","robustness","evaluation benchmarks"],"falsifier":"Recompute each relative improvement in Table 2 from the appendix's raw scores and check against the original papers; if a cornerstone value such as RQ-RAG's reported ~850% HotpotQA F1 gain fails to reproduce, the comparative ranking of design families collapses.","tokens_in":35397,"feed_emoji":"🔍","tokens_out":5584,"duration_ms":57911,"temperature":0.7,"pith_summary":"This paper is a survey that tries to establish a dependable map of retrieval-augmented generation (RAG) design space. It proposes a taxonomy of four architectural families, retriever-centric, generator-centric, hybrid, and robustness-oriented, and then uses relative performance gains across short-form and multi-hop question answering to show which design choices tend to pay off. The value, if the survey is right, is that a practitioner can look at the architecture family and get a realistic expectation of how much improvement to expect over a raw LLM or a standard retrieval baseline, before running their own evaluation. The analysis also claims that retrieval optimization is the most consistent driver of large multi-hop gains, while closed-loop verification is what produces robustness.","feed_headline":"Retriever-side fixes deliver the biggest RAG gains","feed_subtitle":"A four-way taxonomy and normalized comparisons across 30+ frameworks show where retrieval, generation, and robustness work pays off.","key_machinery":"The organizing device is a taxonomy that sorts RAG systems by where the core innovation sits: retriever-centric, generator-centric, hybrid, or robustness-oriented. Carrying the empirical argument is a relative-gain normalization that converts raw reported scores (F1, EM, Accuracy, FactScore) into improvements over two baselines, the raw backbone LLM (B) and the same backbone with standard retrieval (B+R), enabling cross-paper comparison in Table 2 and Table 3. The formal foundation is the conditional decomposition $P(y|x) = \\sum_{d} P(y|x,d) P(d|x)$, which the taxonomy operationalizes by asking which component the system improves.","core_discovery":"The central claim is that the design space of RAG systems is well captured by four architectural orientations, and that reported performance differences across these families form a coherent pattern: retriever-side innovations such as query decomposition, reranking, and granularity control produce the largest and most consistent relative gains on multi-hop QA; generator-side mechanisms like self-reflection help short-form factual recall; and hybrid systems that add correctness-checking loops deliver the strongest robustness improvements in factual consistency (FactScore), with Self-CRAG reporting +0.456 over its retrieval-augmented baseline on Biography. The survey further claims that retrieval alone is insufficient for robust generation, and that the strongest systems couple retrieval, generation, and verification in iterative loops.","pith_inferences":["Editorial extension: a practitioner-facing heuristic follows — for knowledge-intensive multi-hop tasks, improve retrieval first (decomposition, reranking, granularity), and add a self-critique loop when factual consistency matters more than latency.","Editorial extension: the relative-gain normalization could be stress-tested by re-evaluating a handful of highlighted systems on one shared benchmark; the rank order of the four design families should persist if the survey's comparative conclusions are genuinely about architecture rather than evaluation luck.","Editorial extension: the negative gains of extreme context compression (xRAG) suggest a testable hypothesis that compression trades away evidence the generator needs for multi-hop composition; a controlled sweep of compression ratios across HotpotQA and MuSiQue would quantify that trade-off."],"forward_implications":["Retrieval-side investment is the highest-leverage move for multi-hop accuracy: query decomposition and reranking systems (RQ-RAG, LQR, RankRAG) report the largest relative gains over raw backbones.","Generator-side mechanisms such as self-reflection and context compression produce solid short-form gains but often lose to standard retrieval baselines when compression is extreme (xRAG's negative gains).","Robustness gains require closed-loop verification: critique-based hybrids (Self-CRAG, CRAG, SELF-RAG) lead FactScore improvements, while retrieval without verification (Stochastic RAG) shows near-zero gains.","The relative-gain comparison framework lets practitioners transfer expected performance improvements across different backbones and datasets, as long as they accept the normalization assumptions."],"supporting_citations":[{"why":"Supplies the largest multi-hop query-decomposition gains (HotpotQA >800% relative) that anchor the retriever-centric family.","marker":"[6]"},{"why":"Provides the self-reflective critique-generate mechanism and the PopQA/ARC gains used for generator-centric comparisons.","marker":"[1]"},{"why":"Provides corrective retrieval and the Self-CRAG variant with the largest FactScore improvement (+0.456) in the robustness analysis.","marker":"[79]"},{"why":"Supplies token-level adaptive retrieval triggers and multi-hop F1 gains for the hybrid family.","marker":"[61]"},{"why":"Supplies adversarial training results on RAG-Bench that anchor robustness-oriented claims.","marker":"[18]"},{"why":"Supplies unified reranking-generation gains (MRR@10 improvements) used in the reranking analysis.","marker":"[83]"},{"why":"Supplies the evaluation framework that the survey uses for retrieval-aware assessment dimensions.","marker":"[55]"}],"fun_headline_variants":["Survey: Retriever upgrades outperform generator tweaks in RAG","RAG survey: Retriever gains dominate, hybrid loops best for robustness","Retriever innovations drive biggest RAG improvements, study finds","RAG taxonomy: Retriever-centric wins multi-hop, hybrid wins robustness","Four RAG architectures compared: retriever fixes top multi-hop gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that relative improvements computed from numbers reported in different original papers remain comparable across backbones, prompts, and datasets, even though the evaluation protocols were never standardized.","fun_headline_variants_meta":{"raw":{"variants":["Survey: Retriever upgrades outperform generator tweaks in RAG","RAG survey: Retriever gains dominate, hybrid loops best for robustness","Retriever innovations drive biggest RAG improvements, study finds","RAG taxonomy: Retriever-centric wins multi-hop, hybrid wins robustness","Four RAG architectures compared: retriever fixes top multi-hop gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001092,"raw_usage":{"total_tokens":4553,"prompt_tokens":928,"completion_tokens":3625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":3530}},"tokens_in":544,"tokens_out":3625,"duration_ms":26538,"temperature":1.0,"reasoning_tokens":3530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:56:28.066772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute each relative improvement in Table 2 from the appendix's raw scores and check against the original papers; if a cornerstone value such as RQ-RAG's reported ~850% HotpotQA F1 gain fails to reproduce, the comparative ranking of design families collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides corrective retrieval and the Self-CRAG variant with the largest FactScore improvement (+0.456) in the robustness analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies unified reranking-generation gains (MRR@10 improvements) used in the reranking analysis."}],"review_version":1}