{"id":"b20706fb-2bb9-4106-9508-44c5ff7a2d5b","arxiv_id":"2506.12571","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"DoTA-RAG combines query rewriting, namespace routing, dense retrieval, BM25 pruning, and reranking to answer questions over a 15M-document corpus, with reported correctness gains but fragile faithfulness under output caps.","lead":"DoTA-RAG is a RAG system for the SIGIR 2025 LiveRAG Challenge that rewrites queries, routes them to topical sub-indexes, and mixes dense search with BM25 and reranking. It reports correctness gains on its own synthetic benchmark and a 0.929 live score, but its faithfulness collapsed under the competition's 300-word cap.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed correctness gain depends on automated judges that are never validated against humans, and the live run's near-zero faithfulness makes the 0.929 correctness score difficult to interpret; the reported numbers may not measure what the paper claims.","rationale":"The central claim is an engineering claim: the assembled pipeline improves correctness over baseline at low latency. What must be true is that the correctness measurements are valid, comparable, and not an artifact of the evaluation protocol. The paper provides no human ratings, no per-item judge-agreement statistics, no released benchmark or code; Table 2's 'agreement' is just two mean vectors. This matters because Table 3 shows an internal configuration (+Rewrite) that is worse on the paper's own metric being shipped, and the live evaluation shows a near-zero faithfulness score on the same system that allegedly achieves high correctness. The two observations are not mutually inconsistent, since correctness and faithfulness are different constructs, but they make the live score uninterpretable without the official scoring template. I agree with the reader's core concern about judge validity but only partially with the framing: the final internal numbers are Claude-judged, not Falcon3-judged; the Falcon3 self-judge issue affects design choices and earlier reported numbers rather than the headline Table 3 rows. There is independent support in the live submission and the coherent ablation trajectory, with the embedding swap providing the dominant jump, which is why I would not move the verdict to REJECT. The right outcome is to keep the conditional acceptance: the claim is plausible but not yet established without external validation or released artifacts.","tokens_in":9430,"tokens_out":7144,"duration_ms":77308,"concrete_test":"Take a stratified random sample of 100 baseline and 100 DoTA-RAG outputs (both uncut and 300-word-capped) from Morgana-MultiDocQA; have human raters score correctness and faithfulness with the paper's rubrics, and independently score the same outputs with Claude 3.5 Sonnet and Falcon3-10B. Compute per-item quadratically weighted kappa between each automated judge and the human majority, and recompute the baseline-vs-DoTA-RAG mean correctness delta under human ratings. If the human-rated delta is not positive and significant, or if inter-judge kappa is below 0.6, the headline improvement is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 3 reports the central internal improvement (0.752 to 1.478) using Claude 3.5 Sonnet as judge, so the reader's 'same-model self-judge' concern is only partially accurate. The load-bearing gap is that Section 3.4 validates neither judge at the item level: Table 2 is titled 'Judge agreement' but reports only mean scores (Claude correctness 1.382, Falcon3 1.430), with no correlation, kappa, or human agreement study, and Section 3.1's Morgana-MultiDocQA set is synthetic, unpublished, and unreleased. The fragility is visible inside Table 3: adding query rewriting (+Rewrite) lowers Claude-judged correctness from 1.652 to 1.478, yet the configuration was shipped anyway, and on the official Live Challenge Day the same configuration scored correctness 0.929 with faithfulness 0.043 under the 300-word cap. Section 4.2 attributes the collapse to the cap but gives no evidence that the official correctness score is insensitive to the same truncation. Without human labels or a per-item judge-agreement study, the baseline-vs-DoTA-RAG delta and the live corroboration are not established as valid measurements of answer quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"DoTA-RAG is a five-stage retrieval-augmented generation pipeline developed for the SIGIR 2025 LiveRAG Challenge over a 15M-document FineWeb-10BT index. The stages are query rewriting, dynamic namespace routing with self-consistency voting, hybrid retrieval (dense search with Arctic-embed-m-v2.0, BM25 pruning, Cohere Rerank 3.5), context aggregation with an 8k-token cap, and Falcon3-10B-Instruct answer generation. The authors construct a 500-question internal benchmark, Morgana-MultiDocQA, via DataMorgana and WebOrganizer taxonomies, and report an internal correctness improvement from 0.752 (baseline) to 1.478 (DoTA-RAG) with faithfulness improving from -0.496 to 0.640. On the Live Challenge Day they report correctness 0.929 and faithfulness 0.043, attributing the low faithfulness to the live 300-word output cap. The paper concludes that metadata-guided routing and hybrid retrieval deliver substantial gains in accuracy and latency.","tokens_in":9635,"tokens_out":4546,"duration_ms":50535,"significance":"If the reported measurements are valid, the paper provides a useful engineering demonstration that dynamic topic/format routing and multi-stage hybrid retrieval can reduce latency substantially (100.84s to 19.01s per query) while improving answer correctness on a web-scale corpus. The ablation table is transparent, the latency measurements are concrete, and the decision to re-embed FineWeb-10BT with a stronger model is clearly described. However, the significance is conditional on the evaluation being trustworthy: the central correctness gains rest entirely on unvalidated LLM judges, the final shipped configuration is internally worse than an ablated alternative, and the live faithfulness score is near zero. These issues must be resolved before the practical-deployment claim is supported.","major_comments":[{"comment":"The paper's central correctness delta (Baseline 0.752 to DoTA-RAG 1.478 in Table 3) and the live-corroboration claim rest on LLM-as-a-judge scores, yet Table 2, titled \"Judge agreement,\" reports only mean scores (Claude 3.5 Sonnet correctness 1.382, Falcon3-10B 1.430; faithfulness 0.520 vs 0.580). It contains no per-item correlation, Cohen's kappa, or human-agreement study, so the paper does not establish that either judge is a valid proxy for answer quality. I ask for per-item agreement statistics and a human-annotated subset (or a published judge-validation argument) before the 0.752-to-1.478 improvement can be accepted as a measurement of answer quality.","section":"Section 3.4, Table 2"},{"comment":"The shipped configuration, +Rewrite, is strictly worse internally than the +Rerank configuration: correctness drops from 1.652 to 1.478 and faithfulness from 0.672 to 0.640. Section 2.1 justifies this choice by anecdotal live-day failure cases involving misspellings, but no data are presented showing that rewriting improves performance on those queries or on the official live set. The paper's headline improvement is therefore achieved by a configuration that is not the best on the internal benchmark; the authors need to demonstrate that the rewriting stage helps on the target distribution, or report the +Rerank configuration as the primary system.","section":"Section 4.1, Table 3"},{"comment":"The Live Challenge Day correctness of 0.929 is presented as corroboration, but the faithfulness collapse from 0.702 to 0.336 under the 300-word cap (re-evaluated with Claude) shows the cap can materially change judged answers. No analogous re-evaluation of correctness under the cap is reported, so the 0.929 correctness score may also be an artifact of truncation rather than a measure of the same answers that scored 1.478 internally. The authors should re-judge the capped and uncapped live answers for correctness, or otherwise show that correctness is insensitive to the cap.","section":"Section 4.2"},{"comment":"The internal benchmark is generated with DataMorgana and auto-tagged with the same WebOrganizer topic/format taxonomy used to define the routing namespaces, and all component selections are made on this benchmark. This creates a risk of overfitting to synthetic data and to the taxonomy: routing to the top two of 24 self-invented namespaces may generalize poorly to the live corpus. The benchmark is also unpublished and unreleased. To support generalization claims, the authors should provide (i) the benchmark or a public validation set, and (ii) an evaluation on an independently labeled sample of FineWeb-10BT, not just the official leaderboard number paired with a failed faithfulness score.","section":"Section 3.1 and Appendix C"},{"comment":"All internal results are single-run mean scores on 500 questions with no standard errors, significance tests, or multiple-seed variation. The drop caused by +Rewrite (0.174 correctness) and the gains from +Arctic-M (0.864) could in part reflect evaluation noise; the paper should report variance or at least a significance test for the key comparisons.","section":"Table 3"}],"minor_comments":[{"comment":"The query-rewriting prompt says to leave intentional typos unchanged, but the motivation is to correct misspellings; please clarify how the model distinguishes intentional from unintentional typos and provide examples of rewritten queries.","section":"Section 2.1, Appendix A"},{"comment":"References [5] and [6] are the same paper (Gao et al., \"Precise Zero-Shot Dense Retrieval without Relevance Labels\") listed twice with different venues; deduplicate and cite the published ACL version.","section":"References"},{"comment":"The abstract omits the Live Challenge Day faithfulness score of 0.043, which is important context for the headline correctness figure of 0.929; please report both scores together.","section":"Abstract and Section 4.2"},{"comment":"The baseline row has no runtime, and the footnote explains that the baseline uses a pre-built index; the caption should state explicitly that the runtime comparison is not end-to-end for the baseline.","section":"Table 3"},{"comment":"The claim that Arctic v2.0 embedding models are state-of-the-art is based on the MTEB leaderboard at time of writing; cite the leaderboard version/date and report the retrieval-task subset rather than the mean over all tasks.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"This is a competition-system paper; the novelty is mostly engineering integration rather than a new method. The main risk is evaluation validity rather than idea novelty. Even for a workshop venue, the missing benchmark release and the unvalidated judge are serious; for a journal-level assessment they are load-bearing. The authors should be encouraged to provide the missing validation evidence and to consider reporting the +Rerank configuration as the primary system."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, know this: it's a competent competition-system paper, not a new idea. The pipeline is an integration of known pieces, and the 'Dynamic of Thought Aggregation' label is marketing. The real contributions are a specific configuration and a 500-question synthetic benchmark taxonomy.\n\nThe good parts: the ablation table is genuinely transparent. They show each stage's incremental effect, and they admit that adding query rewriting lowers correctness (from 1.652 to 1.478) yet ship it anyway because the live queries have typos. That's a defensible engineering tradeoff, reported honestly. Routing cuts latency from ~100s to ~19s per question, which is plausible and useful. They also report a live-challenge result rather than only self-constructed numbers.\n\nThe soft spots are about measurement. Table 2 is titled 'Judge agreement' but shows only means: no per-item correlation, no kappa, no human labels. The main ablation uses Claude 3.5 Sonnet, so the reader's self-judge concern is partially overstated, but the deeper issue remains: neither judge is validated. The internal benchmark is generated with DataMorgana and tagged with the same WebOrganizer taxonomy that defines the routing namespaces; it's unreleased, so overfitting to it is unfalsifiable. There are no error bars or tests, so differences of 0.1 on 500 questions are within noise. The faithfulness collapse on the live day (0.043 official, and 0.702 to 0.336 on re-evaluation with Claude) is real, and they attribute it to the 300-word cap without showing that correctness is unaffected. So the headline 0.752 to 1.478 gain is plausible but not established as a valid measure of answer quality.\n\nWho it's for: people working on RAG systems at competition scale, and anyone building synthetic benchmarks. It deserves a serious referee for a systems or competition track, but the referee should demand the benchmark be released and the judge agreement be shown per item. With those, the paper would be a decent reference point.","headline":"Assembles known RAG pieces into a fast competition pipeline and ships a new synthetic benchmark, but the evaluation leans on unvalidated judges and an unreleased test set.","tokens_in":10255,"tokens_out":3059,"would_cite":false,"duration_ms":35214,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A routing-aware RAG pipeline nearly doubles answer correctness on a 15-million-document web index.","keywords":["Retrieval-Augmented Generation","dynamic routing","hybrid retrieval","RAG benchmark synthesis","LiveRAG challenge","FineWeb-10BT","LLM-as-a-judge","embedding model selection"],"falsifier":"Take a random sample of about 100 answers from the 500-question internal set, have human annotators score them on the same -1 to 2 correctness rubric, and compare against Falcon3-10B-Instruct's scores: if the human-model agreement is weak or Falcon3 systematically inflates, the headline improvement fails. A cheaper check is to rerun the baseline and DoTA-RAG on the official Live Challenge Day with the live judge and see whether the internal gap reproduces.","tokens_in":9179,"feed_emoji":"⚡","tokens_out":8565,"duration_ms":80963,"temperature":0.7,"pith_summary":"DoTA-RAG claims that a RAG system can serve a 15-million-document web corpus both fast and accurately if it steers each query to a small sub-index before retrieving. The paper reports correctness climbing from 0.752 (baseline with E5-base-v2 embeddings) to 1.478 for the full pipeline on its 500-question benchmark, with end-to-end latency around 35 seconds per question, and 0.929 correctness on the official challenge day. The two load-bearing ideas are dynamic namespace routing, which uses self-consistency voting to pick the right topical sub-index and cuts dense-retrieval latency from about 101 to 19 seconds, and hybrid retrieval, where dense search, BM25 pruning, and a cross-encoder reranker lift faithfulness from -0.108 to 0.428. The claim matters because live web-scale RAG typically trades accuracy against latency; the paper argues the tradeoff can be resolved by metadata-guided routing plus staged filtering. All scores come from LLM judges, and the paper notes Falcon3-10B-Instruct (the answer generator itself) scores higher than Claude 3.5 Sonnet with no human agreement study.","feed_headline":"Routing shrinks RAG search 92% and nearly doubles answer correctness","feed_subtitle":"Dynamic topic routing plus hybrid retrieval keep 15M-document answers fast, grounded, and correct.","key_machinery":"The central mechanism is dynamic namespace routing combined with hybrid retrieval. Every document in the 15M-document FineWeb-10BT index is tagged by WebOrganizer's TopicClassifier and FormatClassifier into one of 24 topic namespaces; for each query, Falcon3-10B-Instruct produces four independent namespace classifications and self-consistency voting selects the top two namespaces, shrinking the search space by 92 percent. Retrieval then fans out with Arctic-embed-m-v2.0 dense search (k=100), prunes with BM25 to 20 passages, and selects the top 10 with Cohere's Rerank 3.5 cross-encoder. This staged design is what does the work: routing buys latency, and the dense-to-sparse-to-rerank cascade buys grounding.","core_discovery":"On its own terms, the paper's claim is that the DoTA-RAG pipeline — query rewriting for noisy input, ensemble routing into topical namespaces, dense-then-sparse-then-cross-encoder retrieval, and answer generation by Falcon3-10B-Instruct — reconciles web-scale knowledge integration with precision and low latency. The ablation path shows the contribution of each stage: swapping in Arctic-embed-m-v2.0 embeddings raises correctness from 0.752 to 1.616 and Recall@10 from 0.469 to 0.518; routing preserves that accuracy while cutting retrieval latency by 81 percent; BM25 pruning plus Cohere Rerank 3.5 raises faithfulness from -0.108 to 0.672. Query rewriting is the one stage that lowers internal scores (1.652 to 1.478), kept because the live challenge queries contained typos and specialized terms that broke retrieval without it. The paper also constructs MorganaMultiDocQA, a 500-question benchmark stratified across 24 topics times 24 document formats with eight question types, including temporal-evolution and verification questions that require synthesizing or adjudicating across documents.","pith_inferences":["The self-consistency routing idea transfers: any corpus with a cheap metadata taxonomy could use the same vote-then-top-k namespace selection, so the latency win is not specific to WebOrganizer.","MorganaMultiDocQA's question taxonomy (temporal-evolution, verification, multi-aspect) tests retrieval-and-synthesis behaviors that single-document QA benchmarks miss, so it could serve as a reusable stress test for other RAG pipelines, though its 500 questions are synthetic.","The judge-inflation gap between Falcon3 and Claude 3.5 Sonnet suggests that pipelines tuned against one LLM judge may overfit to that judge's preferences; a human-annotated subset of LiveRAG answers would settle which configuration actually answers best.","Because faithfulness collapsed under the output cap while correctness stayed high, grounding metrics may need to separate 'the answer is right' from 'the answer is fully supported within a verbosity budget'."],"forward_implications":["Metadata-guided routing delivers large latency gains without sacrificing accuracy: shrinking the search space by 92 percent drops dense-retrieval latency from 100.84 to 19.01 seconds per question.","Hybrid retrieval materially improves factual grounding: adding BM25 pruning and cross-encoder reranking raises faithfulness from -0.108 to 0.428.","Embedding model choice is the single largest correctness lever in the pipeline: Arctic-embed-m-v2.0 beats E5-base-v2 by 0.86 correctness points and lifts Recall@10 from 0.469 to 0.518.","The 300-word output cap imposed by the live evaluation sharply depresses faithfulness scores (0.702 to 0.336 under Claude 3.5 Sonnet), so reported faithfulness depends heavily on evaluation truncation policy.","Query rewriting for typos and misspellings is harmful on clean benchmarks, but the paper argues it is necessary for realistic web queries."],"supporting_citations":[{"why":"Supplies the replacement dense retriever whose swap drives the largest correctness gain in the ablation.","marker":"[18]"},{"why":"Supplies the topic and format classifiers that tag the corpus into the namespaces routing selects.","marker":"[16]"},{"why":"Provides the self-consistency voting method used to aggregate the four routing classifications.","marker":"[15]"},{"why":"The cross-encoder reranker that selects the final top-10 passages after BM25 pruning.","marker":"[2]"},{"why":"The fixed generation LLM that also performs routing classification and serves as the internal judge.","marker":"[13]"},{"why":"Generates the seed Q&A pairs from which the 500-question MorganaMultiDocQA benchmark is stratified.","marker":"[4]"},{"why":"Defines the 15M-document FineWeb-10BT corpus the system re-embeds and indexes.","marker":"[12]"},{"why":"The second LLM judge used to cross-check correctness and faithfulness scores.","marker":"[1]"},{"why":"The E5-base-v2 embedding model of the baseline that DoTA-RAG improves upon.","marker":"[14]"}],"fun_headline_variants":["Dynamic routing cuts RAG latency 81% and nearly doubles accuracy","DoTA-RAG: hybrid retrieval nearly doubles correctness, slashes latency","Fast, accurate RAG at web scale: routing reduces search 81%","From 0.752 to 1.478: DoTA-RAG doubles answer correctness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The internal results rest on an LLM judge — usually Falcon3-10B-Instruct, the same model that writes the answers — scoring correctness and faithfulness with no human-annotation check, and Falcon3 scores notably higher than Claude 3.5 Sonnet; if those automated scores do not track true answer quality, the claimed improvement from 0.752 to 1.478 is not established.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic routing cuts RAG latency 81% and nearly doubles accuracy","DoTA-RAG: hybrid retrieval nearly doubles correctness, slashes latency","Fast, accurate RAG at web scale: routing reduces search 81%","From 0.752 to 1.478: DoTA-RAG doubles answer correctness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001016,"raw_usage":{"total_tokens":4303,"prompt_tokens":974,"completion_tokens":3329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":3246}},"tokens_in":590,"tokens_out":3329,"duration_ms":27610,"temperature":1.0,"reasoning_tokens":3246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:45:47.288010+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 100 answers from the 500-question internal set, have human annotators score them on the same -1 to 2 correctness rubric, and compare against Falcon3-10B-Instruct's scores: if the human-model agreement is weak or Falcon3 systematically inflates, the headline improvement fails. A cheaper check is to rerun the baseline and DoTA-RAG on the official Live Challenge Day with the live judge and see whether the internal gap reproduces.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The cross-encoder reranker that selects the final top-10 passages after BM25 pruning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The second LLM judge used to cross-check correctness and faithfulness scores."}],"review_version":1}