{"id":"68388ecb-2368-4980-98a3-b3bc5ab64eeb","arxiv_id":"2506.21384","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Omni-RAG, a query-rewriting and decomposition pipeline on top of standard retrieval and reranking, achieved rank 2 in the SIGIR 2025 LiveRAG Challenge.","lead":"This paper describes Omni-RAG, a system that cleans and breaks apart messy user questions before searching a large web corpus to answer them. It reports a second-place finish in the SIGIR 2025 LiveRAG Challenge and studies how changing the number of retrieved passages affects answer quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reranker scores against the original query, not sub-queries; decomposition may be nullified before generation, leaving the central mechanism internally inconsistent.","rationale":"The reader's weakest assumption correctly identifies the absence of an ablation for the query-understanding module. My concern is more specific: even if an ablation were run, the described reranking step in §3.4 would likely mask any benefit of decomposition, because it scores all documents against a single query rather than per sub-query. This makes the paper's causal story internally inconsistent, not merely unverified. I keep the verdict at CONDITIONAL because the externally measured rank-2 result remains credible as a system-level outcome, but the explanation of why the system works is not supported by the paper's own pipeline description and evaluation. The concrete test would settle whether the concern lands: if variant (B) outperforms variant (A), the published pipeline is suboptimal as described; if not, the decomposition module is superfluous under the chosen reranking strategy. Either outcome would require the authors to revise their mechanistic claim, but neither would overturn the rank-2 result itself. Therefore the reader's CONDITIONAL verdict is appropriate and unchanged.","tokens_in":12847,"tokens_out":5340,"duration_ms":56746,"concrete_test":"On the dry-test multi-intent subset, run two variants: (A) as described — retrieve per sub-query, rerank all documents against the original/rewritten query, take top-N; (B) retrieve per sub-query, rerank each sub-query's documents against that sub-query, then union and take top-N. Score both with a fixed, independent judge (e.g., the official challenge judge or a held-out LLM) on relevance and faithfulness. If (B) outperforms (A), the published pipeline's reranking step undermines its decomposition mechanism. If (A) ≈ (B), decomposition provides no measurable benefit under the implemented reranking, and the central contribution is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's claimed contribution is LLM-assisted query understanding (rewriting and decomposition) in §§3.2–3.3. Retrieval is performed per sub-query, producing D_retrieved = ∪ D_s. However, in §3.4 the BGE reranker computes score(q, d) using 'the original query q (or the rewritten query q′)' — not each sub-query q′_s. All documents from all sub-queries are then ranked by a single monolithic query, and the top-N are selected. For a multi-intent query such as 'compare X and Y', documents that are highly relevant to the minority sub-intent may receive low scores against the full query and be pruned, negating the decomposition step. This is not just a missing ablation; it is an internal inconsistency in the described pipeline. If the reranker uses the original/rewritten query, the decomposition's effect is filtered out before generation, so the rank-2 result cannot be attributed to the query-understanding module as described. No experiment isolates this interaction, and the dry-test pseudo-labeling protocol (§3.5 vs §3.6) is itself inconsistent about which model generates scores (Qwen2.5-7B/Falcon-10B vs Qwen2.5-72B), further weakening the internal evidence for the mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Omni-RAG, a retrieval-augmented generation pipeline submitted to the SIGIR 2025 LiveRAG Challenge. The pipeline consists of LLM-based query rewriting and decomposition, per-sub-query retrieval from FineWeb using OpenSearch, BGE reranking, and Falcon-10B generation with chain-of-thought prompting. The authors report that their team (RUC_DeepSearch) achieved rank 2 in Session 1 of the challenge, and they supplement this with dry-test pseudo-labeling experiments using 50 samples. The central claimed contribution is that LLM-assisted query understanding (denoising and decomposition) improves retrieval and final answer quality in live, noisy, multi-intent settings.","tokens_in":13073,"tokens_out":3776,"duration_ms":43225,"significance":"The externally judged rank-2 result on the LiveRAG leaderboard is a genuine and objective piece of evidence that the overall pipeline is competitive; this is the paper's main strength. The paper also addresses a practically important problem: handling noisy and multi-intent queries in live RAG. However, the manuscript does not currently establish that query understanding is the cause of the system's success, and the described architecture contains an internal tension between per-sub-query retrieval and monolithic reranking. If the authors add a proper ablation and resolve the reranking inconsistency, the rank-2 result could support a useful systems paper, but as written the evidence is insufficient to credit the claimed mechanism.","major_comments":[{"comment":"The reranker is described as computing score(q, d) against 'the original query q (or the rewritten query q')' rather than against each sub-query q'_s. Since all documents from all sub-queries are then ranked by a single monolithic query and only the top-N are kept, documents that are highly relevant to a minority sub-intent can receive low scores and be pruned, effectively undoing the decomposition step before generation. This is not merely a missing ablation; it is an internal inconsistency in the described pipeline, because the claimed benefit of intent-aware decomposition cannot be realized as specified. The authors must specify which query representation is actually used for reranking, or modify the architecture so that reranking respects sub-intents (e.g., per-sub-query reranking followed by a merge).","section":"§3.4, Eq. (5)"},{"comment":"No experiment isolates the effect of the query understanding module. Tables 1 and 2 compare different top-k settings and self-consistency path counts, but there is no condition that runs OpenSearch, BGE, and Falcon-10B with and without rewriting/decomposition (Eqs. 2-3). The rank-2 leaderboard result therefore cannot be attributed to LLM-assisted query understanding; it may be entirely due to the retrieval, reranking, and generation configuration. The authors should add an ablation that removes or bypasses the rewriting/decomposition module, and report both dry-test pseudo-label scores and any available leaderboard-based comparison.","section":"§3.2, §3.6, Table 1"},{"comment":"The pseudo-labeling protocol is described inconsistently. Section 3.5 says Qwen2.5-7B-Instruct generates reference answers and Falcon-10B performs relevance and faithfulness evaluations, but Table 2 states that the in-house metrics are generated by Qwen2.5-72B-Instruct. It is unclear which model produced the reference answers, which model scored them, and whether the dry-test analysis in §3.6 uses a different protocol from §3.5. This discrepancy must be resolved before the reported numbers can be interpreted.","section":"§3.5 vs. §3.6, Table 2"},{"comment":"The dry-test evaluation is based on only 50 samples and uses a circular reference: Qwen2.5-7B generates pseudo-answers from the same top-k documents that the system retrieves, and another LLM then scores the system's answers against those pseudo-answers. This does not provide independent evidence of factual correctness, and the paper does not report confidence intervals or statistical tests for the claim that performance 'scales with document count.' The authors should either use official ground truth where available, add human spot-checking, or at minimum report bootstrap intervals and a significance test.","section":"§3.5, 'Dry Test Analysis'"}],"minor_comments":[{"comment":"The introduction cites [32] for Falcon3-10B-Instruct, but [32] is the RefinedWeb dataset paper; the model citation appears incorrect.","section":"§1, reference [32]"},{"comment":"The framework is called 'Omini-RAG' in the overview paragraph, while the rest of the paper uses 'Omni-RAG'; please standardize the spelling.","section":"§3, paragraph 1"},{"comment":"The text refers to 'our RobustRAG framework' in the paragraph following Table 1; this should be 'Omni-RAG'.","section":"§3.6, main result paragraph"},{"comment":"The table columns are not self-explanatory: it is unclear whether the numbers after top-k are counts of samples in each score category and whether the first column under 'Avg' is the total score or another aggregate. Please add a full caption and define every column.","section":"Table 2"},{"comment":"The notation θ_rewrite and θ_decompose suggests fine-tuned parameters, but the method section describes prompt-based rewriting and decomposition; if no fine-tuning is performed, the notation should be changed to avoid implying trained components.","section":"Eqs. (2)-(3)"},{"comment":"The introduction states that the top-10 documents are selected, while Eq. (5) uses a generic top-N; please reconcile the notation and state the actual value used in the LiveRAG submission.","section":"§3.4, Eq. (5)"},{"comment":"The self-consistency strategy is attributed to [42], but [42] is the Self-Instruct paper; the appropriate citation is Wang et al. 2022, 'Self-Consistency Improves Chain of Thought Reasoning in Language Models.'","section":"Reference [42]"}],"recommendation":"major_revision","confidential_remarks":"The rank-2 leaderboard result is the most valuable asset of this paper, but the current manuscript does not support its central attribution claim, and the reranking-description inconsistency is a substantive architectural issue rather than a typo. I would encourage the editor to invite a revision with a proper ablation and a clarified reranking protocol, rather than rejecting outright, because the external evaluation result gives the paper a solid empirical anchor if the mechanism is made credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result is real: Omni-RAG placed second in SIGIR LiveRAG Session 1, and that rank is externally measured. What's actually new is the particular assembly—LLM rewriting plus decomposition into sub-queries, per-sub-query retrieval over FineWeb, BGE reranking, and Falcon-10B generation with self-consistency. As a system recipe for a live RAG competition, it's plausible and clearly described, and the leaderboard position is a useful data point for practitioners.\n\nThat said, the paper's central claim—that the query-understanding module is what drives the performance—is not supported. There is no ablation that runs the pipeline with and without rewriting/decomposition, so the rank could come from any of the assembled components. The internal dry-test evaluation is thin: 50 samples, and the pseudo-labeling description is internally inconsistent (§3.5 says Qwen2.5-7B generates reference answers and Falcon-10B scores them; Table 2 says Qwen2.5-72B generated the scores). Those numbers are the only evidence for the mechanism, so the inconsistency matters.\n\nMore concerning is the reranking detail in §3.4. The BGE reranker scores documents against the original or rewritten query, not against the sub-queries produced. If that's accurate, documents retrieved for minority sub-intents are re-ranked under the full query and can be pruned before generation, which would undo the decomposition step. That is not a missing ablation; it's a potential internal inconsistency in the described pipeline. The authors should either clarify that sub-queries are used at reranking time or show that the decomposition's effect survives this step.\n\nWhat holds up: the leaderboard result itself, and the paper's value as a system description. The related work is fine, and the writing is straightforward. No code or prompts are released, which limits reproducibility, but that's typical for challenge reports.\n\nThis paper deserves a serious referee. It's not a desk reject—the rank-2 result is externally meaningful. But the referee should push for an ablation or a softened claim, and a clean-up of the evaluation description. For a reading group, it's a quick, instructive look at what actually works in a live RAG competition.","headline":"A live RAG competition system with a credible rank-2 result and a clear recipe, but the paper's causal claim about query understanding is not supported by its internal evidence.","tokens_in":13628,"tokens_out":4046,"would_cite":false,"duration_ms":41086,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that rewriting and decomposing noisy user queries with an LLM before retrieval makes live retrieval-augmented generation reliable, and reports its Omni-RAG system placed second among twelve teams in the SIGIR 2025…","keywords":["Retrieval-Augmented Generation","Query Understanding","Query Decomposition","Query Denoising","LiveRAG Challenge","Document Ranking","OpenSearch"],"falsifier":"Run the identical OpenSearch, BGE, and Falcon-10B stack on the same LiveRAG queries with the rewriting/decomposition step disabled; if correctness and faithfulness stay at the same level, the query-understanding module is not the source of the system's performance.","tokens_in":12610,"feed_emoji":"🔍","tokens_out":7749,"duration_ms":74553,"temperature":0.7,"pith_summary":"Omni-RAG is a retrieval-augmented generation framework built for live, open-domain settings where user queries contain typos, ambiguous phrasing, and several intents at once. The paper's central claim is that cleaning and splitting such queries with an LLM before retrieval is what lets a RAG system answer them well. The full pipeline rewrites and decomposes the query, retrieves per sub-query over the FineWeb corpus with OpenSearch, reranks with BGE, and lets Falcon-10B compose the final answer. The reported evidence is a second-place overall finish among twelve teams in Session 1 of the SIGIR 2025 LiveRAG Challenge.","feed_headline":"Rank-2 live RAG pipeline puts LLM query rewriting first","feed_subtitle":"Splitting noisy questions into clean sub-queries before retrieval is the move that put this system above nine of ten rivals.","key_machinery":"The load-bearing component is the Deep Query Understanding and Decomposition module, in which an LLM rewrites the query $q$ into a clean form $q'$ and then splits it into $M$ sub-queries $\\{q'_s\\}_{s=1}^M$ using tailored prompts. This module converts one ambiguous query into several specific retrieval targets, which is what allows the subsequent OpenSearch retrieval, BGE reranking, and Falcon-10B chain-of-thought generation to cover all intents. The paper formalizes the two steps as $q' = \\mathrm{Rewrite}(q, \\theta_{\\mathrm{rewrite}})$ and $\\{q'_s\\}_{s=1}^M = \\mathrm{Decompose}(q', \\theta_{\\mathrm{decompose}})$.","core_discovery":"The paper introduces Omni-RAG, a three-module pipeline whose distinguishing step is LLM-assisted query understanding. Instead of sending the raw user query to the retriever, an LLM rewrites it into a cleaner form and decomposes it into a set of sub-queries, each targeting one intent. Retrieval then runs separately for every sub-query, and the union of results is reranked and trimmed before generation. The paper's claim is that this preprocessing is the reason the system handles the noisy, multi-intent queries generated by the LiveRAG simulator, and that the pipeline demonstrates a practical route to live RAG. The reported outcome is a second-place score in Session 1 of the challenge, with higher correctness and faithfulness than most competing systems.","pith_inferences":["The paper never compares the full pipeline against the same pipeline with the rewriting/decomposition module removed, so the rank-2 result may be driven by other components rather than by query understanding; this is the key open test for the paper's causal claim.","Decomposing a query into independent sub-queries suggests a natural extension: chain the sub-queries so that evidence retrieved for one sub-query informs the reformulation of the next, turning the decomposition into an iterative search plan.","The pseudo-labeling evaluation with Qwen models could be validated by comparing its scores against human ratings on a small labeled sample; if it tracks human judgment, it becomes a reusable way to evaluate live RAG systems that have no ground-truth answers."],"forward_implications":["If the query-understanding module is the cause of the rank-2 result, then any live RAG system that faces noisy or multi-intent queries should insert an LLM-based rewriting and decomposition step before retrieval.","The system shows that a fixed set of standard components, such as OpenSearch, a BGE reranker, and a sub-10B-parameter generator, can be competitive in a live RAG setting when the query is preprocessed well.","The dry-test analysis suggests that performance keeps improving as the number of retrieved documents per sub-query grows, supporting the union-of-sub-queries retrieval design over a single-shot search.","The self-consistency experiments indicate that more sampling paths does not always help: four paths improved relevance, while eight did not, so the path count needs to be tuned rather than raised blindly."],"supporting_citations":[{"why":"Provides the retrieval-augmented generation paradigm that Omni-RAG builds on.","marker":"[13]"},{"why":"Shows prior work on LLM-based query refinement for RAG, which the rewriting module extends.","marker":"[6]"},{"why":"Supplies the FineWeb corpus used as the fixed knowledge source for retrieval.","marker":"[31]"},{"why":"Describes the Falcon model family, from which the Falcon3-10B generator comes.","marker":"[32]"},{"why":"The DataMorgana simulator that generates the noisy, multi-intent queries in the challenge.","marker":"[11]"},{"why":"Source of the Qwen models used as the pseudo-labeling reference and evaluator.","marker":"[4]"},{"why":"The cited source for the self-consistency sampling strategy the paper experiments with.","marker":"[42]"}],"fun_headline_variants":["LLM breaks noisy queries into clean sub-queries for better RAG","Omni-RAG reaches rank 2 by rewriting queries first","Live RAG pipeline cleans up queries before retrieval","For noisy queries, Omni-RAG splits before it retrieves","Query rewriting before retrieval lifts live RAG to rank 2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's advantage is credited to LLM-based rewriting and decomposition, but the paper does not ablate that module, so nothing in the reported experiments shows it causes the rank-2 scores.","fun_headline_variants_meta":{"raw":{"variants":["LLM breaks noisy queries into clean sub-queries for better RAG","Omni-RAG reaches rank 2 by rewriting queries first","Live RAG pipeline cleans up queries before retrieval","For noisy queries, Omni-RAG splits before it retrieves","Query rewriting before retrieval lifts live RAG to rank 2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":1949,"prompt_tokens":966,"completion_tokens":983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":897}},"tokens_in":582,"tokens_out":983,"duration_ms":9952,"temperature":1.0,"reasoning_tokens":897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:25:47.362164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical OpenSearch, BGE, and Falcon-10B stack on the same LiveRAG queries with the rewriting/decomposition step disabled; if correctness and faithfulness stay at the same level, the query-understanding module is not the source of the system's performance.","supporting_citations":[{"cited_title":"Conversational Query Reformulation with the Guidance of Retrieved Documents","cited_arxiv_id":"2407.12363","evidence_quote":"Describes the Falcon model family, from which the Falcon3-10B generator comes."}],"review_version":1}