{"id":"df59d63d-835f-490b-8cb1-5f5ca224508d","arxiv_id":"2506.17493","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PreQRAG classifies questions as single- or multi-document, then rewrites single-document queries and decomposes multi-document queries, improving retrieval and earning preliminary second place in a LiveRAG challenge session.","lead":"This paper describes PreQRAG, a question-answering system that first sorts each question into single-document or multi-document, then rewrites or splits the question so retrieval works better. The system took preliminary second place in Session 2 of the SIGIR 2025 LiveRAG challenge.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No end-to-end ablation links the preprocessing module to the LiveRAG result; the internal retrieval gains come from 216 synthetic questions, so the causal claim in the abstract is not yet established.","rationale":"The reader's weakest assumption is that the DataMorgana synthetic questions, generated with the categorization schema from [2], are representative of the 500 LiveRAG questions. That is a real risk, but I see a more direct hole: even on its own synthetic ground, the paper never runs an end-to-end comparison with and without the preprocessing module. The retrieval tables are informative about the retrieval stage, but the abstract's 'improves the RAG performance' is an end-to-end claim, and the only end-to-end data point is a single unablated challenge run. This is the load-bearing weakness because both major evidential pillars—internal retrieval tables and external challenge rank—leave the causal attribution to preprocessing unexamined. The synthetic-representativeness problem matters mostly because it would make even a good retrieval ablation non-transferable; an end-to-end ablation on the actual challenge questions would address both at once. I would not change the verdict to reject: the system demonstrably works well enough to place second in a live competition, and the missing ablation is an addressable experimental gap rather than a contradiction. I keep CONDITIONAL, with the condition being that the authors provide a within-pipeline control or explicitly soften the abstract's causal claim to a retrieval-level claim. The judge-dependence of the internal metrics and the TOP-3 choice are secondary but reinforce the need for a direct comparison on official metrics.","tokens_in":9583,"tokens_out":7801,"duration_ms":72721,"concrete_test":"Run the complete PreQRAG pipeline and an identical control pipeline with only the classify-and-rewrite stage bypassed (original queries go straight to the same hybrid retrieval, bge reranker, Prompt B, TOP-3, greedy Falcon generation) on the 500 LiveRAG questions, scoring with the official correctness/faithfulness metrics or, if gold judgments are unavailable, with the paper's GPT-4o Mini evaluation prompts. Repeat for the synthetic multi-doc subset as well. If the with-vs-without preprocessing delta is zero or negative, or within sampling noise, the abstract's claim is unsupported; if the delta is positive and consistent across both sets, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is causal: classifying and rewriting questions improves RAG performance. The only direct test of that causal path is missing. Tables A.1, A.2, 2, and 3 show retrieval-level improvements, but they are measured on 110 single-document and 106 multi-document DataMorgana-generated QA pairs, scored by overlap with the source documents used to generate each synthetic question. They are not measured on the 500 LiveRAG questions, and they do not measure correctness or faithfulness of the final answer. The LiveRAG result in Table 5 is a single preliminary run of the complete pipeline with no control arm in which the classify-and-rewrite module is disabled. A second-place score could be driven by the hybrid sparse+dense retrieval, the bge reranker, the role-based prompt, greedy decoding, or the TOP-3 context choice. Two secondary observations reinforce this: Table 4 does not support TOP-3 for single-document questions (equal-weighted averages: Top1 0.851, Top3 0.839), and Appendix C.2 shows multi-document faithfulness at Top3 is 0.820 with GPT-4o Mini but 0.384 with Claude, so the internal answer-quality scores that drive design decisions are judge-dependent. The challenge rank is encouraging, but it cannot by itself establish that preprocessing caused the gain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PreQRAG, the UDInfo team's submission to the SIGIR 2025 LiveRAG Challenge. The system adds a question-preprocessing stage to a standard RAG pipeline: a rule-based classifier labels each question as single-document or multi-document; single-document questions are rewritten with two LLM prompts optimized for sparse and dense retrieval; multi-document questions are decomposed into two sub-questions. Retrieval is hybrid (BM25 plus E5 dense), followed by bge-reranker-v2 reranking and generation with Falcon3-10B-Instruct using a role-based prompt, a fixed TOP-3 context, and greedy decoding. The paper reports a preliminary second place in Session 2 of the LiveRAG Challenge (Table 5). Internal evaluations on DataMorgana-generated synthetic datasets show retrieval-level gains from rewriting (Table A.1), decomposition (Table A.2), and reranking (Table 3), together with generation-stage comparisons over prompts, decoding parameters, and context length.","tokens_in":9841,"tokens_out":3828,"duration_ms":38767,"significance":"If the central causal claim is established, the contribution is a modest, practical result: a lightweight preprocessing step that improves retrieval and answer quality in a constrained RAG setting. The paper has real strengths: the official LiveRAG ranking is independent external evidence; the retrieval-stage ablations are clearly separated by component; and the system description is sufficiently detailed to be reproduced. At the same time, the causal statement that classification and rewriting improve RAG performance is not yet fully supported, because the internal evaluations use synthetic questions generated under the same schema used for design and because the LiveRAG result is a single uncontrolled run. The modular retrieval gains are credible, but their connection to final answer quality on the official 500 questions needs an additional control or a carefully scoped claim.","major_comments":[{"comment":"The abstract's claim that \"This classification and rewriting strategy improves the RAG performance\" is supported only by retrieval-level experiments on 216 synthetic questions and by a single preliminary LiveRAG run with no control arm. The second-place score in Table 5 could be driven by the hybrid retrieval, the reranker, the role-based prompt, the TOP-3 context choice, or greedy decoding, instead of by question preprocessing. Please add an end-to-end ablation on the LiveRAG questions that compares the full PreQRAG pipeline with the classify-and-rewrite module disabled, or explicitly restrict the causal claim to retrieval-stage improvements.","section":"§4, Table 5"},{"comment":"There is a major numerical inconsistency between the two retrieval-effectiveness tables for the same single-document rewritten sparse setting. Table A.1 reports MRR@1 = 0.033 for sparse rewritten queries, while Table 2 reports a 37.5% Top-1 retrieval rate for sparse rewritten single-document questions. These numbers cannot both describe the same configuration; they differ by an order of magnitude. Please clarify whether the datasets, the scoring definition (success at rank 1 versus reciprocal rank at rank 1), or the retrieval runs differ, because the rewriting-improvement argument rests on these measurements.","section":"Appendix A.1 vs §2.3, Table 2"},{"comment":"The design decisions in the paper are tuned on 110 single-document and 106 multi-document DataMorgana-generated QA pairs that use the same categorization schema as the challenge organizers. The paper does not test whether these synthetic questions are representative of the actual 500 LiveRAG questions. A concrete transfer check, such as retrieval recall with and without rewriting on the official 500 questions, would substantially strengthen the claim that the preprocessing gains transfer from the internal evaluation distribution to the LiveRAG benchmark.","section":"§3.2, Appendix D"},{"comment":"The choice of TOP-3 context and other generation-stage decisions rely on GPT-4o Mini judge scores, but Appendix C.2 shows strong judge sensitivity: multi-document faithfulness at TOP-3 is 0.820 with GPT-4o Mini and 0.384 with Claude 3.5 Sonnet. This does not invalidate the pipeline, but it means the internal answer-quality comparisons are judge-dependent. Please report the judge variance explicitly or at least qualify that configuration choices are contingent on the chosen evaluator.","section":"§2.5, Table 4 and Appendix C.2"}],"minor_comments":[{"comment":"The introduction refers to \"Falcon-3B-10B-Instruct,\" which appears to be a typo for \"Falcon3-10B-Instruct\" used elsewhere.","section":"§1"},{"comment":"The caption contains the misspelling \"highligted\"; it should be \"highlighted.\"","section":"Table 5 caption"},{"comment":"The text states \"The results in Table A.2 shows the effectiveness of query decomposition,\" but the comparison with the non-decomposed baseline appears only in the appendix. Consider moving at least one row of this comparison to the main body, since decomposition is a central component.","section":"§2.2.2"},{"comment":"The faithfulness evaluation prompt in Appendix C.1 uses scores 1, 0, and -1, while the reported faithfulness values in the tables are normalized between 0 and 1. Please state the normalization formula explicitly.","section":"§3.1"},{"comment":"The classification accuracy table does not report the number of questions used for each classifier. Given the small synthetic evaluation sets, adding the sample size and ideally a confidence interval would help interpret differences such as 0.731 versus 0.822 for single-document questions.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competition system description, and the bar for a central causal claim should be calibrated accordingly. The missing end-to-end control is the main issue; a carefully scoped claim about retrieval improvements would already address much of it. No concerns about citation practice; the DataMorgana schema and challenge-organizer references are appropriately credited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a workmanlike challenge submission, not a research breakthrough. The novel bit is the specific combination — rule-based single/multi classification, two rewriting strategies for sparse and dense retrieval, and two-subquestion decomposition for multi-doc queries — evaluated on the LiveRAG challenge and reported as preliminary second in Session 2. That's a real result, though preliminary.\n\nWhat I like: the paper actually measures things. The retrieval-level ablations are consistent and show real gains: rewriting improves MRR by ~13–14%, decomposition lifts Top-10 recall from 41% to 55% (sparse) and 36% to 56% (dense), and reranking pushes single-doc Top-10 to 85%. The design decisions are documented with experiments — generation method, decoding parameters, prompt strategies, context length. They also include a second judge (Claude) in the appendix, which is more than many papers do.\n\nWhere it's soft: the abstract's causal claim — 'this classification and rewriting strategy improves RAG performance' — is not directly tested. The internal numbers come from 216 synthetic QA pairs generated with DataMorgana using the same categorization schema that drove the system design, so those gains partly reflect tuning on the evaluation distribution. There is no end-to-end ablation on the LiveRAG questions with the preprocessing module disabled. The second-place score is a single preliminary run; it could be driven by hybrid retrieval, reranking, prompt design, or the TOP-3 context choice.\n\nOne specific issue: they say Top-1 is 'best overall' for single-doc questions in Table 4, then choose Top-3 anyway. Under equal weighting, Top-1 actually beats Top-3 (0.851 vs 0.839). The argument is robustness — better chance of hitting the gold document — which is reasonable, but it undercuts the claim that the choice follows from the numbers. Also, the faithfulness scores are judge-dependent: multi-doc Top-3 faithfulness is 0.820 with GPT-4o Mini but 0.384 with Claude. They report this, but it means the internal answer-quality metric that drives several decisions is unstable.\n\nThere's also no code or data released, which limits reproducibility — though that's typical for a challenge report.\n\nBottom line: the paper is honest and internally consistent, the retrieval evidence is solid, and the challenge rank gives it external grounding. The missing end-to-end ablation is the load-bearing gap. I'd send it to a referee who can ask for that ablation and a clearer treatment of the TOP-3 choice. It's a useful data point for anyone building RAG pipelines.","headline":"Competent challenge-system paper whose internal retrieval ablations are solid, but the central causal claim about preprocessing improving RAG is not tested end-to-end; still worth a referee's time.","tokens_in":10360,"tokens_out":2749,"would_cite":false,"duration_ms":25878,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A preprocessing layer that sorts questions by document type and rewrites them accordingly improves retrieval and answer quality in RAG systems.","keywords":["retrieval-augmented generation","question classification","query rewriting","query decomposition","hybrid retrieval","reranking","question-type-aware preprocessing"],"falsifier":"Run the full PreQRAG pipeline and a no-preprocessing baseline on the challenge's 500-question benchmark, scoring each answer with the same equivalence, relevance, and faithfulness rubric; if the rewritten and decomposed queries do not beat the original queries on the live set, the central claim is not supported.","tokens_in":9371,"feed_emoji":"🔎","tokens_out":6586,"duration_ms":62106,"temperature":0.7,"pith_summary":"The paper tries to establish that a cheap preprocessing step can make a retrieval-augmented generation system measurably better. The step classifies each incoming question as answerable from a single document or from multiple documents, then rewrites the question according to that type. Single-document questions get two retrieval-focused rewrites, one for sparse search and one for dense search; multi-document questions are decomposed into two sub-questions whose retrieved evidence is merged. The authors report that this question-type-aware strategy improves retrieval, reranking, and final answer quality, and that their system finished second in its session of the competition.","feed_headline":"Classifying questions lifts RAG retrieval and answer quality","feed_subtitle":"A preprocessing step that sorts questions by document count, then rewrites each type for sparse or dense search.","key_machinery":"The machinery is question-type-aware preprocessing: a rule-based classifier that sorts each question into single-document or multi-document, chosen because it identifies multi-document questions far more reliably than the LLM-based classifier. That label controls two rewriting branches — retrieval-optimized rewrites for single-document questions and decomposition into two sub-questions for multi-document ones — and the rewritten queries are what enter the hybrid sparse-and-dense retrieval stage. The classifier's asymmetric error tolerance, where extra context is acceptable but missing context is costly, is what makes the preprocessing scheme safe for downstream generation.","core_discovery":"PreQRAG's central claim is that classification and rewriting of the input question improves the whole RAG pipeline. On the paper's internal evaluations, rewriting raises Mean Reciprocal Rank for single-document questions, decomposing multi-document questions substantially raises the fraction of ground-truth documents retrieved, hybrid sparse-plus-dense retrieval followed by cross-encoder reranking produces the largest retrieval gains, and a top-three context with a role-based prompt gives the best generation results. The authors present these component-wise improvements and a second-place competition finish as evidence that question-type-aware preprocessing is an effective and inexpensive enhancement to RAG.","pith_inferences":["Beyond the paper, the same classify-then-rewrite layer could be tested on domains whose questions do not fit the challenge's category schema, and the failure pattern would indicate how much of the gain is schema-specific.","Beyond the paper, holding retrieval and generation fixed while varying only the preprocessing branch would separate the contribution of classification accuracy from the contribution of rewriting quality.","Beyond the paper, the asymmetry in misclassification costs suggests a design rule for other RAG systems: when in doubt, treat a question as multi-document, because excess context is cheaper to tolerate than missing context."],"forward_implications":["A cheap preprocessing layer can improve RAG without retraining the retriever, reranker, or generator.","Decomposing multi-document questions into two sub-questions roughly doubles the chance of retrieving a ground-truth document at Top-1 in the paper's internal evaluations.","Hybrid dense and sparse retrieval followed by cross-encoder reranking is where the largest retrieval gains appear, lifting single-document Top-1 retrieval to 64.2% in the paper's internal evaluation.","Context length choice matters: for both question types, top-three documents perform as well as or better than top-one while providing more safety when the top hit is wrong.","For a smaller instruction-tuned model, the design of prompts and context arrangement affects answer quality more than variations in the generation process."],"supporting_citations":[{"why":"Supplies the synthetic question-answer generation tool and the question-category schema used to build the tuning and evaluation datasets.","marker":"[2]"},{"why":"Fixes the corpus that the challenge and the system's retrieval stage search over.","marker":"[6]"},{"why":"Provides the generator model that the whole pipeline is built around and that produces the final answers.","marker":"[7]"},{"why":"Provides the dense embeddings used to build the dense retrieval index.","marker":"[8]"},{"why":"Defines the sentence-based chunking of documents into 512-token units used by both indices.","marker":"[5]"},{"why":"Supplies the cross-encoder reranker that reorders the hybrid retrieval results before generation.","marker":"[1]"}],"fun_headline_variants":["PreQRAG: Preprocess questions to boost RAG","Question preprocessing key to better RAG","Classify and rewrite questions for sharper RAG","Query preprocessing lifts RAG answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the synthetic questions used to tune the classifier, rewrites, and context length behave like the challenge's actual benchmark questions, so the internal gains transfer to the live setting.","fun_headline_variants_meta":{"raw":{"variants":["PreQRAG: Preprocess questions to boost RAG","Question preprocessing key to better RAG","Classify and rewrite questions for sharper RAG","Query preprocessing lifts RAG answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1252,"prompt_tokens":800,"completion_tokens":452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":396}},"tokens_in":416,"tokens_out":452,"duration_ms":4407,"temperature":1.0,"reasoning_tokens":396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:07:45.910253+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full PreQRAG pipeline and a no-preprocessing baseline on the challenge's 500-question benchmark, scoring each answer with the same equivalence, relevance, and faithfulness rubric; if the rewritten and decomposed queries do not beat the original queries on the live set, the central claim is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fixes the corpus that the challenge and the system's retrieval stage search over."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the generator model that the whole pipeline is built around and that produces the final answers."}],"review_version":2}