{"id":"67b70e3f-b426-484a-a7cd-fdb58daa726e","arxiv_id":"2412.15101","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-phase RAG pipeline with dynamic temporal query rewriting and adaptive retrieval improves multi-hop QA accuracy over several tested baselines on four benchmarks.","lead":"This paper proposes review-then-refine, a two-stage RAG framework that decomposes complex questions into temporally annotated sub-queries, decides adaptively when to retrieve external documents, and then aggregates the intermediate answers. The authors report accuracy gains over several baselines on dynamic and static multi-hop QA benchmarks such as FreshQA and 2WikiMultiHopQA.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FreshQA evaluation protocol is internally contradictory (377 vs. 600 questions, top-k 5 vs. 2–3, and a PAT-Questions metric mislabel), so the reported SOTA numbers are not yet verifiable.","rationale":"The paper's central claim is that the proposed framework achieves state-of-the-art multi-hop QA performance, especially on FreshQA. The load-bearing condition is therefore that the reported evaluation numbers are accurate, fairly compared, and reproducible. The manuscript contains multiple concrete inconsistencies in exactly these numbers: the FreshQA dataset size is stated as 377 and then as 600; the PAT-Questions prose in Section 4.4 labels single-hop accuracy as multi-hop accuracy; and the retrieval top-k is inconsistent between Section 4.1's Table 2 and Appendix A. These are not stylistic issues; they directly affect what the headline results mean. If the evaluation set or metric labels are wrong, the claimed superiority over baselines could be an artifact. I do not select the retrieval-indicator calibration as the primary concern because, even if the LLM's self-assessment of retrieval need were miscalibrated, the framework could still achieve high accuracy; the paper's own ablations show retrieval generally helps, but they do not isolate the adaptive decision. The framework itself is coherent, the ablations show the major components contribute, and the case studies illustrate the intended behavior. Those are points in its favor. However, without code or data and with the documented evaluation inconsistencies, the reader's CONDITIONAL verdict is appropriate. My concern reinforces that verdict rather than changing it; if the requested check fails, the appropriate outcome would move toward REJECT.","tokens_in":16584,"tokens_out":5865,"duration_ms":54241,"concrete_test":"Obtain the authors' evaluation script and exact data splits, then rerun review-then-refine and all Table 3 baselines on the official FreshQA 600-question set using the official FreshQA answer-evaluation script and identical retrieval corpora and top-k settings. If the 70.68% single-hop and 60.60% multi-hop accuracies cannot be reproduced, or if the 600-question set and the single-hop/multi-hop labels differ from the paper's description, the SOTA claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: review-then-refine surpasses all baselines on FreshQA (Section 4.4). That claim depends entirely on the validity and reproducibility of Table 3. Several concrete inconsistencies undermine this. Section 4.1 says 'FreshQA consists of 377 questions' but then says 'We evaluated the dataset with 600 questions.' This is not cosmetic: if the 600-question set is not the official FreshQA, or if the single-hop/multi-hop split is defined differently from the baselines, the comparison is uncontrolled. Second, Section 4.4 says review-then-refine 'achieves 39.02% multi-hop accuracy' on PAT-Questions, but Table 3 shows 39.02 is the single-hop accuracy; the multi-hop value is 27.59. The cited baseline numbers (DSPy 35.84, RAT 35.29) are also single-hop values, so the prose mislabels the metric. Third, Table 2 sets FreshQA top-k to 5, while Appendix A says k is 'typically set to 2 or 3 depending on the task,' with no per-dataset resolution. No code, data, or error bars are provided. The reader's concern about retrieval-indicator calibration is plausible, but secondary: even a miscalibrated indicator could still yield high accuracy. The more load-bearing issue is whether the headline numbers measure what the paper claims.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes review-then-refine, a two-phase framework for multi-hop question answering with temporal information. In the review phase, a complex query is decomposed into sub-queries that are dynamically rewritten using the current reasoning history, and an adaptive retrieval indicator decides whether to call an external retriever for each sub-query; in the refine phase, intermediate answers are aggregated into a final answer. The framework is evaluated on FreshQA, PAT-Questions, MultiHopRAG, and 2WikiMultiHopQA using GPT-3.5-turbo, with additional results on GPT-4o-mini, and the authors report improvements over baselines such as DSPy, RAT, ReAct, and Freshprompt. The paper includes a formalization of the pipeline, ablation studies, case studies, and prompt templates, but no code or data release.","tokens_in":16889,"tokens_out":3990,"duration_ms":33085,"significance":"If the reported results are correct, the framework is a practically useful combination of query decomposition, temporal query rewriting, and adaptive retrieval that could improve multi-hop QA on time-sensitive benchmarks. The paper's strengths are its clear ablation design, the inclusion of case studies, and additional experiments on a second base model, which suggest the method is not restricted to a single LLM. However, the empirical claims currently rest on several internal inconsistencies in the experimental reporting, so the headline state-of-the-art claims are not yet verifiable. The methodological idea is incremental but reasonable; the main risk is whether the evaluation protocol measures what is claimed.","major_comments":[{"comment":"Section 4.1 states that 'FreshQA consists of 377 questions' and immediately says 'We evaluated the dataset with 600 questions.' This is not a cosmetic discrepancy: if the 600-question set is not the standard FreshQA benchmark or is a differently filtered split, the comparisons against baselines in Table 3 are uncontrolled. Please specify the exact source, filtering, and split of the 600-question set, and if the official benchmark is 377 questions, report results on that official set as well.","section":"§4.1, Table 2"},{"comment":"The prose reports '39.02% multi-hop accuracy' for PAT-Questions, but Table 3 lists 39.02 as single-hop accuracy and 27.59 as multi-hop accuracy; the baseline values cited in that paragraph (DSPy 35.84, RAT 35.29) are also in the single-hop column. Because the headline comparison in this paragraph uses the wrong metric column, the claimed improvement on PAT-Questions is not established. Correct the metric labels and re-state the comparisons, or re-run the analysis on the intended column.","section":"§4.4, Table 3"},{"comment":"The definition of single-hop accuracy is ambiguous for FreshQA and PAT-Questions, which do not provide gold sub-questions. Section 4.3 says single-hop accuracy evaluates 'individual sub-questions within the overall multi-hop task,' but no sub-question annotations or automatic decomposition evaluation are described. Please specify how questions were assigned to the single-hop/multi-hop buckets and how the single-hop answers were scored.","section":"§4.3, Table 3"},{"comment":"Table 2 sets top-k to 5 for FreshQA, while Appendix A says k is 'typically set to 2 or 3 depending on the task.' The paper never resolves this discrepancy or reports per-dataset retrieval configurations. Since top-k materially affects retrieval quality, the experimental setup is not reproducible as written; provide exact hyperparameters, seeds, and code or a detailed protocol.","section":"Table 2 vs. Appendix A"},{"comment":"The adaptive retrieval mechanism assumes the LLM's self-assessment of whether its internal knowledge suffices is reliable. No experiment measures the agreement between the retrieval indicator and the actual need for retrieval, nor is there an oracle-retrieval or always-retrieve control. Without such a check, the claim that adaptive retrieval reduces hallucinations is not directly supported.","section":"§3.1, §4.5"},{"comment":"The conclusion that the method achieves state-of-the-art results on all evaluated benchmarks is overstated: on 2WikiMultiHopQA in Table 3, DSPy achieves a higher F1 (45.06 vs. 44.28), and the accuracy advantage is only 0.52 points. Please qualify the state-of-the-art claim to the datasets and metrics where it strictly holds.","section":"§4.4, §5"}],"minor_comments":[{"comment":"The notation is inconsistent: equation (1) says qi and ai are generated by Mθ, while equation (3) defines qi = fi(Q,Hi) without specifying how fi relates to Mθ.","section":"§3.1"},{"comment":"The text says sample prompts 'should be included with corresponding exemplars in Table 2,' but no exemplars appear; include the actual few-shot exemplars used.","section":"Appendix D"},{"comment":"There are numerous grammatical errors and typos (e.g., 'a adaptive retrieval mechanism,' 'we experimented experiments'), which should be corrected.","section":"Throughout"},{"comment":"Some baselines are cited via arXiv preprint identifiers without venue details; for example, the RAT, Self-RAG, and Adaptive-RAG entries should be checked for final published versions.","section":"References"},{"comment":"In the 'Without dynamic rewrite' row, the MultiHopRAG accuracy drop is only 0.29 points (47.25 to 46.96), so the text's claim of a 'significant performance decrease' overstates the effect; soften the language or provide significance testing.","section":"Table 4"},{"comment":"The case studies in Tables 8 and 9 are illustrative but not accompanied by counts of how often such reasoning paths succeed; consider reporting error analysis frequencies.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a natural-language-processing or retrieval-augmented-generation venue, and the framework is plausible, but the current experimental reporting is not reproducible enough to support the state-of-the-art claims. I would ask the editor to require the authors to release the evaluation scripts and exact data splits, and to reconcile the FreshQA count, the top-k settings, and the PAT-Questions metric labeling before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible engineering contribution—dynamic temporal rewriting of decomposed sub-queries plus adaptive retrieval and a refine stage—but the headline numbers rest on an evaluation section that contradicts itself in three places. The stress-test note is right: the FreshQA claim is not verifiable as written.\n\nWhat's actually new: the specific combination. Query decomposition, temporal rewriting, adaptive retrieval, and refinement all exist separately; this paper ties them into one loop where the model decides per sub-query whether to rewrite with a time context and whether to retrieve. The ablations in Table 4 are the most useful part: removing decomposition or retrieval clearly hurts, which supports the design choices. The limitations section is honest about dependence on retrieval quality and efficiency.\n\nThe soft spots are real and load-bearing. Section 4.1 says FreshQA consists of 377 questions, then says 'We evaluated the dataset with 600 questions.' That is not a small discrepancy; if the 600-question set is not the official FreshQA, the comparison to baselines is uncontrolled. The PAT-Questions prose labels 39.02 as multi-hop accuracy, but Table 3 shows it as single-hop; the baselines quoted next to it are single-hop too, so the prose is comparing different metrics. And Table 2 sets top-k to 5 for FreshQA while Appendix A says k is typically 2 or 3 with no per-dataset resolution. No code, data, or error bars. The missing baselines—RQ-RAG and Adaptive-RAG, which are cited in related work—matter because they are the closest prior art. These are fixable, but until they are fixed the SOTA claim should not be taken at face value.\n\nThe reader's concern about the retrieval indicator being miscalibrated is plausible but secondary; even a miscalibrated indicator could still work in practice, and the paper's ablations show retrieval helps. The evaluation inconsistencies are the primary problem.\n\nWho's this for: researchers working on RAG pipelines for temporal QA, or anyone building on decomposition-plus-retrieval. A serious referee should see it because the core idea is sensible and the flaws are correctable. I'd accept it for review with a request for major revision: fix the dataset descriptions, align the reported metrics, add the missing baselines, and release code/data. For my own work, I wouldn't cite the current numbers, but I might cite the framework after a clean revision.","headline":"Plausible framework, but headline numbers are undermined by internal contradictions in the evaluation; deserves refereeing for major revision.","tokens_in":17388,"tokens_out":2963,"would_cite":false,"duration_ms":20457,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A review-then-refine loop beats retrieve-then-read on time-sensitive multi-hop questions.","keywords":["multi-hop question answering","retrieval-augmented generation","temporal reasoning","dynamic query rewriting","adaptive retrieval","hallucination mitigation","question decomposition"],"falsifier":"Measure the retrieval indicator's calibration: run the method on a sample of FreshQA sub-queries, record every $I(q_i,H_i)=0$ decision, and check whether the model's internally generated answer agrees with a verified answer from the current web corpus. A high error rate on those skipped hops would show the adaptive-retrieval mechanism is not reliably detecting when external knowledge is needed, and the reported accuracy would then rest on something other than the framework's stated hallucination-reduction claim.","tokens_in":16369,"feed_emoji":"⏱","tokens_out":8387,"duration_ms":47195,"temperature":0.7,"pith_summary":"The paper is trying to establish that the standard retrieve-then-read pipeline, which fetches documents first and then answers, is the wrong order for multi-hop questions whose answers change over time. Its central claim is that inserting a review phase—decomposing the question into sub-queries, rewriting each one with an explicit time anchor, and deciding per sub-query whether retrieval is needed—followed by a refine phase that merges intermediate answers, improves accuracy and reduces fabricated or misleading content. If true, this gives a practical recipe for keeping large language models current on time-sensitive questions without fetching documents at every step. The strongest reported evidence is on FreshQA, where the method reaches 70.68% single-hop and 60.60% multi-hop accuracy with GPT-3.5-turbo, ahead of all compared baselines.","feed_headline":"Review-then-refine beats retrieve-then-read on time-sensitive QA","feed_subtitle":"Dynamic query rewriting plus adaptive retrieval pushes FreshQA multi-hop accuracy to 60.60 percent with GPT-3.5-turbo.","key_machinery":"The load-bearing object is the review-loop state machine: at each step the model $M_\\theta$ takes the original query $Q$ and history $H_i$ and produces both the next sub-query and an anticipated answer, $q_i,\\tilde a_i = M_\\theta(Q,H_i)$. The retrieval indicator $I(q_i,H_i)\\in\\{0,1\\}$ is the gate that decides whether to call the retriever or rely on internal knowledge; reasoning states evolve as $s_{i+1}=T(s_i,q_i,a_i,D_i)$. The refine phase's aggregation function $A$ then fuses the intermediate answers into the final answer. This machinery is what lets the framework attach temporal anchors to each hop and avoid fetching documents for hops the model already knows.","core_discovery":"On its own terms, the paper claims that multi-hop QA with temporal information is better served by a review-then-refine loop than by retrieve-then-read. In the review phase the model generates a chain of sub-queries $q_i = f_i(Q,H_i)$ conditioned on the original question and the reasoning history $H_i$, rewrites them to carry an explicit time context, and uses a retrieval indicator $I(q_i,H_i)$ to decide between external search and internal knowledge. In the refine phase the intermediate answers $\\{a_1,\\ldots,a_n\\}$ are aggregated into the final answer $A_f=A(\\{a_1,\\ldots,a_n\\})$. The paper reports that with GPT-3.5-turbo this design reaches 70.68% single-hop and 60.60% multi-hop accuracy on FreshQA, surpassing all baselines, and also improves over baselines on PAT-Questions, MultiHopRAG, and 2WikiMultiHopQA.","pith_inferences":["Inference: because the retrieval indicator is a single LLM self-assessment, the framework's gains could be made more robust by calibrating that decision against a small labeled set of sub-queries; the paper neither trains nor evaluates such a calibrator.","Inference: a natural stress test is to run the method on questions whose answers change more than once within the corpus window; if dynamic rewriting always anchors to one date, competing temporal anchors could produce conflicting intermediate answers.","Inference: the same two-phase loop could be applied to continuously updated knowledge bases, where the rewrite step could explicitly target the most recent snapshot; the paper does not explore this setting."],"forward_implications":["Removing the dynamic rewrite module drops FreshQA multi-hop accuracy from 60.60% to 55.62%, so the temporal rewriting step is doing real work.","Removing query decomposition drops FreshQA multi-hop accuracy to 48.72% and PAT-Questions multi-hop accuracy to 15.46%, so decomposition is the largest single contributor on dynamic benchmarks.","Removing the retrieval module drops PAT-Questions multi-hop accuracy from 27.59% to 16.03%, confirming that external evidence remains necessary even when internal reasoning is strong.","The same pipeline transfers to static multi-hop benchmarks, where it is competitive with or ahead of DSPy on MultiHopRAG and 2WikiMultiHopQA."],"supporting_citations":[{"why":"Supplies the FreshQA benchmark and the Freshprompt baseline; the paper's headline claim is measured against it.","marker":"Vu et al., 2023"},{"why":"Provides 2WikiMultiHopQA, one of the two regular multi-hop benchmarks used in the evaluation.","marker":"Ho et al., 2020"},{"why":"Provides MultiHopRAG, the retrieval-augmented multi-hop benchmark where the method reports a clear regular-task gain.","marker":"Tang and Yang, 2024"},{"why":"Provides PAT-Questions, the present-anchored temporal QA benchmark used for dynamic evaluation.","marker":"Meem et al., 2024"},{"why":"Defines DSPy, the strongest compared baseline that review-then-refine outperforms on FreshQA multi-hop accuracy.","marker":"Khattab et al., 2024"},{"why":"Supplies ColBERT-v2, the retriever used to fetch external documents within the framework.","marker":"Santhanam et al., 2021"}],"fun_headline_variants":["Review-then-refine boosts temporal multi-hop QA","Adaptive retrieval + query rewriting for time-aware QA","Dynamic review-refine loop outperforms retrieve-read","Time-sensitive QA improved via review-then-refine"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The language model can reliably decide, on its own, whether it needs to look something up before answering each sub-query; if that self-assessment is miscalibrated, the framework will skip needed searches or fetch irrelevant documents and the claimed hallucination reduction collapses.","fun_headline_variants_meta":{"raw":{"variants":["Review-then-refine boosts temporal multi-hop QA","Adaptive retrieval + query rewriting for time-aware QA","Dynamic review-refine loop outperforms retrieve-read","Time-sensitive QA improved via review-then-refine"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1285,"prompt_tokens":962,"completion_tokens":323,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":261}},"tokens_in":578,"tokens_out":323,"duration_ms":3598,"temperature":1.0,"reasoning_tokens":261,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:37:49.605411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the retrieval indicator's calibration: run the method on a sample of FreshQA sub-queries, record every $I(q_i,H_i)=0$ decision, and check whether the model's internally generated answer agrees with a verified answer from the current web corpus. A high error rate on those skipped hops would show the adaptive-retrieval mechanism is not reliably detecting when external knowledge is needed, and the reported accuracy would then rest on something other than the framework's stated hallucination-reduction claim.","supporting_citations":[{"cited_title":"PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering","cited_arxiv_id":"2402.11034","evidence_quote":"Provides PAT-Questions, the present-anchored temporal QA benchmark used for dynamic evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines DSPy, the strongest compared baseline that review-then-refine outperforms on FreshQA multi-hop accuracy."}],"review_version":1}