{"id":"6add509d-67d3-4545-9320-64b33e05c340","arxiv_id":"2504.19436","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A state-aware query reformulation with soft attention retrieval is claimed to improve BLEU and ROUGE-L in RAG, but the experimental comparison omits a static retrieval baseline.","lead":"This paper proposes a dynamic retrieval mechanism for RAG that adjusts the search query with the generator's current state, and reports BLEU and ROUGE-L gains on Natural Questions. A generalist might read it as one more incremental variant in a crowded field, but the lack of a static baseline means the headline improvement is not demonstrated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are uninterpretable: the retrieval pool appears to contain the gold answer passages, so dynamic retrieval is never tested against a real open-domain index.","rationale":"The reader's weakest assumption identifies the same load-bearing problem: the evaluation does not challenge the retrieval module because the candidate passages are tied to each question and contain the gold answer. My read of Section III.A confirms that the knowledge base is described as the associated Wikipedia passages with the gold passage used as supervision. This makes the experiment a near-tautology for the retrieval component: even a weak retriever can locate the answer-bearing passage, and the generator then benefits from answer context regardless of whether the dynamic scheduling mechanism works. The paper provides no static full-system baseline, no error bars, no significance tests, no code, and no formal verification; these omissions are consistent with the central claim being unverified. The proposed architecture may still have merit, but the evidence as presented cannot separate the method's contribution from the retrieval-set construction. Therefore the reader's REJECT verdict is appropriate, and my stress-test does not change it.","tokens_in":6277,"tokens_out":3172,"duration_ms":34800,"concrete_test":"Re-run the Section III experiment with a realistic open-domain index: for each NQ question, build the candidate set D by DPR or BM25 top-100 retrieval from the full Wikipedia corpus, excluding the gold passage, and compare the proposed dynamic method against a static top-k RAG baseline using the same generator. Report Exact Match, F1, and retrieval recall@k. If the dynamic-versus-static difference is within noise, or if the gains vanish when the gold passage is removed from the candidate set, then the central claim that dynamic retrieval improves generation quality is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the state-aware retrieval vector and differentiable document matching improve generation quality by dynamically selecting knowledge during generation. For that claim to hold, the retrieval module must be genuinely challenged by a large corpus in which the answer is not guaranteed to be present. Section III.A says the questions in NQ are used as queries, 'the associated Wikipedia passages as the external knowledge base D,' and 'the provided gold passage is used as the supervision signal.' This implies each question's candidate set contains the very passage from which the gold answer was extracted. With the gold passage inside D, a retrieval module can succeed by selecting it, and the generator is effectively handed the answer context. The dynamic-retrieval effect is therefore confounded with answer leakage. The only comparative evidence, Figure 2, pits retrieval-vector construction variants against each other; it includes no static top-k RAG baseline, no error bars, and no significance testing. Table 1 compares different LLMs under the same dynamic mechanism, not dynamic versus static retrieval. Consequently, the headline result that dynamic retrieval yields 'significant improvements in BLEU and ROUGE-L' does not establish that the dynamic mechanism caused the improvement. The retrieval-pool issue is the decisive flaw; BLEU/ROUGE-L being poorly suited to open-domain QA compounds it but is secondary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a dynamic retrieval mechanism for retrieval-augmented generation (RAG). The method introduces a state-aware retrieval vector q'_t computed from the query and the generator's hidden state via an MLP, a differentiable document selection module based on scaled dot-product attention, and a joint loss combining a generation cross-entropy term with a contrastive retrieval loss. The authors evaluate on the Natural Questions dataset with several LLMs (GPT-3.5, GPT-4, GPT-4o, Qwenmax, DeepSeek) and report BLEU and ROUGE-L scores, an ablation over retrieval vector construction methods, and a robustness study under query ambiguity. The central claim is that dynamically adjusting the knowledge input during generation avoids context drift and significantly improves generation quality.","tokens_in":6543,"tokens_out":5605,"duration_ms":55105,"significance":"The motivation is plausible and the proposed mechanism is simple enough to be implemented in existing RAG pipelines. Conditioning retrieval on the generation state and making the matching path differentiable are both active and reasonable research directions. However, the manuscript provides no code, no data, no hyperparameter specifications, and its experiments are not designed to support the headline claim. The retrieval pool appears to contain the gold answer passage for each question, which confounds the results with answer leakage, and no static RAG baseline is included. If the evaluation were properly redone on a genuine open-domain index with standard baselines, the method might show value, but as it stands the empirical evidence is not convincing. The paper also has significant presentation issues, including garbled equations and missing figures.","major_comments":[{"comment":"The candidate set D is built from 'the associated Wikipedia passages' for each NQ question, and 'the provided gold passage is used as the supervision signal.' Because the gold answer passage is contained in D for every query, the retrieval module is never tested against a genuine open-domain index in which the answer may be absent. The reported generation improvements are therefore confounded by answer leakage: the generator is effectively handed the answer context, so the experiments cannot support the claim that dynamic retrieval improves generation quality. The authors should re-evaluate on a full-corpus retrieval setting, report retrieval metrics such as Recall@k, and include a condition in which the gold passage is excluded from D.","section":"III.A (Datasets)"},{"comment":"The experiments do not compare dynamic retrieval against a static RAG baseline. Table 1 varies only the generator LLM under the same dynamic mechanism. Figure 2's 'basic static query method' is not a conventional static top-k RAG baseline; it is a different retrieval-vector construction within the same differentiable pipeline. Without an identical-condition static RAG comparator, the claimed 'significant improvements' of the dynamic mechanism cannot be attributed to the proposed method. In addition, Figure 2 is only described textually, and no error bars, confidence intervals, or significance tests are provided for any reported BLEU/ROUGE-L values.","section":"III.B (Table 1 and Figure 2)"},{"comment":"The robustness study is not reproducible. The paper does not specify how queries were classified into low, moderate, and high ambiguity, who performed the labeling, or how many queries were in each subset. The 'robustness' percentages (92.3%, 87.5%, 81.2%) are never defined, and the figure itself is absent from the manuscript. Without this information, the reported decline under ambiguity cannot be interpreted.","section":"III.B (Figure 3)"},{"comment":"Several parameters and implementation details needed to reproduce the central result are missing. The hyperparameter λ in the joint loss L_total = L_gen + λ L_ret is never specified or analyzed, the size of the NQ split used in each experiment is not stated, and the retrieval loss construction (positive/negative document pairs) is not described. The conclusion asserts 'significant' improvements, but no statistical tests, variance measures, or exact test-set sizes are reported anywhere.","section":"II (Joint loss) and III.A (splits)"}],"minor_comments":[{"comment":"The mathematical notation in Section II is not typeset correctly (e.g., the equation for q'_t appears as ']);([' tt hqMLPq =' and the retrieval probability formula is garbled). This must be fixed before the method can be understood.","section":"II (Equations)"},{"comment":"Figures 1 and 3 are referenced in the text but do not appear in the manuscript; only Figure 2 is described numerically. The missing figures make the architecture and the robustness results impossible to verify.","section":"I (Figures)"},{"comment":"The manuscript contains several formatting errors: the header 'Abstract-This' runs together, 'Keywords-dynamic retrieval' lacks a space, and the affiliation block repeats 'San Francisco State University' twice. A full proofread is needed.","section":"I (Typos)"},{"comment":"References [17]-[19] are cited as the basis for the hierarchical semantic encoding, differentiable matching, and dynamic scheduling, but the titles of those papers do not transparently support these specific techniques; the authors should cite the original methodological sources (e.g., RAG, DPR, REALM, Atlas) instead.","section":"II (References)"},{"comment":"The evaluation relies on BLEU and ROUGE-L for open-domain question answering without justifying these lexical-overlap metrics. Reporting exact-match or answer F1 alongside would make the results more meaningful.","section":"III.B (Metrics)"},{"comment":"The term 'multi-level perceptive retrieval vector construction' is an overstatement for a single-layer MLP over the concatenation of q and h_t; a more measured description would match the actual architecture.","section":"II (Naming)"}],"recommendation":"reject","confidential_remarks":"The experimental design flaw is fundamental: the retrieval pool contains the gold passage, so the dynamic-retrieval effect is not actually tested. The manuscript also reads as an early draft, with garbled equations and missing figures. I would encourage the authors to completely redo the evaluation on a genuine open-domain index with static baselines and statistical reporting before resubmitting; as it stands, the central claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe punchline: this paper's central claim isn't supported by its own experiments. The retrieval setup looks like it leaks the gold answer, and there's no static-retrieval baseline in the model comparison. The method itself—making the retrieval vector a function of the generation state and training retrieval and generation jointly via a differentiable attention path—is a reasonable, if incremental, way to frame dynamic RAG. The components (query-conditioned hidden-state MLP, soft attention over document vectors, contrastive retrieval loss) are all familiar from DR-RAG, DRAGIN, and REALM-style work, and the authors cite some of that work. The combination is coherent, but I can't call it new in any meaningful sense.\n\nWhat the paper does well: it's clearly written, the architecture diagram matches the equations, and the authors do run experiments across several LLMs. The ablation in Figure 2 at least compares their proposed retrieval-vector construction against a static query encoding and a context-augmented variant, which is the right kind of control.\n\nWhere it falls apart: Section III.A says the questions in NQ are used as queries, the associated Wikipedia passages as the external knowledge base D, and the provided gold passage as the supervision signal. That implies each question's candidate set contains the very passage from which the answer was drawn. With the answer inside D, a retrieval module that picks the gold passage hands the generator the answer context, and the \"dynamic\" advantage is confounded with leakage. The stress-test note makes exactly this point, and after reading the paper I think it's decisive. There are also no error bars or significance testing anywhere, and BLEU/ROUGE-L are weak metrics for open-domain QA. The robustness study uses an unspecified ambiguity labeling, so those numbers are uninterpretable. The method section leaves out training details (batch size, learning rate, lambda schedule, number of retrieval documents) that would be needed to reproduce the work.\n\nThe reference list is padded with unrelated topics (dermatology, traffic scheduling), which doesn't help confidence.\n\nBottom line: this is a desk-reject for a serious venue. The evaluation design breaks the central claim. If the authors redo the experiments with a genuine open-domain index and include a static top-k baseline and proper metrics (e.g., exact match, answer F1), the architecture might earn a workshop-level contribution. As it stands, I wouldn't cite it or recommend anyone spend referee time on it.","headline":"The architecture is a plausible incremental dynamic RAG variant, but the NQ retrieval setup appears to leak gold answers, and the experiments never actually test dynamic retrieval against a real open-domain index.","tokens_in":7014,"tokens_out":3742,"would_cite":false,"duration_ms":36437,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A state-aware retrieval vector, recomputed at each generation step, improves retrieval-augmented generation quality.","keywords":["retrieval-augmented generation","dynamic retrieval","state-aware retrieval vector","differentiable document matching","joint retrieval-generation training","open-domain question answering","Natural Questions","knowledge scheduling"],"falsifier":"Run the same architecture on an open-domain corpus where the target passage must be found among thousands of unrelated documents, and compare retrieval recall and generation BLEU/ROUGE-L against a static retriever; if the dynamic mechanism does not improve either metric when the gold passage is not guaranteed to be present, the central claim would be falsified.","tokens_in":6094,"feed_emoji":"🔍","tokens_out":8121,"duration_ms":71749,"temperature":0.7,"pith_summary":"The paper argues that a retrieval-augmented generator should not decide once, from the initial question, which documents to use. Instead, it recomputes a state-aware retrieval vector at every generation step from the original query and the text produced so far, so the knowledge brought in can track the model's evolving information needs and avoid the context drift of a static document block. If that mechanism works as claimed, retrieval and generation become one jointly trained system: a differentiable matching path lets the generation objective shape what the retriever selects, and the result is higher BLEU and ROUGE-L scores on open-domain question answering. The paper reports those gains on Natural Questions across several large language models, with GPT-4o scoring highest, and it reports that retrieval vectors built with more context are more robust when queries are ambiguous.","feed_headline":"Adapt knowledge retrieval as the model writes to lift RAG quality","feed_subtitle":"State-aware retrieval vectors are rebuilt at every generation step, beating static context on Natural Questions.","key_machinery":"The load-bearing object is the state-aware retrieval vector $\\mathbf{q}'_t$ and the differentiable softmax path that turns it into the dynamic context embedding $\\mathbf{c}_t$. The vector is produced by an MLP over the concatenated original query and current decoder hidden state, which lets each generated token alter what will be retrieved next; the softmax scores $\\alpha_i$ then fuse the top documents into $\\mathbf{c}_t$, which is passed to the generator. Because the matching is differentiable, training can minimize $L_{\\mathrm{total}} = L_{\\mathrm{gen}} + \\lambda L_{\\mathrm{ret}}$ end-to-end, so the retrieval vectors evolve in the direction that lowers text-generation loss. This is the mechanism that is supposed to deliver the paper's measured BLEU and ROUGE-L improvements.","core_discovery":"The central claim is that dynamic retrieval representation, a context embedding $\\mathbf{c}_t$ recomputed at every decoding step, improves RAG generation quality by preventing context drift. Concretely, the retrieval query is rebuilt as $\\mathbf{q}'_t = \\mathrm{MLP}([\\mathbf{q}; \\mathbf{h}_t])$, documents are scored by a softmax over scaled dot products between $\\mathbf{q}'_t$ and pre-encoded document vectors $\\mathbf{d}_i$, and the weighted context $\\mathbf{c}_t = \\sum_i \\alpha_i \\mathbf{d}_i$ is fed into the Transformer generator together with the hidden state. Since the entire matching path is differentiable, the retriever learns from the generation loss as well as from a contrastive retrieval loss, so the knowledge selected at each step is shaped by what the generator still needs. The paper's experiments on Natural Questions are offered as evidence that this dynamic loop outperforms static query retrieval and query-plus-context variants, and that the advantage grows as the retrieval vectors incorporate more contextual signal.","pith_inferences":["The paper does not test multi-turn dialogue, but the same state-conditional mechanism suggests its largest gains would appear there, since the accumulated dialogue history is a much richer steering signal than a single question.","If the Natural Questions knowledge base is just the passages paired with each question, the retrieval module is choosing among passages that already contain the answer; replacing that index with a large open-domain corpus and measuring passage recall would show whether the gain is genuine retrieval improvement or the generator being handed the gold context.","The differentiable matching path could be extended to adaptive retrieval frequency, so the model decides at each step whether to retrieve zero, one, or many documents; that would make the reported latency and knowledge-scheduling claims directly testable."],"forward_implications":["Static RAG pipelines can be changed so that retrieval is conditioned on the partial output, letting knowledge selection follow the information need as it emerges rather than fixing it from the first query.","End-to-end training of retriever and generator should let weaker generators compensate for missing parametric knowledge by learning when to pull external context.","On ambiguous or multi-document inputs, generation consistency should degrade less, because later tokens can steer the retriever toward more relevant passages.","Under the paper's across-model comparison, the dynamic loop's quality gain sits on top of base-model strength: stronger generators realize larger BLEU and ROUGE-L improvements."],"supporting_citations":[{"why":"Supplies the hierarchical semantic encoding strategy that motivates the multi-level retrieval-vector construction.","marker":"[17]"},{"why":"Provides the joint retrieval-generation framework and differentiable path the paper relies on for gradient flow between modules.","marker":"[18]"},{"why":"Gives the compression and tuning techniques used to balance retrieval quality and latency in large-scale deployment.","marker":"[19]"},{"why":"Supplies the differentiable similarity-estimation method used for document matching.","marker":"[20]"}],"fun_headline_variants":["Rebuild retrieval at each decoding step","Dynamic retrieval boosts RAG quality on NQ","Let the generator steer retrieval in RAG","State-aware retrieval improves generation","Context-guided retrieval lifts RAG output"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the Natural Questions passages used as the knowledge base form a realistic retrieval setting, with the gold answer present but not trivially given; if the index contains only question-associated passages, the retrieval module is not actually being challenged, and the reported gains may reflect the generator receiving the answer context rather than the dynamic retrieval mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Rebuild retrieval at each decoding step","Dynamic retrieval boosts RAG quality on NQ","Let the generator steer retrieval in RAG","State-aware retrieval improves generation","Context-guided retrieval lifts RAG output"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1161,"prompt_tokens":926,"completion_tokens":235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":173}},"tokens_in":542,"tokens_out":235,"duration_ms":3183,"temperature":1.0,"reasoning_tokens":173,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:52:26.587658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same architecture on an open-domain corpus where the target passage must be found among thousands of unrelated documents, and compare retrieval recall and generation BLEU/ROUGE-L against a static retriever; if the dynamic mechanism does not improve either metric when the gold passage is not guaranteed to be present, the central claim would be falsified.","supporting_citations":[],"review_version":1}