{"id":"94d3fe9b-c1b1-46a8-8fdf-c5a42238355d","arxiv_id":"2509.04820","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new benchmark on 40,000 Chinese government documents shows that token-budgeted one-shot retrieval and agentic iterative retrieval both improve RAG answer quality by about 10 points over top-k retrieval.","lead":"This paper compares two practical ways to improve retrieval-augmented QA on government documents: stuffing the context with every relevant chunk that fits a token budget, or letting an AI agent search repeatedly and delete unhelpful chunks. Both approaches beat a simple top-5 baseline by about 9 to 10 points on a new 1,000-question benchmark, with caveats about how the answers were scored.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +10% gains rest on an unvalidated proprietary judge; without human-rated calibration or a second judge, the measured improvements could reflect a length/structure artifact rather than answer correctness.","rationale":"The reader's weakest_assumption identifies the same load-bearing premise that I would flag: the SenseChat-5 judge is unvalidated, and both proposed strategies change contextual structure in ways that could inflate an LLM judge's score independent of true answer quality. Because every improvement cited in the abstract and conclusion is expressed in these judge scores, this is the least secure condition for the central claim. I did not find a stronger internal inconsistency: the retrieval procedures are clearly described, the ablations are directionally consistent, and the failure-mode analyses (query drift, retrieval laziness, chunk-delete failure in the combined setting) are concrete and internally coherent. The paper also reports a real deployment over two years, which is supportive but not a substitute for measurement validation. The absence of error bars, significance testing, and a second baseline is concerning but secondary; those weaknesses would matter less if the judge were calibrated. I therefore recommend no change to the CONDITIONAL verdict: the paper should be accepted only if the authors release data/code and provide judge calibration against human ratings, ideally with a second judge and variance reporting. The concrete test above is the minimal check that would settle whether the measured +10/+9 improvements reflect genuine answer-quality gains or judge artifact.","tokens_in":12697,"tokens_out":2684,"duration_ms":26670,"concrete_test":"Take a stratified random sample of at least 100 question-answer outputs spanning Basic RAG, One-SHOT, and the iterative strategy, and re-score them under two conditions: (a) domain-expert human raters grade factual correctness and completeness on the same 0-100 rubric, blinded to which system produced each answer; (b) a different LLM judge (e.g., GPT-4o or Qwen3-32B) scores the same outputs with the identical prompt. If SenseChat-5's scores correlate with human scores below approximately r=0.8, or if the average human-assigned gain over Basic RAG is below +5 points while SenseChat-5 reports +9 to +10, the central claim is not established. If correlations are high and the alternative judge reproduces the same ordering and magnitude, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the Conclusion, is that both strategies achieve over +10% performance gains over basic RAG on complex government documents. That claim is entirely mediated by SenseChat-5's 0-100 LLM-as-a-judge scores (Section 4.1, Evaluation Setting), and no human-rated validation, inter-annotator agreement, or correlation with ground-truth correctness is reported. The two proposed strategies both change the form of the input to the generator: One-SHOT deliberately packs in many more chunks under a token budget, and the iterative strategy accumulates and deletes chunks across turns. A judge that is sensitive to context length, formatting, or self-confidence would systematically reward these longer, more structured outputs even when the underlying factual content is not better. The score gaps in Tables 1 and 4 (81.0 to 91.0 and 81.0 to 90.0) are therefore not separable from potential judge bias on the available evidence. This is a measurement premise, not a flaw in the retrieval logic, but it is load-bearing for every numerical result in the paper. A secondary measurement problem appears in Table 5, where the DeepSeek* row renormalizes scores to 'questions DeepSeek can generate results for,' changing the comparison basis; that issue would not be resolved by judge calibration alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses a known limitation of basic RAG—fixed top-k retrieval often misses relevant evidence—in the context of complex QA over government documents. It proposes and evaluates two retrieval strategies. The One-SHOT strategy removes the fixed top-k constraint and instead selects as many relevant chunks as fit within a token budget (Token-Constrained Top-Kmax), followed by a rule-based chunk filter and an LLM-based chunk cropping module. The iterative strategy uses a reasoning LLM agent that performs multi-turn retrieval, with a first-turn fallback search using the original query and a chunk_delete tool to manage context. Experiments are conducted on a self-constructed 1,000-question benchmark with four difficulty levels, using SenseChat-5 as an LLM-as-a-judge. The paper reports that both strategies achieve roughly +10 point average improvements over a Basic RAG (Top-5) baseline, and that combining the two strategies underperforms the iterative strategy alone due to context overload. The paper also analyzes query drift and retrieval laziness as failure modes.","tokens_in":12929,"tokens_out":6597,"duration_ms":57575,"significance":"If the reported gains are real, the paper offers practically useful recipes for improving RAG on legal and regulatory documents, and the negative result on combining one-shot and iterative strategies is an interesting and non-obvious finding. The ablations are internally consistent: each added component improves over its predecessor, and the largest gains concentrate on multi-chunk L3/L4 questions, which aligns with the proposed mechanisms. The retrieval-laziness measurement in Appendix B provides a plausible cognitive-load explanation for the chunk_delete design. However, the evaluation rests entirely on an unvalidated proprietary LLM judge, and the benchmark is not released; until the judge is calibrated against human ratings or a second independent judge, the headline +10 point improvements are not separable from potential judge bias toward longer, more structured contexts. The paper's practical insights would be much stronger with human-validated evaluation and a public benchmark.","major_comments":[{"comment":"The central claim of over +10% performance gains over basic RAG is measured solely through SenseChat-5 LLM-as-a-judge scores, normalized to a 0-100 scale. The paper reports no human-rated validation, no inter-annotator agreement, no correlation with a second judge, and no details of the judge prompt or scoring rubric. Because both proposed strategies systematically change the length and structure of the context fed to the generator, a judge sensitive to context richness or formatting could reward these outputs even when factual content is not improved. This is load-bearing for every quantitative result in Tables 1 and 4. Please provide a human-validated sample, a second judge, or equivalent evidence that the judge scores track answer correctness, and make the evaluation protocol reproducible.","section":"Section 4.1 (Evaluation Setting) and Conclusion"},{"comment":"The 'DeepSeek*' row renormalizes scores to 'the questions DeepSeek can generate results for,' changing the comparison basis relative to all other rows. Presenting this row without a prominent caveat overstates DeepSeek's performance (93.1 average vs. 90.0 for Qwen3). The raw DeepSeek row shows an average of 81.5, which is essentially at the basic RAG level, so the renormalized row materially changes the conclusion. Please report the raw scores, the rejection rate, and the renormalized analysis separately, and clearly label DeepSeek* as an auxiliary analysis rather than a comparable result.","section":"Section 4.3.2, Table 5"},{"comment":"The design of both strategies is explicitly motivated by an error analysis on the same test set used later for evaluation (Figure 1(a) states 48% of failures are due to golden chunks outside top-k). Since no held-out development or validation set is described, the reported improvements may reflect an eval-set-informed design loop, where modules are chosen to address observed failure modes in the same data used to measure success. Please clarify whether any separate development set was used, and discuss how the measured gains would generalize to a fresh set of questions from the same corpus.","section":"Section 4.1 and Figure 1(a)"}],"minor_comments":[{"comment":"Typos: 'dase a larger net' should be 'cast a larger net' in Section 1, and 'overal' should be 'overall' in Section 4.2.3.","section":"Section 1 and Section 4.2.3"},{"comment":"Equation (1) formulates a knapsack-style selection problem, but the implementation is not specified. Please state the algorithm used (e.g., greedy selection by relevance-per-token) and whether the token budget T_max includes the query and prompt overhead.","section":"Section 3.1.1, Eq. (1)"},{"comment":"The follow-up retrieval probabilities are reported without the number of trials or any variance measure. Please specify the sample size and, ideally, multiple runs, so the reader can assess the reliability of the retrieval-laziness measurement.","section":"Appendix B, Table 6"},{"comment":"The subscripts in Table 5 are described as 'average retrieval times executed'; this is unusual notation that could be misread as wall-clock time. Please rename to 'average number of search calls' or similar.","section":"Table 5"},{"comment":"The reference list contains duplicate entries for Search-o1 (Li et al., 2025a and 2025b) and Search-R1 (Jin et al., 2025a and 2025b). Please consolidate them.","section":"References"},{"comment":"The paper states that answers are 'retrievable from the original document corpus,' but it does not describe how ground-truth answers were constructed or validated, nor how the four-level taxonomy was applied to the 1,000 questions (e.g., inter-annotator agreement). A short description of the annotation procedure would strengthen the benchmark.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The evaluation judge, SenseChat-5, is a proprietary model from the authors' own institution (SenseTime), and the paper provides no independent or human validation of that judge. Given that the central quantitative claim is mediated entirely by this judge, I recommend the editor require external validation before publication. Additionally, the benchmark is closed and the code/dataset are promised only upon acceptance, so reproducibility currently depends on the authors' internal resources."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take. This paper gives practitioners a real comparison of two ways to fix the top-k recall bottleneck in RAG on government documents: pack more chunks under a token budget with rule-based filtering, or run an iterative agentic loop with fallback search and chunk deletion. Both beat a top-5 baseline by roughly 10 points on a new 1,000-question benchmark built from 40k Chinese government documents. The ablations are internally consistent, and the mechanisms (query drift, retrieval laziness) are concrete and tested. I think the qualitative conclusion is believable.\n\nWhat's genuinely new here is the benchmark itself, with four question levels separating retrieval reasoning from generation reasoning, and the two specific modules: chunk filter (year/location/entity alignment) and chunk delete to combat retrieval laziness. The lazy-retrieval measurement in Appendix B, where follow-up search probability drops from 95% to 25% as context grows from 3k to 12k tokens, is a nice concrete demonstration. The paper also honestly reports that combining the two strategies fails, which is the kind of negative result that is often useful.\n\nWhere it is soft. The evaluation layer is the load-bearing weak point. All scores come from SenseChat-5 as judge, and there is no human-rated validation, no inter-annotator agreement, no correlation with factual correctness, and no error bars or significance testing. The two strategies produce longer, structured contexts, so a judge that rewards length or formatting would inflate the gains. I'm not asserting the judge is biased — I'm saying the evidence doesn't let us tell. Table 5's renormalization of DeepSeek scores to only the questions it answered is also problematic; comparing on different subsets isn't a fair comparison. The baseline is only top-5 vector retrieval; adding BM25 or a reranker would give a stronger comparison. And the design is informed by an error analysis on the same test set, which is a mild circularity, though the ablations mitigate it. Code and data are promised but not released.\n\nWho this is for: practitioners building RAG systems for legal/regulatory text will get actionable guidance. Researchers will want the judge issue fixed before trusting the numbers. The paper deserves a serious referee. I'd ask for human judge calibration, a second judge, variance reporting, and stronger baselines before acceptance, but it is not a desk reject.","headline":"Plausible practical strategies with a solid new benchmark, but the numbers ride on an unvalidated LLM judge — read as engineering evidence, not established fact.","tokens_in":13515,"tokens_out":2458,"would_cite":false,"duration_ms":21715,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On complex government-document QA, token-budgeted one-shot retrieval and agentic iterative retrieval each beat fixed top-k RAG by more than 10 points.","keywords":["Retrieval-Augmented Generation","One-SHOT retrieval","iterative retrieval","agentic RAG","token-constrained top-k","government document QA","query drift","retrieval laziness"],"falsifier":"A human expert panel rating a random sample of answers for factual correctness would settle the claim: if the human-score gap between basic RAG and the best strategy is much smaller than the reported +10 points, the improvement is largely a judge artifact rather than better retrieval.","tokens_in":12423,"feed_emoji":"🎣","tokens_out":8965,"duration_ms":64843,"temperature":0.7,"pith_summary":"Basic retrieval-augmented generation fails on complex legal and regulatory QA partly because the fixed top-k retriever misses the chunks that contain the answer; the paper's error analysis attributes 48% of failures to golden chunks falling outside the top-k. This paper tests two remedies. One-SHOT retrieval replaces the fixed top-k with a token-budgeted selection that packs as many relevant chunks as possible into the context window, then filters and crops them. Iterative retrieval uses a reasoning LLM as an agent that searches, evaluates, and refines over multiple turns, with a fallback search to stop query drift and a chunk-delete tool to stop retrieval laziness. On 1,000 questions over 40,000 government documents, the best one-shot configuration scores 91.0 versus 81.0 for basic RAG, and the best iterative configuration scores 90.0; both are reported as over +10% gains.","feed_headline":"Bigger net or repeated casts: both lift RAG by 10+ points","feed_subtitle":"Token-budgeted one-shot and agentic iterative retrieval both beat fixed top-k on complex legal QA.","key_machinery":"The load-bearing mechanism is replacing fixed top-k retrieval with a token-constrained selection, formalized as a 0-1 knapsack over chunks with relevance scores $r_i$ and token counts $t_i$: maximize $\\sum_i r_i x_i$ subject to $\\sum_i t_i x_i \\le T_{\\max}$, which makes chunks compete by relevance per token. The one-shot pipeline then adds a rule-based chunk filter that drops chunks whose metadata (year, location, named entities) does not match the query and adds chunks that do match, followed by an LLM-based chunk cropper that shortens surviving chunks. The iterative pipeline keeps a small top-5 net but gives a reasoning LLM control of multi-turn search, with two fixes for named failure modes: fallback search also retrieves with the original query on the first turn to counteract query drift, and a chunk-delete tool counteracts retrieval laziness, the measured drop in follow-up search probability from 95% at 3k tokens to 25% at 12k tokens.","core_discovery":"The paper's central claim is that the top-k bottleneck in retrieval-augmented generation can be attacked in two opposed ways, and both work on complex government documents. Casting a bigger net means solving a token-budgeted selection problem, $\\max \\sum_i r_i x_i$ subject to $\\sum_i t_i x_i \\le T_{\\max}$, so chunks compete by relevance per token instead of by fixed rank; adding a rule-based chunk filter that uses metadata such as year and location lifts the average score to 91.0 from an 81.0 baseline, with the largest gains on the two hardest question levels (L3 +14.0, L4 +19.0). Casting the same small net repeatedly means letting a reasoning LLM issue its own queries over up to five turns; adding a fallback search with the original query on the first turn and a chunk-delete tool to trim the working context lifts the average to 90.0 from the same 81.0 baseline (L3 +12.5, L4 +19.0). The paper also reports that the two paths are not complementary as implemented: combining token-budgeted first retrieval with the iterative loop underperforms the pure iterative strategy, because the longer first-round context makes the chunk-delete tool discard useful chunks and makes the agent stop searching too early.","pith_inferences":["A reader might infer that per-token relevance ranking is a cheap, model-agnostic fix that could transfer to any domain with long heterogeneous documents, not just government records.","The reported relation between context length and follow-up search probability (95% at 3k tokens, 25% at 12k tokens) gives agentic retrieval systems a concrete diagnostic: measure your own retrieval-laziness curve to decide when to cap context or force a search.","A testable next step would be adaptive chunk deletion that compresses or summarizes context before deciding what to remove, which could reconcile one-shot recall with iterative refinement."],"forward_implications":["Replacing fixed top-k with token-budgeted selection alone raises the average score from 81.0 to 87.5, with the biggest gains on reasoning-heavy question levels L3 and L4.","The best one-shot configuration, adding a rule-based chunk filter, reaches 91.0 overall, a +10.0 gain over basic RAG, and L4 improves by +19.0.","The best iterative configuration, adding fallback search and chunk delete, reaches 90.0 overall, a +9.0 gain; fallback helps simple questions while chunk delete helps hard ones.","The two strategies are not complementary as implemented: combining token-budgeted first retrieval with the iterative loop underperforms the pure iterative system, because long first-round contexts make the chunk-delete tool remove useful chunks and worsen retrieval laziness."],"supporting_citations":[{"why":"Original RAG formulation that defines the retrieval-generation architecture the baseline modifies.","marker":"Lewis et al., 2020"},{"why":"RAG survey that positions basic, advanced, and modular RAG, framing the two strategies.","marker":"Gao et al., 2023"},{"why":"ReAct pattern behind the reasoning-and-acting loop used in iterative retrieval.","marker":"Yao et al., 2023"},{"why":"Search-o1, the agentic search-enhanced reasoning model the iterative design extends.","marker":"Li et al., 2025a"},{"why":"Search-R1, prior work on learning when to search, contextualizing the agentic approach.","marker":"Jin et al., 2025a"},{"why":"Qwen3 technical report that supplies the base language model for all main experiments.","marker":"Yang et al., 2025a"},{"why":"DeepSeek-R1, the alternative reasoning model in the cross-LLM comparison.","marker":"Guo et al., 2025"},{"why":"Choice-overload finding used to explain why combining strategies degrades chunk-delete performance.","marker":"Iyengar and Lepper, 2000"}],"fun_headline_variants":["Both one-shot and iterative retrieval beat top-k, but not together","Casting a bigger net or recasting repeatedly both lift RAG","Token-budgeted one-shot vs agentic iterative: both beat top-k","Fishing for answers: both retrieval strategies work, but not combined","One big cast or many small ones: RAG retrieval gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All numerical conclusions depend on the automated judge's 0-100 scores being a faithful measure of answer quality, yet no human-rated validation or correlation check is reported.","fun_headline_variants_meta":{"raw":{"variants":["Both one-shot and iterative retrieval beat top-k, but not together","Casting a bigger net or recasting repeatedly both lift RAG","Token-budgeted one-shot vs agentic iterative: both beat top-k","Fishing for answers: both retrieval strategies work, but not combined","One big cast or many small ones: RAG retrieval gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1532,"prompt_tokens":1032,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":408}},"tokens_in":648,"tokens_out":500,"duration_ms":4874,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:26:23.799054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human expert panel rating a random sample of answers for factual correctness would settle the claim: if the human-score gap between basic RAG and the best strategy is much smaller than the reported +10 points, the improvement is largely a judge artifact rather than better retrieval.","supporting_citations":[],"review_version":2}