REVIEW 3 major objections 6 minor 19 references
Fishing for Answers: Exploring One-shot vs. Iterative Retrieval Strategies for Retrieval Augmented Generation
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read On complex government-document QA, token-budgeted one-shot retrieval and agentic iterative retrieval each beat fixed top-k RAG by more than 10 points.
desk verdict Plausible practical strategies with a solid new benchmark, but the numbers ride on an unvalidated LLM judge — read as engineering evidence, not established fact. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is replacing fixed top-k retrieval with a token-constrained selection, formalized as a 0-1 knapsack over chunks with relevance scores $r_i$ and token counts $t_i$: maximize $\sum_i r_i x_i$ subject to $\sum_i t_i x_i \le T_{\max}$, which makes chunks compete by relevance per token. The one-shot pipeline then adds a rule-based chunk filter that drops chunks whose metadata (year, location, named entities) does not match the query and adds chunks that do match, followed by an LLM-based chunk cropper that shortens surviving chunks. The iterative pipeline keeps a small top-5 net but gives a reasoning LLM control of multi-turn search, with two fixes for named failure modes: fallback search also retrieves with the original query on the first turn to counteract query drift, and a chunk-delete tool counteracts retrieval laziness, the measured drop in follow-up search probability from 95% at 3k tokens to 25% at 12k tokens.
What would settle it
A human expert panel rating a random sample of answers for factual correctness would settle the claim: if the human-score gap between basic RAG and the best strategy is much smaller than the reported +10 points, the improvement is largely a judge artifact rather than better retrieval.
Extended reading notes
Core claim
The paper's central claim is that the top-k bottleneck in retrieval-augmented generation can be attacked in two opposed ways, and both work on complex government documents. Casting a bigger net means solving a token-budgeted selection problem, $\max \sum_i r_i x_i$ subject to $\sum_i t_i x_i \le T_{\max}$, so chunks compete by relevance per token instead of by fixed rank; adding a rule-based chunk filter that uses metadata such as year and location lifts the average score to 91.0 from an 81.0 baseline, with the largest gains on the two hardest question levels (L3 +14.0, L4 +19.0). Casting the same small net repeatedly means letting a reasoning LLM issue its own queries over up to five turns; adding a fallback search with the original query on the first turn and a chunk-delete tool to trim the working context lifts the average to 90.0 from the same 81.0 baseline (L3 +12.5, L4 +19.0). The paper also reports that the two paths are not complementary as implemented: combining token-budgeted first retrieval with the iterative loop underperforms the pure iterative strategy, because the longer first-round context makes the chunk-delete tool discard useful chunks and makes the agent stop searching too early.
Load-bearing premise
All numerical conclusions depend on the automated judge's 0-100 scores being a faithful measure of answer quality, yet no human-rated validation or correlation check is reported.
Editorial extensions
If this is right
- Replacing fixed top-k with token-budgeted selection alone raises the average score from 81.0 to 87.5, with the biggest gains on reasoning-heavy question levels L3 and L4.
- The best one-shot configuration, adding a rule-based chunk filter, reaches 91.0 overall, a +10.0 gain over basic RAG, and L4 improves by +19.0.
- The best iterative configuration, adding fallback search and chunk delete, reaches 90.0 overall, a +9.0 gain; fallback helps simple questions while chunk delete helps hard ones.
- The two strategies are not complementary as implemented: combining token-budgeted first retrieval with the iterative loop underperforms the pure iterative system, because long first-round contexts make the chunk-delete tool remove useful chunks and worsen retrieval laziness.
Reading between the lines
- A reader might infer that per-token relevance ranking is a cheap, model-agnostic fix that could transfer to any domain with long heterogeneous documents, not just government records.
- The reported relation between context length and follow-up search probability (95% at 3k tokens, 25% at 12k tokens) gives agentic retrieval systems a concrete diagnostic: measure your own retrieval-laziness curve to decide when to cap context or force a search.
- A testable next step would be adaptive chunk deletion that compresses or summarizes context before deciding what to remove, which could reconcile one-shot recall with iterative refinement.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses a known limitation of basic RAG—fixed top-k retrieval often misses relevant evidence—in the context of complex QA over government documents. It proposes and evaluates two retrieval strategies. The One-SHOT strategy removes the fixed top-k constraint and instead selects as many relevant chunks as fit within a token budget (Token-Constrained Top-Kmax), followed by a rule-based chunk filter and an LLM-based chunk cropping module. The iterative strategy uses a reasoning LLM agent that performs multi-turn retrieval, with a first-turn fallback search using the original query and a chunk_delete tool to manage context. Experiments are conducted on a self-constructed 1,000-question benchmark with four difficulty levels, using SenseChat-5 as an LLM-as-a-judge. The paper reports that both strategies achieve roughly +10 point average improvements over a Basic RAG (Top-5) baseline, and that combining the two strategies underperforms the iterative strategy alone due to context overload. The paper also analyzes query drift and retrieval laziness as failure modes.
Significance. If the reported gains are real, the paper offers practically useful recipes for improving RAG on legal and regulatory documents, and the negative result on combining one-shot and iterative strategies is an interesting and non-obvious finding. The ablations are internally consistent: each added component improves over its predecessor, and the largest gains concentrate on multi-chunk L3/L4 questions, which aligns with the proposed mechanisms. The retrieval-laziness measurement in Appendix B provides a plausible cognitive-load explanation for the chunk_delete design. However, the evaluation rests entirely on an unvalidated proprietary LLM judge, and the benchmark is not released; until the judge is calibrated against human ratings or a second independent judge, the headline +10 point improvements are not separable from potential judge bias toward longer, more structured contexts. The paper's practical insights would be much stronger with human-validated evaluation and a public benchmark.
major comments (3)
- [Section 4.1 (Evaluation Setting) and Conclusion] The central claim of over +10% performance gains over basic RAG is measured solely through SenseChat-5 LLM-as-a-judge scores, normalized to a 0-100 scale. The paper reports no human-rated validation, no inter-annotator agreement, no correlation with a second judge, and no details of the judge prompt or scoring rubric. Because both proposed strategies systematically change the length and structure of the context fed to the generator, a judge sensitive to context richness or formatting could reward these outputs even when factual content is not improved. This is load-bearing for every quantitative result in Tables 1 and 4. Please provide a human-validated sample, a second judge, or equivalent evidence that the judge scores track answer correctness, and make the evaluation protocol reproducible.
- [Section 4.3.2, Table 5] The 'DeepSeek*' row renormalizes scores to 'the questions DeepSeek can generate results for,' changing the comparison basis relative to all other rows. Presenting this row without a prominent caveat overstates DeepSeek's performance (93.1 average vs. 90.0 for Qwen3). The raw DeepSeek row shows an average of 81.5, which is essentially at the basic RAG level, so the renormalized row materially changes the conclusion. Please report the raw scores, the rejection rate, and the renormalized analysis separately, and clearly label DeepSeek* as an auxiliary analysis rather than a comparable result.
- [Section 4.1 and Figure 1(a)] The design of both strategies is explicitly motivated by an error analysis on the same test set used later for evaluation (Figure 1(a) states 48% of failures are due to golden chunks outside top-k). Since no held-out development or validation set is described, the reported improvements may reflect an eval-set-informed design loop, where modules are chosen to address observed failure modes in the same data used to measure success. Please clarify whether any separate development set was used, and discuss how the measured gains would generalize to a fresh set of questions from the same corpus.
minor comments (6)
- [Section 1 and Section 4.2.3] Typos: 'dase a larger net' should be 'cast a larger net' in Section 1, and 'overal' should be 'overall' in Section 4.2.3.
- [Section 3.1.1, Eq. (1)] Equation (1) formulates a knapsack-style selection problem, but the implementation is not specified. Please state the algorithm used (e.g., greedy selection by relevance-per-token) and whether the token budget T_max includes the query and prompt overhead.
- [Appendix B, Table 6] The follow-up retrieval probabilities are reported without the number of trials or any variance measure. Please specify the sample size and, ideally, multiple runs, so the reader can assess the reliability of the retrieval-laziness measurement.
- [Table 5] The subscripts in Table 5 are described as 'average retrieval times executed'; this is unusual notation that could be misread as wall-clock time. Please rename to 'average number of search calls' or similar.
- [References] The reference list contains duplicate entries for Search-o1 (Li et al., 2025a and 2025b) and Search-R1 (Jin et al., 2025a and 2025b). Please consolidate them.
- [Section 4.1] The paper states that answers are 'retrievable from the original document corpus,' but it does not describe how ground-truth answers were constructed or validated, nor how the four-level taxonomy was applied to the 1,000 questions (e.g., inter-annotator agreement). A short description of the annotation procedure would strengthen the benchmark.
Circularity Check
No load-bearing circularity: the retrieval strategies are empirically evaluated designs, and none of the reported scores reduces to its own inputs by construction.
full rationale
I walked the claimed derivation chain from the error analysis (Fig. 1a) through the One-SHOT strategy (Sec. 3.1) and the iterative agentic strategy (Sec. 3.2), then to the ablations in Tables 1 and 4. The One-SHOT selection is a token-constrained optimization over relevance scores, followed by rule-based filtering and LLM-based cropping; the iterative loop uses fallback search and chunk deletion. These are architectural choices evaluated on a shared benchmark, not fitted parameters whose values are recovered from the reported scores. The self-citations, such as Liang et al. 2025 for framing reasoning agentic RAG, are not load-bearing for the empirical claims. Two limitations are worth noting outside circularity: the evaluation uses SenseChat-5 as an LLM judge without a reported human correlation, and the design is motivated by an error analysis on the same test set later used for evaluation. These are measurement and generalizability concerns rather than circularity, because the answer-quality scores are not forced by construction from the method definitions. No step in the paper's derivation reduces to its own inputs as demonstrated by an equation or by definition, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Token budget T_max (One-SHOT context window) =
32,000 tokens; 12k-28k in Figure 5
- Retriever top-k in chunk_search =
5
- Maximum retrieval turns =
5
- Chunk filter rules (entity, year, location alignment) =
Not specified
assumptions (5)
- domain assumption Query-chunk embedding similarity is a valid proxy for evidence relevance.
- domain assumption SenseChat-5 judge scores faithfully reflect answer quality.
- domain assumption All 1,000 questions are answerable from the 40,000-document corpus and the golden chunks are complete.
- ad hoc to paper The four-level taxonomy (retrieval reasoning x answer reasoning) captures QA difficulty in this domain.
- ad hoc to paper The Appendix B retrieval-laziness measurements generalize to the deployed iterative system.
Cite this review
Pith. "Pith review of Fishing for Answers: Exploring One-shot vs. Iterative Retrieval Strategies for Retrieval Augmented Generation." pith.science (2026). https://pith.science/paper/KA3ACDQL
@misc{pith2026250904820,
author = {Pith},
title = {Pith review of: Fishing for Answers: Exploring One-shot vs. Iterative Retrieval Strategies for Retrieval Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/KA3ACDQL}},
note = {Machine review of arXiv:2509.04820}
}
abstract
Retrieval-Augmented Generation (RAG) based on Large Language Models (LLMs) is a powerful solution to understand and query the industry's closed-source documents. However, basic RAG often struggles with complex QA tasks in legal and regulatory domains, particularly when dealing with numerous government documents. The top-$k$ strategy frequently misses golden chunks, leading to incomplete or inaccurate answers. To address these retrieval bottlenecks, we explore two strategies to improve evidence coverage and answer quality. The first is a One-SHOT retrieval method that adaptively selects chunks based on a token budget, allowing as much relevant content as possible to be included within the model's context window. Additionally, we design modules to further filter and refine the chunks. The second is an iterative retrieval strategy built on a Reasoning Agentic RAG framework, where a reasoning LLM dynamically issues search queries, evaluates retrieved results, and progressively refines the context over multiple turns. We identify query drift and retrieval laziness issues and further design two modules to tackle them. Through extensive experiments on a dataset of government documents, we aim to offer practical insights and guidance for real-world applications in legal and regulatory domains.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[3]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 oth- ers
-
[4]
Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning.Preprint, arXiv:2503.00223. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025a. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. Preprint, arXi...
-
[6]
En- hancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases.arXiv preprint arXiv:2403.10446. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025a. Search-o1: Agentic search-enhanced large reasoning models.Preprint, arXiv:...
-
[7]
Reasoning rag via system 1 or system 2: A survey on reasoning agentic retrieval- augmented generation for industry challenges.arXiv preprint arXiv:2506.10408. Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Ja- cob Kahn, Gergely Szilvasy, Mike Lewis, and 1 oth- ers
-
[8]
Query rewriting in retrieval- augmented large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis
work page 2023
-
[9]
Chidaksh Ravuru, Sagar Srinivas Sakhinana, and Venkataramana Runkana
Measuring and narrowing the compositionality gap in language models.Preprint, arXiv:2210.03350. Chidaksh Ravuru, Sagar Srinivas Sakhinana, and Venkataramana Runkana
-
[10]
Agentic retrieval- augmented generation for time series analysis.arXiv preprint arXiv:2408.14484. Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, SM_Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das
-
[11]
InNeurIPS 2023 Foundation Models for Decision Making Workshop
Tptu: Task plan- ning and tool usage of large language model-based ai agents. InNeurIPS 2023 Foundation Models for Decision Making Workshop. Aditi Singh
work page 2023
Show all 19 references
-
[12]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others
Agentic reasoning: Reasoning llms with tools for the deep research.Preprint, arXiv:2502.04644. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report.arXiv preprint arXi...
-
[13]
Bin Zhang, Hangyu Mao, Jingqing Ruan, Ying Wen, Yang Li, Shao Zhang, Zhiwei Xu, Dapeng Li, Ziyue Li, Rui Zhao, and 1 others
React: Synergizing reasoning and acting in language models.Preprint, arXiv:2210.03629. Bin Zhang, Hangyu Mao, Jingqing Ruan, Ying Wen, Yang Li, Shao Zhang, Zhiwei Xu, Dapeng Li, Ziyue Li, Rui Zhao, and 1 others. Controlling large lan- guage model-based agents for large-scale d...
-
[14]
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhen- gren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui
Siren’s song in the ai ocean: a survey on hallucination in large language models.arXiv preprint arXiv:2309.01219. Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhen- gren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui
-
[15]
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others
Retrieval-augmented generation for ai-generated con- tent: A survey.arXiv preprint arXiv:2402.19473. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others
-
[16]
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu
A survey of large language models.arXiv preprint arXiv:2303.18223, 1(2). Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu
-
[17]
Yizhang Zhu, Shiyin Du, Boyan Li, Yuyu Luo, and Nan Tang
Deepresearcher: Scaling deep research via reinforce- ment learning in real-world environments.Preprint, arXiv:2504.03160. Yizhang Zhu, Shiyin Du, Boyan Li, Yuyu Luo, and Nan Tang
-
[18]
I have been away from my previous company for 3 years. Can I still re- cover my housing provident fund contributions?
Are large language models good statisticians?arXiv preprint arXiv:2406.07815. A Query Drift To illustrate the query drift phenomenon discussed in Section 4.3, we provide a concrete example here: Original User Query:"I have been away from my previous company for 3 years. Can I ...
-
[19]
Resulting Error:Without the correct chunk, the model provided an incorrect answer based on incomplete information
Chunk Delete Tool Behavior:When presented with this extended context, the chunk delete tool incorrectly removed 9 out of 10 chunks, including the chunk containing the correct answer about the 2023 exhibition. Resulting Error:Without the correct chunk, the model provided an inc...
2023
-
[2023]
arXiv preprint arXiv:2312.10997, 2:1
Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2:1. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others
-
[2024]
InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 371–385
Tptu- v2: Boosting task planning and tool usage of large language model-based agents in real-world industry systems. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 371–385. Patrick Lewis, Ethan Perez, Aleksandra ...
2024
-
[2025]
Atty Eleti, Jeff Harris, and Logan Kilpatrick
Rezero: Enhancing llm search ability by trying one-more-time.Preprint, arXiv:2504.11001. Atty Eleti, Jeff Harris, and Logan Kilpatrick
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.