{"id":"ba4d8cba-537c-4d5b-84bc-485262a3d13d","arxiv_id":"2608.06305","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"READ, a deterministic grep-outline-read agent over a raw financial PDF, answers 58.8% of 51 verified questions versus 15.7% for dense top-k retrieval, and the gain sits in the interface, not the iteration loop.","lead":"This paper measures why chunk-and-embed retrieval fails on long financial reports, where a number's unit can sit 13 lines above it, and proposes READ, an agent that searches, outlines, and reads the document directly. On 51 questions READ answers 58.8% versus 15.7% for dense retrieval, and the evidence shows the gain comes from the interface, not from letting the agent iterate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gap may be inflated by the author-built, non-blindly grown 51-question benchmark; an independent pre-registered benchmark on another financial report is needed to confirm READ's interface-level advantage.","rationale":"The paper is unusually careful: paired McNemar with Holm correction, a tuned dense baseline with all swept cells reported, a no-iteration AgenticVec control, mechanical gold validation via arithmetic identities, a conversion-limited category, and full trajectory release. These support the internal validity of the measured gap. However, the external validity of the central attribution claim rests on the 51-question set. The paper's own limitations section states the benchmark was built by the authors and grew non-blindly; the initial 25 questions showed no significant differences, and the added batches were chosen after the authors knew the document's failure profile. That is the classic setting for benchmark selection bias: the hypothesis and the instrument were not separated. The AgenticVec control isolates iteration, but it does not isolate question selection—both READ and AgenticVec are evaluated on the same questions, so a biased question set can favour whichever interface the author expects to win. The BM25 result (not significantly different from READ) is additional evidence that the advantage is not 'agentic search wins' but 'embedding-free beats embedding-based on these particular questions.' No internal inconsistency was found in the statistics or the released artifacts; the concern is about representativeness, not soundness. An independent pre-registered replication is the single check that would settle it. Since the reader's verdict was already CONDITIONAL with this exact assumption flagged, my read does not change the verdict.","tokens_in":21550,"tokens_out":4832,"duration_ms":44141,"concrete_test":"Take a second audited financial report (e.g., another state's Finance Accounts or an RBI annual report), pre-register a fixed set of at least 51 exactness-critical questions before running any system, and run READ, AgenticVec, Dense (with the Section D sweep), and BM25 under the same paired protocol. The decisive quantity is the paired READ-vs-dense difference and its Holm-corrected p-value on the pre-registered set. If the gap is substantially smaller than 23.5 points or no longer significant, the interface-attribution claim must be narrowed; if it replicates, the benchmark-selection concern is discharged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that replacing top-k with READ's deterministic operations, not iteration, produces the 23.5–43.1 point gains—depends entirely on the 51-question benchmark, and the paper's own Section 9 and C.3 concede that benchmark was authored by the implementers of the system under test and grew non-blindly. An initial 25-question version yielded no significant differences; two later batches were added after coverage inspection; both 'happen to fall where chunk-based retrieval is weak' (Section 9); and dense accuracy drifted 24.0%→16.7%→15.7% as questions were added. The questions were drafted against the converted document and after the authors had already measured the document's structural weaknesses (Section 4), so the affordances READ exploits—lexical anchors, outline navigation, span extension to unit headers—are likely over-represented relative to an independently sampled set of exactness-critical questions. The BM25 null (51.0% vs 58.8%, p=1.00 after correction) reinforces this reading: the demonstrated advantage is over embedding-based retrieval, not over lexical search, and a benchmark with less guessable lexical anchors or questions spread across the full 745-page volume could shrink the READ-vs-dense gap. Because every quantitative result is one document, one backbone, 51 questions, the non-blind selection is the load-bearing weak point; the careful paired statistics and disclosed tuning do not remove it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the dominant chunk-embed-top-k retrieval design is structurally unsound for an important class of long, tabular, exactness-critical documents, and proposes READ, an MCP-based interface in which an agent searches and reads the raw document through deterministic operations: normalized lexical search, outline navigation, and bounded span reads. The authors first measure document properties on the Gujarat Finance Accounts 2024-25, Volume I: 86.8% of non-empty lines are table rows, numbers repeat 3.68-fold, and a figure's unit header is a median of 13 lines above it; they then show that a table-aware chunker, offered as a steelman, still leaves 27-30% of numeric chunks without a fiscal-year header at every chunk size. On 51 author-verified questions, READ scores 58.8% versus dense retrieval's 15.7% (p_Holm = 2 x 10^-5), versus dense's best tuned configuration of 35.3% (p_Holm = 0.017), and versus an agent with a vector-search tool instead of READ's operations at 27.5% (p_Holm = 0.012). The paper also reports that BM25 is statistically indistinguishable from READ and that READ is not the most grounded system, explicitly narrowing the claim to separating embedding-based from embedding-free retrieval. The appendices provide extensive protocol detail, a dense-configuration sweep, example trajectories, and a candid discussion of benchmark-construction limitations.","tokens_in":21806,"tokens_out":4613,"duration_ms":45252,"significance":"If the central empirical result holds, the paper would substantively challenge the default application of dense retrieval to structured financial documents, with practical implications for auditability and verifiability in high-stakes settings. The paper is unusually strong in its disclosure practices: it reports dense retrieval's best tuned configuration rather than its worst, includes an agentic control that separates interface from iteration, reports a minimum detectable difference, separates conversion-limited questions from retriever failures, and releases the MCP server and per-cell trajectory records for independent replay. These are genuine strengths. However, the quantitative foundation is a single 51-question benchmark written by the authors of the system under test, and the paper itself documents that the benchmark was extended non-blindly after observing where chunk-based retrieval was weak. Because the headline gaps are large but rest on this one benchmark, the external validity of the interface-attribution claim is not yet established.","major_comments":[{"comment":"The benchmark authorship and non-blind growth are load-bearing for every quantitative claim in the paper. Section 9 states that an initial 25-question version produced no significant differences, that two later batches were added after inspection revealed coverage defects that 'happen to fall where chunk-based retrieval is weak,' and that dense accuracy drifted 24.0% -> 16.7% -> 15.7% as questions were added. Appendix C.3 repeats that the benchmark was drafted by the authors, who also built the system under test. This does not prove the results are artifacts, but it means the central 23.5- to 43.1-point gaps, and the interface-versus-iteration attribution built on them, have not been separated from question-selection effects. I ask that the authors either add an independently constructed, pre-registered question set (ideally on a second document of the same class) or, failing that, substantially weaken the causal language in the abstract and Section 8.1 and mark the result explicitly as an exploratory finding on a self-authored benchmark.","section":"Section 8.1 and Table 8"},{"comment":"The claim that 'the gain belongs to the interface, not to the loop' is underdetermined by the AgenticVec comparison. AgenticVec differs from READ in more than one property: it lacks normalized lexical search, structural navigation, and span-addressed reads simultaneously. Since BM25 (a non-agentic lexical retriever) reaches 51.0% and is statistically indistinguishable from READ, the evidence is consistent with the alternative explanation that the decisive factor is lexical matching over the converted text rather than the specific span-addressed agentic operations. The paper's own ablations reinforce this reading: READ-lit and READ-no-outline each score 66.7%, numerically above READ itself. To support the interface-level attribution, the authors should include a control in which the same agentic loop receives a lexical search tool but not the span-addressed read, or otherwise disentangle the retrieval primitive from the agentic operations. Absent that, the conclusion should be limited to what the data actually separate: embedding-free versus embedding-based retrieval.","section":"Appendix B.3 and Section 4.3"},{"comment":"The table-aware chunker has several free parameters that are fixed but not swept: the backward-search horizon (200 lines), the seed budget, the carried-line budget, and the maximum chunk-body line length. Section D sweeps chunk size and k for the dense baseline, but not these parameters. Since the paper argues the dense baseline is 'steelmanned,' a sensitivity analysis over these parameters, or at least a statement of why the chosen values are not favorable to READ, would strengthen the claim that the residue identified in Section 4.3 is truly invariant. Without it, a reader cannot rule out that some other parameterization of the steelman chunker would recover part of the year-header coverage and change the tuned dense result.","section":"Appendix B.3"}],"minor_comments":[{"comment":"In Table 7, the row labels for the two ablations are somewhat confusing: 'no-outline' and 'naive-grep' may be read as negatively framed even though their accuracies are numerically above READ. Consider renaming them to 'grep+read only' and 'literal-grep' for clarity.","section":"Table 7"},{"comment":"The sentence 'The first row is important' appears after Table 3 but refers to a row not explicitly numbered in the prose; consider adding a row number or an explicit reference such as 'the first data row of Table 3.'","section":"Section 6.3"},{"comment":"In Section 6.3, the description of the numeric matcher says it normalizes 'Indian digit grouping, parenthesised and (-) negatives, and lakh/crore units, at 0.5% relative tolerance,' but the abstract and Section 4.2 mention that lakh and crore differ by a factor of 100. It would be helpful to state explicitly whether the 0.5% tolerance is applied before or after unit normalization.","section":"Section D"},{"comment":"Appendix D reports that the 2,000/k=16 cell scored 35.3% while the same configuration in Table 7 is listed as 15.7% (the k=8 cell), and the footnote attributes a one-question difference to run-to-run nondeterminism. Since the paper otherwise uses paired statistics, it would be useful to report the variance across repeated runs of at least one configuration, so readers can gauge the stability of the reported gaps.","section":"Section 9"}],"recommendation":"major_revision","confidential_remarks":"The paper is far more transparent than most empirical submissions, which is to its credit. The core weakness is not hidden: the benchmark was authored by the system's proponents and grew non-blindly, and the paper says so plainly. I believe this is fixable within a revision through an independent question set or a second document, and the existing disclosure makes the manuscript a strong basis for that work. My recommendation of major revision is driven by the gap between the strength of the causal language in the abstract ('locating the gain in the interface') and the current evidence base, not by any suspicion of misconduct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a careful empirical paper that mostly delivers what it claims, and it's unusually candid about what it doesn't. READ, a deterministic span-addressed retrieval interface, beats dense top-k by 43 points on a 780-page government financial report, and the AgenticVec control (same agent loop, same budget, but a top-k tool) trails by 31 points — so the gain is in the interface, not in iteration. The paper also reports that BM25 is statistically indistinguishable from READ, which is the right boundary: the evidence separates embedding-based from embedding-free retrieval, not agentic from lexical.\n\nWhat's new and useful: the unit-inheritance measurement (median 13 lines from a figure to its governing unit header), the yearless residue that stays ~28% at every chunk size, the conversion-artifact analysis showing digit grouping is the load-bearing normalization, and the conversion ceiling that separates converter damage from retrieval failure. The evaluation protocol is strong: mechanical gold validation, arithmetic identity checks on source rows, derived answers confirmed to not appear in the corpus, and all 663 cells' trajectories are released. The dense baseline is tuned and reported at its best configuration, not its worst.\n\nThe soft spots are real but they're not hidden. 51 questions from one document, one backbone, one converter, and a benchmark that grew non-blindly after the authors already knew where chunk-based retrieval was weak. The paper discloses this and shows dense accuracy drifting from 24% to 15.7% as questions were added. That's an honest limitation, and it means the quantitative gap shouldn't be treated as a general result. An independent pre-registered benchmark on another document is the obvious next test. Two smaller issues: READ's ablations (no-outline and naive-grep) actually score above the full system, so the paper doesn't claim the full operation set is optimal, and READ is not the most grounded system and costs several times more than BM25. Both reported.\n\nThis is for IR/RAG researchers and anyone building tool-using agents over structured documents. It deserves a serious referee. I'd send it out, and I'd let the benchmark authorship concern be discussed openly rather than used to desk-reject.","headline":"Candid, well-controlled empirical paper showing deterministic span-addressed retrieval beats dense top-k on a single financial report; the author-built, non-blindly grown benchmark limits external validity but the core result is believable for this document.","tokens_in":22360,"tokens_out":4113,"would_cite":true,"duration_ms":32535,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a 780-page government financial report, an agent that greps, outlines, and reads the raw document answers 58.8% of verified questions against 15.7% for dense vector retrieval — the gain belongs to the interface, not to iteration.","keywords":["agentic retrieval","retrieval-augmented generation","dense retrieval","financial documents","table-aware chunking","lexical search","interpretability","PDF conversion artifacts"],"falsifier":"An independent team pre-registers 51 questions on a comparable document — a different year's or state's finance accounts, annotated before any retrieval system runs — and reproduces the paired protocol; if the READ–dense gap falls below roughly the 27.7-point minimum detectable difference this design resolves at 80% power, the interface-attribution claim weakens. A cheaper probe on this same document: any chunking scheme that pushes the yearless residue below 27–30% of numeric chunks would refute the invariance observation that anchors the structural argument.","tokens_in":21167,"feed_emoji":"🔍","tokens_out":16273,"duration_ms":108179,"temperature":0.7,"pith_summary":"This paper argues that the dominant retrieval design — chunk the text, embed the chunks, surface the top-k nearest neighbours — is structurally unsound for exactness-critical documents such as financial statements, audit reports, and regulatory returns, where a figure's unit and fiscal year live in a header a median of 13 lines above it. On a 780-page state government financial report, the authors measure the damage: up to 67.4% of numeric chunks carry no unit declaration at small chunk sizes, and even a table-aware chunker built as a steelman leaves an invariant 27–30% of numeric chunks without a fiscal-year header at every chunk size tried. They propose READ, an embedding-free agentic interface whose three deterministic operations — normalized lexical search, structural navigation, and bounded span reads — let the agent extend its reading upward to capture governing headers, so each step is a replayable, citable line range. On 51 verified questions READ answers 58.8% against dense retrieval's 15.7%, and against dense's best tuned configuration (35.3%) it still leads by 23.5 points; an agent with the same loop but a top-k tool reaches only 27.5%, placing the gain in the interface rather than in iteration. The boundary the paper defends is narrower than 'agentic search wins': BM25 is statistically indistinguishable from READ, so the evidence separates embedding-based from embedding-free retrieval.","feed_headline":"Grep-and-read agent beats vector search on a 780-page report","feed_subtitle":"READ answers 58.8% of verified questions vs 15.7% for dense retrieval; even tuned, dense trails by 23.5 points.","key_machinery":"The load-bearing object is READ (Reliable Embedding-free Agentic Document-search), an MCP server that renders the document as ordered lines and exposes four deterministic tools: grep (line search over two normalized shadows of each line, so thousands-commas and words split across table cells do not break matching), list and outline (structural navigation with line numbers), and read (bounded spans capped at 400 lines). The property that carries the argument is span-addressing: the agent selects the interval $[a,b]$ after seeing where its evidence lies, so it can extend upward to include the governing unit and fiscal-year headers that a fixed partition cannot reach. The measurements that make the structural case concrete are the unit-inheritance distance (median 13 lines, p90 26, max 143) and the chunker's invariant yearless residue (27.2–30.1% of numeric chunks across a five-fold chunk-size range).","core_discovery":"The central claim is that on documents whose meaning lives in layout, the retrieval interface — not chunk size, iteration count, or embedding model — determines whether the evidence arrives. The paper establishes this by measuring a mechanism: a figure inherits its unit and fiscal year from headers a median of 13 lines above it, and a chunk partition fixed before the query is known cannot guarantee that context travels with the figure. A steelmanned table-aware chunker nearly eliminates the unit problem (0.3% unitless at 2,000 characters) but the fiscal-year residue stays flat at 27–30% across a five-fold chunk-size sweep, which the authors read as a property of the paradigm rather than a tuning failure. READ leaves the document intact and exposes normalized grep, outline navigation, and bounded reads, so the agent chooses the span after seeing where its evidence lies. The critical control is an agent with the identical loop and budget but a vector-search tool, which reaches 27.5% against READ's 58.8%: the gap is attributed to span-addressed deterministic operations, not to the ability to iterate.","pith_inferences":["If the interface-attribution finding generalizes, then chunker and reranker research on structured documents is optimizing the wrong layer; the testable prediction is that a pre-registered benchmark on other statutory filings reproduces the ordering READ at or above BM25, both above dense, even if the magnitudes shrink.","The yearless-residue mechanism predicts a boundary condition: documents that repeat fiscal-year headers per page or per table section should show a smaller chunking penalty and a correspondingly smaller advantage for span-addressed reading, a prediction a controlled corpus could test.","Giving conversion damage its own question category implies a methodological inference: some published retrieval gaps on PDF-derived corpora may be measuring the PDF converter rather than the retriever, and re-reporting those questions separately would change how existing benchmarks are read.","The cost asymmetry (READ at $0.058 per question against BM25 at $0.020) points to a two-tier deployment for this document class: cheap lexical search by default, with span-addressed agentic reading reserved for questions where exactness or auditability is mandatory."],"forward_implications":["On this document class, dense retrieval's best tuned configuration (2,000-character chunks, k=16, 35.3% accuracy) still trails READ by 23.5 points (Holm-corrected p = 0.017), and every one of the six swept dense configurations is beaten significantly.","An agent with the same loop and budget but a top-k tool reaches 27.5% against READ's 58.8% (p = 0.012), so the gain is attributed to the span-addressed deterministic interface, not to the ability to iterate.","Because every READ operation is a pure function of the document text, trajectories are replayable audit trails: a third party can re-execute the exact line spans and verify each figure in an answer, a property top-k similarity scores cannot offer.","BM25's statistical parity with READ (51.0% vs 58.8%, confidence interval spanning zero) means the evidence separates embedding-based from embedding-free retrieval rather than agentic from lexical search; a practitioner with a lexical index captures most of the available gain at roughly a third of READ's cost.","READ is the most expensive and slowest system per question ($0.058 and 31.4 s against dense's $0.023 and 12.7 s), so adopting it is a correctness-and-auditability trade rather than a cost–accuracy win."],"supporting_citations":[{"why":"The 780-page audited state financial report that is the testbed for every measurement, baseline, and trajectory in the paper.","marker":"Comptroller and Auditor General of India, 2025"},{"why":"Defines dense passage retrieval, the chunk-embed-top-k paradigm that READ is designed to replace and is measured against.","marker":"Karpukhin et al., 2020"},{"why":"Provides BM25, the sparse baseline that is statistically indistinguishable from READ and anchors the paper's narrower claim that the result separates embedding-based from embedding-free retrieval.","marker":"Robertson et al., 1995"},{"why":"Defines the Model Context Protocol through which READ exposes its tools, making trajectories replayable audit trails.","marker":"Anthropic, 2024"},{"why":"Precedent for a tool-using agent navigating a large corpus with grep-and-read primitives, which READ adapts to a single structured document.","marker":"Yang et al., 2024"},{"why":"Shows a fixed grep-and-read pipeline can rival full agent frameworks, supporting the paper's choice of deterministic operations over learned retrieval.","marker":"Xia et al., 2024"},{"why":"Closest prior comparison of grep against vector search inside agent harnesses, which this paper extends to a single tabular document with conversion artifacts.","marker":"Sen et al., 2026a"},{"why":"Formalizes direct corpus interaction as a retrieval interface, the framing lens for READ's claim that the interface, not the loop, explains the gain.","marker":"Li et al., 2026"}],"fun_headline_variants":["Agentic search beats vectors when layout carries meaning","Replacing top-k with agentic reads: 58.8% vs 15.7%","The retrieval interface, not chunk size, decides who wins","Grep beats embeddings when numbers get units from headers 13 lines up"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central result rests on the assumption that its 51-question benchmark — built by the system's own authors and grown non-blindly after an earlier 25-question version showed no significant differences — is representative of exactness-critical questions on this document class and was not unconsciously shaped to favour lexical and structural retrieval.","fun_headline_variants_meta":{"raw":{"variants":["Agentic search beats vectors when layout carries meaning","Replacing top-k with agentic reads: 58.8% vs 15.7%","The retrieval interface, not chunk size, decides who wins","Grep beats embeddings when numbers get units from headers 13 lines up"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000501,"raw_usage":{"total_tokens":2541,"prompt_tokens":1126,"completion_tokens":1415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":742,"completion_tokens_details":{"reasoning_tokens":1338}},"tokens_in":742,"tokens_out":1415,"duration_ms":8735,"temperature":1.0,"reasoning_tokens":1338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:33:39.147076+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent team pre-registers 51 questions on a comparable document — a different year's or state's finance accounts, annotated before any retrieval system runs — and reproduces the paired protocol; if the READ–dense gap falls below roughly the 27.7-point minimum detectable difference this design resolves at 80% power, the interface-attribution claim weakens. A cheaper probe on this same document: any chunking scheme that pushes the yearless residue below 27–30% of numeric chunks would refute the invariance observation that anchors the structural argument.","supporting_citations":[],"review_version":2}