Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG

Daeyoung Roh; Donghee Han

Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG

T0 review · 2 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Wrong answers in agentic RAG often come from answering before reading the evidence, not from bad reasoning after reading it.

desk verdict Genuinely useful trajectory-level error decomposition for agentic RAG, with strong empirical hygiene; the zero-read-as-pre-evidence label needs reconciling with the paper's own snippet exception, but the central intervention result holds up. read the letter →

arxiv 2608.02011 v2 pith:GOHVRAXP submitted 2026-08-03 cs.AI

Daeyoung Roh , Donghee Han This is my paper

classification cs.AI

keywords agenticRAGretrieval-augmentedgenerationtrajectory-levelevaluationevidenceinspectionREAD-GATEdisciplinefailuremulti-hopQAruntimeconstraint

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a large share of wrong answers from agentic retrieval-augmented generation systems are procedural failures: the agent searches, sees plausible snippets, and finalizes an answer without ever reading a retrieved passage. The authors show these 'pre-evidence' failures are distinct from ordinary reasoning errors that happen after evidence is inspected, using a trajectory-level decomposition across 12,000 paired runs on three multi-hop QA benchmarks. They then test a minimal runtime rule called READ-GATE that blocks finalization until the agent issues a read action. Forced reading recovers 14.9 to 19.9 accuracy points on trajectories that would otherwise skip reading, and improves overall accuracy by 3.2 to 9.4 points on low-effort settings. The paper also shows that larger hidden thinking budgets do not reliably increase evidence inspection, so evidence-gathering needs to be treated as a separate control problem from answer-side reasoning.

What carries the argument

The central object is the trajectory-level error decomposition with two non-redundant axes: discipline failure (no-read final, snippet-only final, or low-evidence final) versus post-gold-read failure (read at least one gold-supporting chunk, still wrong). The intervention is READ-GATE, a deterministic environment-level invariant that rejects the final action when read count is zero and returns a corrective hint telling the agent to read a promising chunk. READ-GATE does not change model weights, retrieval, the judge, or the reasoning budget; it only enforces that a read tool call happens after search and before finalization. The decomposition is what makes the failure measurable, and the gat

What would settle it

Re-annotate the zero-read correct answers on gold-annotated datasets (HotpotQA and 2WikiMultiHopQA) to separate cases where the snippet actually contains the full extractive answer span from cases of parametric guessing; if a large share of zero-read finals are legitimate snippet-grounded answers, the reported 14.9–19.9 point rescue effect of forcing a read would shrink. A direct check: compute READ-GATE's accuracy gain on the subset where the snippet does NOT contain the answer span, versus the subset where it does.

Watch

Extended reading notes

Core claim

The central claim is that agentic RAG systems can fail before evidence-conditioned reasoning is tested: an agent may retrieve candidate snippets but finalize without inspecting any of them. The paper operationalizes this as a 'discipline failure' measured from saved tool-call traces, and separates it from 'post-gold-read failures' where the agent read at least one gold-supporting chunk and still answered incorrectly. Across 3,807 wrong trajectories, the two failure types overlap in only 11.2–13.1% of cases (depending on the entity extractor), showing they are largely non-redundant error regimes. A minimal runtime invariant, READ-GATE, rejects finalization attempts made before any read action

Load-bearing premise

The decomposition assumes that a 'read' tool call is the only valid evidence-inspection event and that snippet text does not count as inspected evidence, even though the system prompt permits answering without read when the snippet contains the literal answer span.

Editorial extensions

If this is right

If the paper is right, evaluating agentic RAG only by final-answer accuracy hides a distinct, actionable failure mode: the agent never inspects the evidence it retrieved.
Runtime constraints on tool execution (like READ-GATE) can recover errors that stronger reasoning or larger thinking budgets do not necessarily prevent.
Evidence inspection should be tracked and controlled as its own axis, separate from answer-side reasoning, in agent evaluation and deployment.
The benefit of a read-before-final gate is conditional: it helps most when residual discipline error is high, and can hurt when the agent already reads reliably, so it should be enabled based on observed zero-read rates in logs.
Prompt-only instructions can reduce zero-read behavior but do not replicate the accuracy gain of an execution-level constraint, suggesting the environment, not just the prompt, must enforce the procedure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same trajectory-level lens could extend beyond multi-hop QA to any agentic tool-use setting where an agent can act on surface signals without inspecting the full artifact—for example, code execution, database queries, or web browsing—where 'read' generalizes to any evidence-inspection action.
A testable extension: monitor the zero-read finalization rate in production agent logs as a cheap, label-free proxy for when to enable a gate like READ-GATE; the paper's H2 predicts the gate's benefit scales with this rate.
The action-commitment hypothesis (self-issued read beats passively injected context) suggests that agency itself—issuing the tool call—may affect how the model weighs evidence; this could be explored by varying tool-role vs user-role delivery across more backbones and domains.
The finding that larger hidden thinking budgets can increase zero-read finalization warns that compute scaling alone is not a substitute for trajectory-level procedural controls; this is worth checking in non-Wikipedia RAG domains where snippets have different semantics.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one-sentence take: this paper isolates a real failure mode that most RAG evaluation ignores—agents that retrieve and then answer without reading—and it does so with unusually careful controls. The decomposition into discipline failures (no-read/snippet-only/low-evidence) versus post-gold-read failures is new to me, and the 11-13% overlap between the two regimes across two entity extractors makes the case that these are distinct error axes. READ-GATE, a simple environment-level "must read after search" invariant, gets consistent gains where zero-read finalization is common, and the mechanism ablations (prompt-only, context-injection, within-strata reclassification, cross-judge kappa=0.924) are exactly what you want to see. The public release of 33,950 trajectories and reproduction scripts is a real asset; it turns the paper from a claim into a dataset for the community.

The soft spots are real but manageable. The H1 non-redundancy test uses an arbitrary 60% collapse threshold without a proper null model; the conclusion is plausible, but the test is more descriptive than inferential. Bigger issue: the paper's own system prompt explicitly permits zero-read answers when the snippet contains the literal answer span, so the "no-read = pre-evidence failure" label counts some prompt-sanctioned answers as failures. Appendix I quantifies this: about 20% of correct zero-read answers on gold-annotated datasets show clean snippet leakage. That doesn't necessarily kill the rescue effect—my back-of-envelope on Table 1 suggests excluding those already-correct leakage cases could actually make the relative gain look larger, not smaller—but the paper should run that reanalysis instead of leaving the label fuzzy. The medium-reasoning READ-GATE ablations are on n=100 matched subsamples and show a real risk of budget-truncation artifacts (the 2Wiki -7pt cost is mostly loop-cap abstentions); that's acknowledged, but it limits how much weight you can put on the negative result. The Qwen cross-family check is similarly underpowered.

None of this undermines the central claim that evidence-gathering is a separate control axis from answer-side reasoning. The paper is honest about its limitations and the trajectory corpus will be useful to people building agentic RAG eval tools.

Recommendation: send it to a serious referee. It deserves a full review, not a desk reject. The referee should push on the H1 null and the snippet-exception reclassification, but the core contribution is solid.

Referee Report

2 major / 4 minor

Summary. The paper proposes a trajectory-level decomposition of wrong answers in agentic RAG into 'pre-evidence discipline failures' (the agent finalizes without reading retrieved evidence) and 'post-gold-read failures' (the agent reads gold-supporting evidence but still answers incorrectly). Using 12,000 paired OpenAI-controller trajectories on HotpotQA, 2WikiMultiHopQA, and MuSiQue, it reports that these failure types are largely non-redundant (both-trigger rate 11.2–13.1%), introduces READ-GATE (a runtime read-before-final invariant), and finds forced reading improves LLM-Acc by +14.9–19.9 points on the zero-read subset and +3.2–9.4 points on full minimal-reasoning cells. It also reports diagnostics showing that larger hidden thinking budgets do not reduce zero-read finalization. The paper releases code, trajectories, and cached analysis outputs.

Significance. If the decomposition is valid, the paper identifies a distinct, actionable failure axis in agentic RAG: agents can retrieve plausible evidence and answer without inspecting it, and a minimal runtime gate can recover a substantial share of these errors without changing the model or retriever. The empirical work is strong in several respects: paired McNemar tests, bootstrap CIs, a cross-judge re-evaluation (κ=0.924), prompt-only and context-injection controls, threshold sweeps, and public trajectories. The H1–H3 predictions are falsifiable and mostly well tested. However, the central measurement assumption, that zero-read finalizations are pre-evidence failures, is in tension with the paper's own agent prompt and leakage analysis. This issue must be resolved before the reported effect sizes can be attributed to the proposed mechanism.

major comments (2)

[§3.2, App. A.1, App. I, Table 1] The discipline-failure definition equates 'read count = 0' with 'no evidence inspection', but the system prompt (App. A.1) explicitly allows answering without read when the snippet contains the literal extractive answer span and matches the question entity exactly. Zero-read finals are therefore not uniformly procedural violations. App. I's own leakage numbers show 23.7% of wrong zero-read finals and 19.5% of correct zero-read finals have the predicted entity in a retrieved-but-unread snippet. Table 1's rescue gain is computed on the entire zero-read subset, which mixes prompt-sanctioned snippet answers with unsupported guesses. Please reclassify or exclude the exception cases and report the induced-read gain on the genuine no-evidence subset; otherwise the +14.9–19.9 pp estimate cannot be assigned to pre-evidence failures.
[§5.1] The non-redundancy test is summarized as 'the overlap sits far below the 60% level at which the two axes would collapse into one.' The 60% threshold is not derived from any model of independence, maximum possible overlap, or cost structure, and it is not a test statistic. The claim would be more defensible if quantified with a principled measure, for example the overlap expected under independence given the marginals, or an odds ratio / tetrachoric correlation between the two labels. The permutation and interaction tests in §5.5 already provide stronger evidence for non-redundancy; the 60% reference should be removed or justified.

minor comments (4)

[Abstract / §5.1] The both-trigger rate is conditional on wrong trajectories; make this explicit in the abstract so readers do not read it as a population-level rate.
[Table 3] The mixture of n=1,000 and n=100 rows is easy to misread. Please add a sample-size column or otherwise make the paired n=100 boundary nature of the medium-reasoning rows visually explicit.
[Appendix I] The label 'conservative 300-character definition' is ambiguous for the correct-ZR sweep. Clarify whether 'conservative' means minimizing leakage (favoring the paper's interpretation) or maximizing it.
[Figure 3] The caption states that the x-axis is regime-level, not a scaling curve. A brief explanation in the text of why no connecting line is drawn would help readers unfamiliar with the design.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: READ-GATE gains are empirical, and the paper's operational definitions are disclosed rather than smuggled in.

full rationale

The derivation chain is self-contained. READ-GATE is a runtime invariant that blocks finalization when read count = 0; the reported accuracy gains (+14.9–19.9 on the zero-read subset and +3.2–9.4 on full minimal-reasoning cells) are paired empirical contrasts that require the model to do something after the forced read, so they are not tautological. The paper explicitly separates the mechanical reclassification effect on Ppost from the behavioral effect using read-exposure-matched strata (Table 4, OR 1.00 [0.84,1.19]) and a within-cell label-permutation test, so it does not present a definitional consequence as an independent finding. Entity-coverage thresholds and priority ordering are deterministic definitions that are swept in robustness checks, not fitted parameters renamed as predictions. There are no load-bearing self-citations: the central claims rest on the authors' own saved trajectories and re-measured controls, and the only externally cited rows (A-RAG medium, from Du et al. 2026) are a secondary interface comparison, not the basis for the pre-evidence decomposition. The one caveat is that 'pre-evidence' is operationalized as 'no read tool call,' which is a definitional choice. Appendix A.1 even permits answering without read when the snippet contains the literal extractive answer span, and Appendix I estimates that 19.5% of correct zero-read gold-annotated answers show clean snippet leakage. This is a construct-validity disclosure rather than a circular derivation, because the paper's headline quantities—frequencies, gains, overlap rates—are measured from saved traces and are not implied by the definition alone. Score 1 reflects this minor definitional labeling issue without treating it as circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced. READ-GATE is a software environment rule and the corrective hint is a prompt-level mechanism, not an invented entity. The free parameters are definitional thresholds used in the labeling pipeline, and they are swept or validated rather than fitted to the headline accuracy gains.

free parameters (2)

low-evidence entity coverage threshold = 0.8
Discipline subtype low-evidence final fires when question-entity coverage in read chunks is below 80% (Section 3.2, Appendix G). Chosen by hand; sweep 0.6-0.9 shows stability on discipline rates.
snippet length for leakage analysis = 300 characters
Appendix I defines the headline snippet-leakage rate using 300-character prefixes; sweeping from 100 characters to full chunk changes the wrong-zero-read leakage estimate from 12.2% to 60.3%, so the headline is a definitional choice.

assumptions (4)

domain assumption The 'read' tool call is a faithful observable proxy for evidence inspection; snippet text does not count as inspected evidence.
Establishes discipline labels based on read count and entity coverage in Sections 3.1-3.2; contradicted in part by the prompt's snippet exception (Appendix A.1), making the proxy conservative but incomplete.
domain assumption Per-chunk gold supporting-evidence annotations are available and correct for Epost classification.
Epost requires gold chunk IDs; Section 4.1 restricts Epost analyses to HotpotQA and 2WikiMultiHopQA because MuSiQue lacks complete gold-evidence fields.
domain assumption LLM-as-judge (gpt-5-mini) judgments of semantic equivalence are adequate for measuring answer accuracy.
Section 4.4 uses a fixed judge; Appendix L cross-judge agreement with Gemini 2.5 Pro at kappa=0.924 on n=450 supports but does not eliminate judge bias.
domain assumption The entity matcher (regex/spaCy) identifies question entities well enough to label low-evidence finals.
Appendix G validation on n=60 reports overall recall 0.734 and single-token entity recall 0.148, so coverage labels may miss single-token entities.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG." pith.science (2026). https://pith.science/paper/GOHVRAXP

@misc{pith2026260802011,
  author       = {Pith},
  title        = {Pith review of: Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GOHVRAXP}},
  note         = {Machine review of arXiv:2608.02011}
}

read the original abstract

Agentic retrieval-augmented generation (RAG) systems can fail before evidence-conditioned reasoning is tested: an agent may retrieve candidate snippets but finalize without inspecting them. We study this failure mode as a procedural property of the agent trajectory, decomposing wrong answers into pre-evidence discipline failures and post-gold-read failures using saved tool-call traces, retrieved evidence, read passages, and final answers. Across 12,000 paired trajectories on HotpotQA, 2WikiMultiHopQA, and MuSiQue, the two failure types are largely non-redundant: the both-trigger rate is in [11.2%, 13.1%] across regex and spaCy entity extractors. We then evaluate Read-Gate, a minimal runtime invariant requiring an agent to read after search and before finalization. Forced reading improves LLM-Acc by 14.9-19.9 points on trajectories that would otherwise skip reading and by 3.2-9.4 points on full minimal-reasoning cells. Additional diagnostics show that larger hidden thinking budgets do not necessarily increase evidence inspection. Together, these results indicate that evidence-gathering should be evaluated as a trajectory-level control problem, separately from answer-side reasoning.