{"id":"f3becd5b-d723-497c-8571-9d69c8788dcf","arxiv_id":"2607.27353","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Schema normalization repairs schema-drift failures in an agentic RAG benchmark (0.000 to 0.913) but does not recover stale, missing, denied, or wrong-session evidence, so reliability should be evaluated layer by layer.","lead":"This paper introduces a benchmark that injects controlled faults into enterprise RAG systems and tests whether a schema-repair intervention fixes only its target layer. Across 38,880 records from nine models, schema repair recovered schema drift but not stale evidence, missing tool output, denied permissions, or wrong-session context.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Layer isolation of negative controls is not directly evidenced: zero repair gain in stale/wrong-session scenarios cannot distinguish schema-invalid outputs from schema-valid outputs with wrong evidence, and the paper reports no schema-validity component rates for those scenarios.","rationale":"The central claim has two pillars: (1) schema normalization repairs schema drift; (2) it does not repair other layers. Pillar (2) is only as strong as the single-layer isolation of the negative-control scenarios. Because only the conjunctive success is reported, a zero repair gain cannot disambiguate 'schema-valid output with wrong evidence/session' from 'schema-invalid output plus wrong evidence/session.' This is not a dispute with the field; it is an internal measurement gap. The paper is otherwise careful: it discloses the metadata-leakage caveat, publishes privacy-safe archives, pairs strict/repair records, and uses a hierarchical bootstrap. Those strengths make the missing component-level audit especially easy to remedy. If the audit passes, the existing conditional acceptance can be confirmed; if it fails, the layer-specific attribution is weakened. The reader's weakest_assumption identified the same general risk, so this stress test agrees with the reader and makes the check concrete.","tokens_in":7453,"tokens_out":10792,"duration_ms":104019,"concrete_test":"Use the public scored-record archives (or deterministic harness) to recompute per-scenario component flags, not just the conjunction. For strict-mode records in stale_index, missing_tool_output, permission_denied, and wrong_session_state, count records with V=true (all tool calls schema-valid) and with E∧R∧G=true. If V=true in at least 95% of each negative-control scenario's strict records, Table 1's single-layer isolation is confirmed and the paper's layer-specific conclusion holds. If V=false in a nontrivial fraction, those scenarios co-vary with the tool-contract layer, and the repair-gain attribution needs revision. Include the same component breakdown for schema_drift to verify that strict failures are V-only and that the 0.913 repair gain is entirely accounted for by restoring V.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 asserts each scenario injects exactly one layer fault, and the paper's layer-specific reading of Table 4 depends on it. The reported success is the conjunction S=E∧R∧G∧V, and repair-mode success for stale_index, missing_tool_output, permission_denied, and wrong_session_state is 0.000. That zero is compatible with a scenario that has both a schema-format problem (V=false) and an evidence/state problem (E/R/G=false): repair fixes the schema component but the record still fails on E/R/G. The paper does not report the schema-validity component rate (V) in the negative-control scenarios, so the reader cannot tell whether those scenarios are single-layer or contain a hidden tool-contract fault. The positive control is internally consistent: schema-drift repair success of 0.913 implies at least 91.3% of schema-drift strict failures had E/R/G=true and V=false, because normalization cannot repair evidence or answer mismatches. No analogous component-level evidence is given for the scenarios that remain at 0.000. Consequently, the claim that those failures 'are not schema-format problems' is an interpretation, not a demonstrated measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LayerRAG-Bench, a benchmark for evaluating agentic retrieval-augmented generation (RAG) reliability across distinct operational layers: evidence freshness, tool contracts, permissions, completeness, metadata integrity, and session state. The benchmark comprises 8 enterprise domains, 240 tasks, 9 fault scenarios, 2 contract modes, and a live matrix of 38,880 records across nine models. The main empirical claim is that bounded schema normalization repairs schema-drift failures (raising success from 0.000 to 0.913) while leaving stale-index, missing-tool-output, permission-denied, and wrong-session-state failures at 0.000, and that groundedness-only evaluation produces substantial false positives especially under wrong-session evidence. The authors argue for a layer-specific evaluation principle: a reliability intervention should be credited for fixing its intended layer without being mistaken for a universal fix. The paper also reports a focused LangChain integration study and releases privacy-safe archives for recomputation.","tokens_in":7693,"tokens_out":8258,"duration_ms":70773,"significance":"If the empirical pattern holds, the benchmark is a useful methodological contribution: it offers a controlled, multi-layer fault matrix with matched strict and repair conditions, and it explicitly argues against collapsing heterogeneous failure modes into a single answer-quality score. The paper is honest about several limitations, including the synthetic corpus, the retriever's domain-resolution failures, and a historical runner that included a version-local answer field in document metadata. It also ships reproducible archives and deterministic checks, which are strengths. However, the central layer-specific attribution is currently under-supported: the paper does not report component-level success rates (E, R, G, V) for the negative-control scenarios, and it does not disclose how many live records terminated before prompt construction. These omissions leave open the possibility that the headline zeros are structural artifacts rather than evidence about the limits of schema repair. The benchmark idea is promising, but the empirical demonstration needs additional analysis and reporting before the claims can be accepted.","major_comments":[{"comment":"The claim that the negative-control scenarios (stale index, missing tool output, permission denied, wrong session state) 'are not schema-format problems' is not directly evidenced. The success metric is S_t = E_t ∧ R_t ∧ G_t ∧ V_t, and repair mode modifies V_t only. A repair success of 0.000 in a negative-control scenario is compatible with both (i) V_t=false alongside E_t/R_t/G_t=false, and (ii) V_t=true while E_t/R_t/G_t=false. The paper does not report component-level rates, in particular the schema-validity rate V_t, for those scenarios. Consequently, the zero repair gain cannot distinguish between a hidden tool-contract fault and a genuinely schema-clean failure. The positive control is internally consistent because a repair success of 0.913 for schema drift implies that at least 91.3% of strict failures had E_t, R_t, and G_t true; but no analogous evidence is given for the negative controls. Please report per-scenario and per-mode component rates (E_t, R_t, G_t, V_t), or at minimum the V_t rate, to substantiate Table 1's one-fault-per-scenario mapping and the layer-specific conclusions in Sections 4.6 and 5.","section":"4.3 and 5 (Table 4)"},{"comment":"The paper reports that prompt hashes are present for only 28,080 of 38,880 records; the remaining 10,800 (27.8%) 'terminate before prompt construction because retrieval is unavailable or invalid.' The paper does not state how these promptless records are distributed across scenarios and contract modes. If they are concentrated in the negative-control scenarios, then the zero success rates in Table 4 for stale index, missing tool output, permission denied, and wrong session state are artifacts of the harness aborting before the model is invoked, not empirical measurements of whether schema normalization can repair those faults. This would make the central claim trivially true by construction. Please report the number of promptless records per scenario and mode, and either (a) modify the harness so that a prompt is always constructed (e.g., with an explicit notice of empty retrieval) so models can attempt abstention or recovery, or (b) explicitly qualify that the main live matrix does not test model-side handling of missing retrieval and that the focused LangChain study is the only evidence for safe handling of such conditions.","section":"6 and 4.3 (Table 4)"},{"comment":"The primary live-matrix results in Table 4 were produced by the historical runner, which serialized a version-local answer field in document metadata alongside answer-bearing prose. The current code removes this field. The paper discloses this and interprets the historical results as end-to-end pipeline measurements, but the disclosure does not resolve the reproducibility concern: the headline numbers (e.g., schema-drift repair gain of 0.913, clean success of 0.906) cannot be reproduced with the current benchmark code, and the metadata leak could inflate exact-match (E_t) rates across scenarios, including the negative controls. Please provide a sensitivity analysis, for example running the deterministic benchmark (or a subset of the live tasks) both with and without the metadata field, to show that the scenario-level pattern is unchanged. Absent such evidence, the paper should state clearly that the quantitative claims are specific to the pre-hardening pipeline and are not representative of the current benchmark version.","section":"2.3, 5, and 7"}],"minor_comments":[{"comment":"The retriever's query-derived domain heuristic is unresolved for 138 tasks and incorrect for 14 tasks. The paper does not report how domain-resolution status interacts with the scenario-level success rates, especially for the clean and schema-drift rows. Please provide a breakdown (e.g., success by domain-resolved vs. unresolved) or explicitly state that the retriever's failures are part of the system under test and are expected to affect all scenarios equally.","section":"4.2 / Table 3"},{"comment":"The definition of V_t ('every tool call satisfies its schema') is ambiguous when the model makes no tool call at all: is the schema requirement vacuously satisfied, or does the expected tool call count as a missing required call that violates V_t? Please clarify, as this affects the interpretation of the component rates in the negative-control scenarios.","section":"2.3"},{"comment":"The statement about the 28,080 prompt hashes appears only in the Reproducibility section. Because it materially affects the interpretation of Table 4, it should also be stated in Section 4.3 or 5, with a note that a large fraction of records never reached the model.","section":"6"},{"comment":"In the wrong-session repair row, the observed answer is '30 days' while the expected answer is '45 days'; the interpretation says the answer is grounded in text from the wrong session context. It would be helpful to know how often this failure shape (E_t true, R_t/G_t false) occurs across the 2,160 wrong-session records, to clarify the groundedness false-positive analysis in Table 5.","section":"4.4 / Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about several limitations, and the release of privacy-safe archives is commendable. However, the missing component-level rates and the undisclosed distribution of promptless records are load-bearing for the central layer-specific claim. The authors should be asked to either supply the missing analyses or substantially weaken the empirical claims. The benchmark framework itself is a useful contribution that does not require the current precise numbers to survive, but the reported zeros and repair gain need to be placed on firmer evidentiary ground."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know up front: this is a genuinely useful systems-evaluation artifact, not a conceptual breakthrough. It ships reproducible scored-record archives, a matched strict/repair protocol, and a result that matches intuition: schema normalization fixes schema-drift failures and does not fix stale, missing, unauthorized, or wrong-session failures. I read the paper a bit more charitably than the conditional verdict suggests; the core result holds.\n\nWhat is new: the matched fault matrix with negative controls is a real artifact. Nine models, nine scenarios, two contract modes, and 38,880 live records with privacy-safe archives and bootstrap code make the aggregates recomputable. The wrong-session groundedness false-positive rate (77.6% strict) is a sharp measurement. The paper is also careful and honest: it discloses the historical metadata-field caveat, the retriever's unresolved domain issue on 138 of 240 tasks, and the synthetic correlated task structure.\n\nThe main soft spot is the one the stress-test flags: layer isolation is asserted more than evidenced. Success is S = E ∧ R ∧ G ∧ V. The negative controls stay at 0.000 under repair, but the paper never reports the V component in those scenarios. Strictly, that zero is compatible with a scenario having both a schema-format fault and the intended evidence/state fault. I suspect by construction the scenarios are single-fault—stale index replaces the current document, permission denied filters it out—but that construction is not demonstrated with component rates, and the paper leans on Table 1 for the layer-specific reading. That is a real gap, but it does not overturn the conclusion. The clean-to-schema-drift contrast is strong enough.\n\nThe metadata leakage is handled honestly but still muddies the historical live numbers; the paper already says to treat them as end-to-end pipeline measurements, which is fair. The retriever's unresolved domain issue is disclosed before the live matrix, which makes the live results conservative rather than inflated.\n\nWho this is for: people building or evaluating agentic RAG systems, especially those who want to stop a narrow intervention from being sold as a universal reliability fix. It deserves peer review. I would send it out, with the request that the revision add V-component diagnostics for the negative-control scenarios and phrase 'exactly one layer fault' as a design property backed by artifact checks rather than a label.","headline":"A carefully built RAG reliability benchmark whose main result survives scrutiny; the layer-isolation claim needs a component-level diagnostic before the benchmark becomes a standard.","tokens_in":8182,"tokens_out":2631,"would_cite":true,"duration_ms":25509,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Answers in agentic retrieval-augmented generation can be grounded and still wrong: schema repair fixes only the tool-contract layer, leaving stale, missing, denied, and wrong-session evidence at zero success.","keywords":["agentic retrieval-augmented generation","RAG reliability","layer-specific evaluation","schema drift","negative controls","groundedness false positives","tool contracts","benchmark"],"falsifier":"Regenerate the 240 task rows from the same 80 policy units with a fresh random assignment of effective dates, permission flags, document versions, and session labels, keeping the fault definitions fixed, and rerun the strict/repair matrix. If schema repair shows nonzero gains on any non-contract scenario, or if stale-index and wrong-session strict success move above zero without their intended interventions, the layer-specific attribution fails. A page-level check would be to scan the published archives for any wrong-session record whose repair-mode raw result passes the scalar scorer; the paper reports none.","tokens_in":7243,"feed_emoji":"🧩","tokens_out":9973,"duration_ms":79913,"temperature":0.7,"pith_summary":"This paper argues that reliability failures in agentic retrieval-augmented generation (RAG) live on distinct layers—evidence freshness, tool contracts, authorization, completeness, and session context—and that evaluation should treat those layers separately rather than collapse them into one answer-quality score. To make the point testable, it introduces LayerRAG-Bench, a controlled benchmark with 8 enterprise domains, 240 tasks, 9 fault scenarios, 2 contract modes, and 38,880 live task-level records from nine models across three API providers. The central result is that schema normalization, a bounded repair of tool-output shape, raises schema-drift success from 0.000 to 0.913, while stale-index, missing-tool-output, permission-denied, and wrong-session scenarios remain at 0.000 in both strict and repair modes. The paper also shows groundedness alone is not a reliability signal: under wrong-session state, 77.6% of strict records were grounded yet failed. The upshot is that a reliability intervention should be credited for repairing its target layer without being mistaken for a universal fix.","feed_headline":"0% to 91%: schema repair fixes one RAG fault, leaves four at zero","feed_subtitle":"A 38,880-record benchmark shows grounded answers can still fail on stale, unauthorized, or wrong-session evidence.","key_machinery":"The load-bearing object is the matched strict-versus-repair contract condition, implemented as a small schema-normalization function. Strict mode rejects any tool output whose structure does not match the expected schema, so schema drift yields zero success even though the evidence is present. Repair mode applies bounded normalization that reshapes the payload into the expected contract without adding or reconstructing evidence, without bypassing permissions, and without refreshing index state. Crossing 9 models, 9 scenarios, 2 contract modes, and 240 task rows yields 162 cells and 38,880 paired records, and the paired structure lets the paper attribute the 91.3-point schema-drift gain to contract repair while the unchanged zero rows act as negative controls. A supporting deterministic retriever audit shows that validity reranking is the source of current-version preference and that explicit task-domain filtering is brittle to corrupted metadata, which sets the boundary of what query text alone can determine.","core_discovery":"The measured asymmetry in what contract repair can and cannot do is the discovery. LayerRAG-Bench injects exactly one non-adversarial operational fault per scenario into an enterprise-like corpus and evaluates 240 tasks across nine models under two contract modes: strict, which rejects tool outputs that do not match the expected schema, and repair, which applies bounded schema normalization. Schema drift is the repairable fault: strict success is 0.000, repair success is 0.913, with a hierarchical 95% interval of [0.854, 0.956] on the paired improvement and a 90.6-point specificity margin over the largest off-target change. Stale index, missing tool output, permission denial, and wrong session state stay at 0.000 in both modes because the missing conditions are not schema-format problems. Groundedness checks miss the distinction: 77.6% of strict wrong-session records and every grounded stale-index record are grounded yet unsuccessful. The paper concludes that layer-specific evaluation with negative controls is necessary: a mitigation should be verified to repair its own layer and not silently credited with others.","pith_inferences":["A natural extension is to inject compound faults, such as stale index plus schema drift, to test whether repairs compose, interfere, or require a shared provenance mechanism; the paper's one-fault-per-scenario design does not cover that case.","The reported retriever's 138 unresolved and 14 incorrect domain inferences suggest query-derived domain cues are a bottleneck; a provenance-aware retriever that binds each document to its session and effective date could be tested for whether it preserves the layer-specific repair pattern.","The paper's specificity margin suggests a general evaluation statistic: report a repair's gain on its target layer alongside its largest off-target gain, so a narrow intervention cannot masquerade as a universal fix.","Because wrong-session records were grounded but failed at high rates, a cheap production check would be to append a session-validity gate to grounding classifiers and measure how many false positives it removes; the paper does not test this."],"forward_implications":["Schema normalization should be deployed and evaluated only as a contract-layer repair; it cannot restore stale evidence, missing tool output, denied permissions, or wrong-session context.","Groundedness-only evaluation is not a reliability signal: a wrong-session answer can cite retrieved evidence correctly and still be operationally wrong.","Reliability benchmarks for agentic RAG should include negative controls so that a repair that works on one layer is not over-read as a general fix.","Stale evidence, missing tool output, permission denial, and wrong-session context each need their own mechanism—freshness-aware retrieval, retry or abstention, access-aware abstention, and session isolation.","The repetition of the failure pattern across all nine models means stronger base models alone do not obviate layer-specific system controls."],"supporting_citations":[{"why":"Introduces retrieval-augmented generation, the pipeline family whose reliability layers this benchmark targets.","marker":"Lewis et al., 2020"},{"why":"Surveys RAG variants and establishes the broader RAG evaluation context that the layer-specific approach complements.","marker":"Gao et al., 2023"},{"why":"Introduces the ReAct-style agent loop that produces the tool calls and session state whose faults are injected here.","marker":"Yao et al., 2023"},{"why":"Provides DPR, a retrieval baseline for relevance quality that the paper argues cannot capture layer failures.","marker":"Karpukhin et al., 2020"},{"why":"Provides BEIR, a ranking benchmark showing retrieval metrics do not distinguish evidence freshness, authorization, or session context.","marker":"Thakur et al., 2021"},{"why":"Provides RAGAS, an automated RAG evaluation whose answer-quality score can conflate distinct layer failures.","marker":"Es et al., 2023"},{"why":"Provides ARES, an automated evaluation framework whose faithfulness emphasis the wrong-session false positives challenge.","marker":"Saad-Falcon et al., 2023"},{"why":"Provides CRAG, a comprehensive RAG benchmark that treats answer correctness rather than fault-layer structure.","marker":"Yang et al., 2024"},{"why":"Provides Self-RAG, a reflection-based groundedness approach that the stale-index and wrong-session results show to be insufficient.","marker":"Asai et al., 2023"}],"fun_headline_variants":["Schema repair: 0 to 91% on one fault, four stay at zero","Grounded RAG answers can still be wrong: benchmark shows why","RAG reliability: one fault repairable, four not—benchmark proves it","Cross-layer RAG benchmark: schema drift fixed, stale evidence not","0% to 91% on one RAG fault, 0% on four others: layer-specific fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each benchmark scenario injects exactly the fault it names—schema drift only changes the tool contract, staleness only touches index freshness, and wrong session only alters context—so repair gains and zero-effect controls can be attributed to layers rather than to task features that co-vary across scenarios.","fun_headline_variants_meta":{"raw":{"variants":["Schema repair: 0 to 91% on one fault, four stay at zero","Grounded RAG answers can still be wrong: benchmark shows why","RAG reliability: one fault repairable, four not—benchmark proves it","Cross-layer RAG benchmark: schema drift fixed, stale evidence not","0% to 91% on one RAG fault, 0% on four others: layer-specific fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":3084,"prompt_tokens":931,"completion_tokens":2153,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2045}},"tokens_in":547,"tokens_out":2153,"duration_ms":13299,"temperature":1.0,"reasoning_tokens":2045,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:23:05.689879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the 240 task rows from the same 80 policy units with a fresh random assignment of effective dates, permission flags, document versions, and session labels, keeping the fault definitions fixed, and rerun the strict/repair matrix. If schema repair shows nonzero gains on any non-contract scenario, or if stale-index and wrong-session strict success move above zero without their intended interventions, the layer-specific attribution fails. A page-level check would be to scan the published archives for any wrong-session record whose repair-mode raw result passes the scalar scorer; the paper reports none.","supporting_citations":[{"cited_title":"Dense passage retrieval for open-domain question answering","cited_arxiv_id":null,"evidence_quote":"Introduces retrieval-augmented generation, the pipeline family whose reliability layers this benchmark targets."},{"cited_title":"Dense passage retrieval for open-domain question answering","cited_arxiv_id":null,"evidence_quote":"Provides DPR, a retrieval baseline for relevance quality that the paper argues cannot capture layer failures."}],"review_version":1}