{"id":"3bb2944f-072f-4664-93b7-8c95b5c955c9","arxiv_id":"2607.15253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Static retrieval utility and counterfactual causal utility are nearly independent (rho = -0.026) in multi-step agentic search, with about a third of read documents being 'bridge' evidence that static metrics miss.","lead":"The paper shows that whether a document helps when read alone is almost unrelated to whether it helps a multi-step search agent: the two scores correlate at -0.026 across 23,322 observations. It defines 'bridge documents' that are useless to a static reader but causally essential to the agent, and traces their effect to discriminative entities carried into the next query.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Near-zero Spearman may be an artifact of a degenerate static-utility axis: only 3.30% of ΔSRU values are positive, and the proxy quadrant, which drops 58.84% of records, does not fully resolve this.","rationale":"The reader's weakest-assumption analysis identifies exactly the same load-bearing concern: the static-utility axis is near-degenerate, so the near-zero correlation may be a measurement artifact rather than a true independence result. The paper itself flags this in Section 3.6 and attempts to mitigate it with the proxy quadrant. However, the proxy quadrant does not fully settle the issue because it selects only 41.16% of records where BM25 and the cross-encoder agree, and it replaces reader-based SRU with a different operationalization. The proposed test directly addresses the original static-utility definition by strengthening the reader while keeping all other components fixed. This is a feasibility check, not an adversarial re-derivation. Given the authors' transparency and the suggestive proxy evidence, the concern does not justify moving the verdict beyond CONDITIONAL; it does, however, reinforce that the current CONDITIONAL verdict is appropriate until such a test is run and artifacts are released. No further concerns of comparable weight were identified: the CTU construction is reasonable, the confound in the turn-count breakdown is correctly withdrawn, and the OER propagation experiment, while secondary, is cleanly executed.","tokens_in":15458,"tokens_out":8281,"duration_ms":79465,"concrete_test":"Recompute SRU on the same 23,322 document observations (or a stratified sample of them) using a substantially stronger stateless reader, e.g., Qwen2.5-72B-Instruct or a reader prompted to reason step-by-step before giving an answer, while holding the agent, trajectories, and CTU computation fixed. If the new ΔSRU axis has a materially larger fraction of positive values (e.g., >10%) and the Spearman correlation with CTU remains within ±0.05, the concern is resolved. If the correlation becomes appreciably positive or negative, the near-zero result in Table 4 is a measurement artifact of reader weakness, and the central claim should be restricted to 'this particular weak reader.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim of near-independence between static utility and causal utility rests on Spearman ρ = −0.026 between ΔSRU and CTU over 23,322 observations. Table 2 shows that ΔSRU is zero for the median document and positive for only 3.30% of observations; the reader's F1 is 0.0170 with the document and 0.0122 without. Thus, 96.7% of observations are tied at zero on the static axis. Spearman computed on an almost degenerate variable does not establish that the latent construct 'static utility' is independent of CTU; it may only indicate that the measurement instrument cannot discriminate among documents under the given reader prompt. The proxy quadrant (Section 3.4) uses a more spread axis, but it requires BM25 and cross-encoder agreement, excluding 58.84% of records, and it measures raw retrieval scores rather than the reader-based static utility that the paper's definition (Section 2.6.1) targets. Since Section 3.6 acknowledges that the bridge percentage is a restatement of the marginals, the correlation itself is load-bearing. If that correlation is an artifact of a degenerate axis, the paper's headline conclusion becomes unsupported for the reader-defined notion of static utility.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether static retrieval utility (SRU), defined as the improvement a document gives a stateless reader, predicts the causal utility of that document for a multi-step ReAct-style search agent. Using 1,000 HotpotQA questions, the authors replay trajectories, counterfactually delete each read document, and compute a Counterfactual Trajectory Utility (CTU) from answer quality, next-query retrieval quality, and turn-count deltas. Crossing SRU with CTU over 23,322 document observations, they report near-zero Spearman correlation (rho = -0.026) and a bridge cell of 35.72% of documents that are causally load-bearing while appearing useless to a static reader. A robustness analysis replaces the reader-based SRU axis with BM25/cross-encoder agreement scores, yielding a bridge cell of 27.16% on 9,600 observations. A second experiment measures entity propagation and finds that entities with high Observable Entity Relevance (OER) appear in the agent's next query 4.02 times more often than low-OER entities. The paper explicitly reports limitations, including the skew of the SRU axis and the partial dependence of the bridge percentage on marginals.","tokens_in":15761,"tokens_out":5183,"duration_ms":47016,"significance":"If the central claim were fully established, this paper would be significant: it challenges the common assumption that static relevance metrics transfer to agentic retrieval, and it proposes a counterfactual methodology for measuring causal document utility. The OER propagation result is a concrete, falsifiable mechanism and the use of machine-checked counterfactual replay over a large observation set is a strength. However, the reader-based SRU axis is nearly degenerate (only 3.30% positive DeltaSRU; mean reader F1 0.017), so the near-zero correlation between SRU and CTU is consistent with measurement failure rather than true independence between the constructs. The proxy quadrant is a useful check but changes the construct and excludes 58.84% of records. The manuscript is unusually candid about its limitations, but the headline claim as stated is not fully supported by the current evidence.","major_comments":[{"comment":"The central independence claim relies on Spearman rho = -0.026 between DeltaSRU and CTU, but Table 2 shows that DeltaSRU is zero for the median observation and positive for only 3.30% of the 23,322 observations. A rank correlation computed on an almost degenerate variable does not establish that the latent construct 'static utility' is independent of CTU; it may just reflect the reader's inability to discriminate among documents. The paper acknowledges the skew but does not supply a decisive fix, e.g., restricting to the 3.30% non-zero DeltaSRU observations, using a stronger reader with measurable dynamic range, or showing that the correlation remains near zero after such a restriction. Without this, the reader-based result does not support the paper's title claim.","section":"Section 3.3, Table 4; Section 3.6"},{"comment":"The proxy quadrant replaces the SRU axis with an agreement-based BM25/cross-encoder axis and excludes 13,722 of 23,322 records (58.84%). This is a different construct: it measures raw retriever scores, not the reader-based static utility defined in Section 2.6.1. It also selects the subpopulation where the two retrieval signals agree, which may have different dependence with CTU. The proxy result can support a claim about 'static retrieval scores do not predict causal utility,' but it does not rescue the SRU-specific claim. The paper should either present the proxy as the primary evidence and reframe the abstract/title accordingly, or improve the SRU reader to obtain a non-degenerate static utility axis.","section":"Section 3.4, Table 5"},{"comment":"The stratified sample is designed to approximate 70% bridge and 30% comparison questions, but the HotpotQA development set contains 5,918 bridge and 1,487 comparison questions (about 80/20). The analyses in Section 3 are unweighted, so the 23,322 document observations overrepresent comparison questions by about 50%. This can bias the reported marginals, the bridge cell percentage, and the correlation. Please either apply survey weights to the analysis or justify that the 70/30 split does not materially affect the conclusions.","section":"Section 2.4, Section 3.1"},{"comment":"The CTU composite uses unit weights and min-max normalization over the intervention set. The 'zero effect point' at CTU=1.4 is derived from the observed raw ranges [-1,1], [-1,1], [-2,3]. The claim that values above 1.4 mean the document was 'on balance, helping' assumes that a raw delta of zero on all components is the only way to achieve CTU=1.4, and that the three normalized components are commensurable with equal weights. This is a modeling choice; the paper should include a sensitivity analysis (e.g., alternative weights, or using only the sign of each delta) to show that the quadrant results are not an artifact of this particular aggregation.","section":"Section 2.6.2, Eq. (10)-(11)"}],"minor_comments":[{"comment":"The sentence 'Over all records rho = -0.0161 with p = 0.116' is ambiguous because the proxy is only defined on the 9,600 records where BM25 and cross-encoder agree. Clarify whether this is over the 9,600 or over all 23,322 with some imputation.","section":"Section 3.4"},{"comment":"The normalization in Eq. (10) uses observed min/max ranges; state explicitly that these are observed ranges and that the zero-effect point depends on them.","section":"Section 2.6.2, Eq. (11)"},{"comment":"The note that 4,115 observations carry no cell assignment is important, but it would help to state what those observations are (e.g., documents without CTU or with missing quadrant assignments) and whether they are excluded from all cell-level analyses or only this table.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent and the counterfactual replay methodology is a strong contribution, but the headline claim is currently overreaching relative to the evidence. The near-degenerate SRU axis is an acknowledged weakness, and the proxy quadrant, while suggestive, does not test the SRU construct. I would ask the authors to either significantly improve the static utility measurement or explicitly reframe the paper's claim to static retrieval signals rather than SRU specifically. The stratified sampling concern is also worth raising in the revision. The OER propagation experiment is well designed and reported, including the honest reporting of the failed H2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2607.15253. The paper does something genuinely useful: it measures, rather than asserts, the gap between static document utility and causal utility in a multi-step agentic search. Using a ReAct-style Qwen agent on HotpotQA, they replay 1000 questions, delete each document the agent read one at a time, and compare the original trajectory against the counterfactual. That CTE (Counterfactual Trajectory Exploration) framework is new to me, and the CTU score with its zero-effect threshold at 1.4 is a clean construction. The bridge evidence concept—documents that look useless to a stateless reader but are causally load-bearing—is intuitive and the paper gives it numbers: 35.7% of read documents fall in that cell. The robustness check with BM25+cross-encoder proxies still shows a 27.2% bridge cell, and the OER propagation experiment (discriminative entities appear in next queries 4x more often) is a nice mechanism test. H2 fails and they report it as inconclusive; the turn-count confound is discovered and withdrawn. That kind of honesty is rare.\n\nNow the soft spots. The headline correlation of rho=-0.026 between DeltaSRU and CTU is the load-bearing number, but the SRU axis is nearly degenerate: only 3.30% of documents have positive DeltaSRU, and the reader's F1 is 0.017 with a document. Spearman on a variable that is 96.7% tied at zero tells you almost nothing about whether a well-behaved static utility measure would be independent of CTU. The paper acknowledges this and the proxy quadrant is a good attempt, but that quadrant only uses records where BM25 and cross-encoder agree—58.84% of the data is discarded. So the strong claim \"static utility does not predict causal utility\" is supported only for a narrow segment of the population, and for the reader-based definition it's not really supported at all because the measure can't discriminate. The bridge percentage is also largely a restatement of the marginals, as they admit. None of this kills the paper, but it shifts the result from \"independence established\" to \"we observe no correlation in this setup, with caveats.\"\n\nBottom line: this deserves a serious referee. The framework and the empirical honesty are valuable, and the OER mechanism result is solid. But the central independence claim needs either a better static axis or a full-population proxy before it can carry the weight the abstract puts on it. I'd send it to review, and recommend the authors release their code and data—reproducibility is currently limited. Worth a reading group discussion.","headline":"Honest and clever counterfactual study of agentic retrieval, but the headline independence claim is weakened by a near-degenerate static utility axis.","tokens_in":16270,"tokens_out":3532,"would_cite":true,"duration_ms":30023,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Static utility and causal utility are nearly independent in agentic search","keywords":["agentic retrieval","counterfactual evaluation","static utility","causal utility","bridge evidence","entity relevance","multi-hop question answering","retrieval evaluation"],"falsifier":"Rerun the quadrant experiment with a strongly calibrated static reader whose absolute answer F1 is far above 0.017, so the static axis is not near-degenerate; if the Spearman correlation with CTU rises substantially above zero and the bridge cell collapses, the reported independence would be a measurement artifact rather than a true gap.","tokens_in":15321,"feed_emoji":"🔍","tokens_out":4956,"duration_ms":37086,"temperature":0.7,"pith_summary":"Retrieval systems are built on the assumption that a document is useful if a reader can answer the question better with it. This paper tests that assumption when a language model acts as a search agent, issuing multiple queries and reasoning across turns. By deleting each document the agent read and replaying the rest of the trajectory, the authors compute a counterfactual measure of causal utility and cross it against static utility over 23,322 observations. They find the two are close to statistically independent, and that roughly a third of documents are 'bridge evidence': causally load-bearing yet scoring near zero on static measures. The paper also identifies a mechanism — these documents hand the agent a discriminative entity that redirects the next query — and shows that entity observability predicts propagation by a factor of four.","feed_headline":"Static utility does not predict causal utility in agentic search","feed_subtitle":"Counterfactual replays show a third of agent-read documents are invisible to standard retrieval scores.","key_machinery":"The load-bearing instrument is the omission intervention: delete one document from the ranked list at one step and replay the rest of the trajectory with everything else held fixed. The resulting Counterfactual Trajectory Utility (CTU) combines deltas in final answer F1, next-query nDCG@10, and turn count, normalized and summed with unit weights; its zero-effect threshold is analytically 1.4, which coincides with the empirical median. Static RAG Utility (SRU) is the improvement a document gives to a stateless reader over a parametric baseline. Crossing SRU against CTU produces the quadrant table, and the Observable Entity Relevance (OER) measure from prior work — whether an entity's presence","core_discovery":"The central claim is that static relevance and causal usefulness are different quantities in agentic retrieval, and optimizing the first does not deliver the second. The authors demonstrate this by replaying trajectories with one document removed and measuring whether the final answer, the next query's retrieval quality, or the number of turns changes. Over 23,322 observations the Spearman correlation between static reader utility and counterfactual trajectory utility is -0.026, and 35.72% of documents fall in the bridge cell: high causal utility, low static utility. The pattern persists when the static axis is replaced by a ranking-score proxy, giving a 27.16% bridge cell. A separate experi","pith_inferences":["If static and causal utility are independent, then conventional evaluation metrics such as nDCG and MRR may systematically mis-rank systems for agentic use; a testable extension is to re-rank existing systems by CTU and compare agent outcomes.","The result suggests a new class of 'stepping-stone' evidence: retrieval for agents may need to optimize for what a document enables next, not what it states; a concrete design would train a ranker to maximize expected OER of propagated entities.","The paper's CTU is measured for one agent; an open, testable question is whether the bridge phenomenon generalizes across backbones and prompts — if different agents need different bridge documents, then agent-specific evidence selection becomes a first-class problem.","Because the bridge cell percentage largely restates the independence result, the paper's strongest standalone evidence is the propagation ratio; future work could test the mechanism directly by ablating discriminative entities from candidate sets."],"forward_implications":["A retrieval system that perfectly optimizes static utility carries almost no information about which documents an agent actually needed.","The bridge cell persists when the static axis is a BM25/cross-encoder proxy, so the mismatch is not an artifact of a weak reader.","Bridge documents work by supplying discriminative entities; entity observability, not topical relevance, predicts what gets propagated into the agent's next query.","Counterfactual replay is too expensive for training loops, so the paper points to Observable Entity Relevance as a computable proxy for the causal signal.","Documents that contain the answer outright can be causally inert, while documents containing no answer can be causally essential."],"fun_headline_variants":["Static relevance misses a third of causally vital docs","Bridge documents: invisible to static ranking, key to agent success","Agentic search: causal utility ≠ static utility (ρ=-0.03)","Why static retrieval scores fail for agents: bridge docs","Counterfactuals reveal static scores ignore load-bearing docs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The static-utility axis must be a valid measurement of what current retrieval signals optimize; with only 3.30% of documents scoring above zero, the near-zero correlation may largely reflect a weak reader failing on both sides rather than true independence between static and causal utility.","fun_headline_variants_meta":{"raw":{"variants":["Static relevance misses a third of causally vital docs","Bridge documents: invisible to static ranking, key to agent success","Agentic search: causal utility ≠ static utility (ρ=-0.03)","Why static retrieval scores fail for agents: bridge docs","Counterfactuals reveal static scores ignore load-bearing docs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000707,"raw_usage":{"total_tokens":3101,"prompt_tokens":901,"completion_tokens":2200,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":2130}},"tokens_in":645,"tokens_out":2200,"duration_ms":14236,"temperature":1.0,"reasoning_tokens":2130,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:42:51.952056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the quadrant experiment with a strongly calibrated static reader whose absolute answer F1 is far above 0.017, so the static axis is not near-degenerate; if the Spearman correlation with CTU rises substantially above zero and the bridge cell collapses, the reported independence would be a measurement artifact rather than a true gap.","supporting_citations":[],"review_version":1}