{"id":"37bf814c-0837-4705-a77d-10030b887f0b","arxiv_id":"2606.06880","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RISE uses BM25 to bound interaction spaces for agentic search and pre-processes documents for shell navigation, matching direct corpus interaction accuracy at roughly one-quarter the cost on BrowseComp-Plus.","lead":"The paper proposes that retrieval for AI search agents should create bounded 'interaction spaces' in large corpora rather than just ranking documents for the LLM context. This framing could let agents use shell tools efficiently on manageable subsets without latency exploding as collections grow.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"BM25 boundary may exclude documents needed for task solution, breaking the interaction-space premise","rationale":"The reader's weakest_assumption is exactly the load-bearing precondition for the interaction-space claim; the experiments do not yet isolate or stress-test retrieval failures, so the concern stands.","tokens_in":1807,"tokens_out":277,"duration_ms":18584,"concrete_test":"Sample 30 BrowseComp-Plus queries whose gold documents include at least one with BM25 rank >200; run both RISE-BM25 and full DCI on the same gpt-5.4-mini setup; if RISE accuracy falls >15 points below DCI while DCI still succeeds, the boundary assumption is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (RISE matches pure-shell DCI accuracy at lower cost) requires that the single BM25-retrieved set supplies every document the agent must read or grep to reach the answer. If any gold document lies outside the top-k (low term overlap, long-tail entities, etc.), the agent has no path to it inside the bounded space, whereas unbounded DCI can still locate it. The reported 78 % match on BrowseComp-Plus therefore only shows sufficiency for the particular query distribution; it does not establish that the boundary construction is reliable in general.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes RISE (Retrieving Interaction SpacE) as a proof-of-concept approach to agentic search that uses BM25 to construct a bounded interaction space from the corpus; documents in this space are pre-processed for shell-style navigation tools. It claims that on BrowseComp-Plus, RISE matches the accuracy of unbounded direct corpus interaction (DCI) at 78% using gpt-5.4-mini while incurring roughly one quarter the per-query cost, and that at a 1M-document scale RISE-BM25 reaches 81% accuracy on gpt-5.4-mini whereas DCI on gpt-5.4-nano falls to 60% with frequent wall-clock failures.","tokens_in":1941,"tokens_out":509,"duration_ms":15925,"significance":"If the central empirical result holds, the work would be significant for demonstrating that retrieval can usefully bound the interaction space for agents rather than merely supplying a context window, thereby improving scalability of direct corpus interaction methods. The concrete accuracy and cost numbers at 1M scale constitute a useful data point for the emerging agentic-IR literature.","major_comments":[{"comment":"Abstract: the headline claim that RISE 'matches the pure-shell DCI baseline at 78% accuracy' is reported without error bars, statistical significance tests, or any description of how many documents are typically included in the BM25-constructed space or how the space boundary is exactly enforced; this leaves the central empirical comparison only weakly supported.","section":"Abstract"},{"comment":"Scaling experiment (1M-document setting): the comparison pits RISE-BM25 on gpt-5.4-mini (81%) against DCI on gpt-5.4-nano (60%); because model capability differs, the result does not isolate the contribution of the bounded interaction space and therefore does not yet establish the claimed scaling advantage.","section":"Scaling experiment"},{"comment":"The central claim that a single BM25-retrieved set supplies every document the agent must read or grep requires that no gold document lies outside the top-k; the manuscript provides no analysis or failure-case enumeration of queries where term overlap is low, leaving the reliability of the boundary assumption untested and load-bearing for the reported 78% match.","section":"Evaluation"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive comments on our manuscript. We address each major comment below and indicate the revisions we will make to strengthen the paper.","responses":[{"response":"We agree that additional details would strengthen the claim. In the revised manuscript, we will include a description of the BM25 interaction space construction, including typical sizes (top-200 documents in our experiments) and the enforcement mechanism (the agent is provided only with tools scoped to the retrieved set). Regarding error bars and significance, the main results are from single runs due to computational cost; we will add a note on variance from pilot runs and, if feasible, report results over 3 seeds for key comparisons. This addresses the support for the comparison.","revision_made":"partial","referee_comment":"[Abstract] Abstract: the headline claim that RISE 'matches the pure-shell DCI baseline at 78% accuracy' is reported without error bars, statistical significance tests, or any description of how many documents are typically included in the BM25-constructed space or how the space boundary is exactly enforced; this leaves the central empirical comparison only weakly supported."},{"response":"The referee correctly identifies that differing model sizes confound direct attribution to the interaction space. Our intent was to demonstrate practical scalability: RISE enables a more capable model to operate effectively at 1M scale, while DCI forces use of a weaker model with failures. However, to better isolate the effect, we will add a note clarifying this and, where possible, include a comparison using the same model (e.g., attempting DCI with mini at smaller scales or noting cost barriers). We will revise the text to avoid overstating the isolation of the bounded space contribution.","revision_made":"yes","referee_comment":"[Scaling experiment] Scaling experiment (1M-document setting): the comparison pits RISE-BM25 on gpt-5.4-mini (81%) against DCI on gpt-5.4-nano (60%); because model capability differs, the result does not isolate the contribution of the bounded interaction space and therefore does not yet establish the claimed scaling advantage."},{"response":"This is an important point. The 78% match to DCI implies that for the queries DCI could solve, the BM25 space included the necessary documents, but we did not explicitly verify recall of gold documents. In revision, we will add an analysis of the fraction of queries where gold documents fall outside the BM25 top-k, and enumerate or characterize cases with low term overlap (e.g., using query-document term statistics). This will test the boundary assumption more rigorously.","revision_made":"yes","referee_comment":"[Evaluation] The central claim that a single BM25-retrieved set supplies every document the agent must read or grep requires that no gold document lies outside the top-k; the manuscript provides no analysis or failure-case enumeration of queries where term overlap is low, leaving the reliability of the boundary assumption untested and load-bearing for the reported 78% match."}],"tokens_in":1531,"tokens_out":652,"duration_ms":21119,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core point is that retrieval for agents should create a bounded interaction space the model can explore with tools, rather than just handing over a few documents or letting the agent scan everything. RISE implements this by using BM25 for the boundary and pre-processing documents for shell-style navigation.\n\nThe paper does a clean job naming the scaling problem with pure direct corpus interaction and showing a workable alternative. On BrowseComp-Plus it matches the unbounded DCI baseline at 78% accuracy while cutting cost by roughly three-quarters with gpt-5.4-mini. At one million documents the same approach keeps accuracy near 81% while the baseline drops and starts timing out.\n\nThe main weakness is that everything hinges on the first-stage retriever never missing a document the agent will need. If a gold file has low term overlap with the query, it sits outside the space and the agent cannot reach it. The abstract gives no error bars, no breakdown by query difficulty, and no count of how large the typical space ends up being, so the 78% match only shows it worked for this particular test distribution.\n\nThe work is aimed at people building tool-using agents that must operate over growing corpora. Readers who care about deployment cost and latency will find the numbers and the framing useful even if they want tighter validation.\n\nIt deserves a serious referee. The scaling issue is real and the proposed fix is simple enough to test further; the authors should be asked for failure-case analysis and variance numbers, but the idea itself is worth the review time.","headline":"RISE gives a practical way to bound agentic search with BM25 so shell tools stay cheap, but the results rest on an untested assumption that the boundary always contains what the agent needs.","tokens_in":2420,"tokens_out":397,"would_cite":false,"duration_ms":14522,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Retrieval for agentic search should construct bounded interaction spaces that agents explore with tools instead of only ranking documents.","keywords":["agentic search","interaction space","retrieval","BM25","direct corpus interaction","shell tools","BrowseComp-Plus"],"falsifier":"A set of queries where the information required to reach the correct answer lies outside the BM25-retrieved interaction space, causing the agent to fail even when given unlimited exploration time inside the space.","tokens_in":2719,"feed_emoji":"🔍","tokens_out":688,"duration_ms":18590,"temperature":0.7,"pith_summary":"The paper claims that retrieval methods inherited from non-agentic search only select documents for an LLM context window, while direct corpus interaction via shell commands fails to scale because broad commands scan the entire corpus. It argues that retrieval's proper role is to define an interaction space: a bounded subset of the corpus that the agent can navigate with associated tools. RISE implements this idea by applying BM25 to set the space boundary and pre-processing the documents inside it for shell-style navigation during indexing. On the evaluated task this yields accuracy matching the unbounded baseline while cutting per-query cost substantially and avoiding the latency and failure spikes that appear as the corpus reaches one million documents.","feed_headline":"Retrieval builds bounded spaces for agents to explore corpora","feed_subtitle":"RISE matches full direct-interaction accuracy at quarter cost and scales to 1M documents without the failures seen in unbounded scans.","key_machinery":"RISE (Retrieving Interaction SpacE), which uses BM25 to supply the boundary of the interaction space and processes documents at indexing time to support shell-style navigation inside that space.","core_discovery":"The central claim is that an interaction space constructed by a first-stage retriever enables an agent to reach the same task accuracy as unbounded direct corpus interaction while incurring roughly one quarter the per-query cost and maintaining performance as corpus size grows to one million documents.","pith_inferences":["The same boundary-plus-navigation design could be tested on agent tasks that currently rely on repeated retrieval rounds.","Replacing BM25 with a stronger first-stage retriever might tighten the interaction space further without losing coverage.","The approach suggests a general pattern in which retrieval hands the agent a navigable subspace rather than a static list of passages."],"forward_implications":["On BrowseComp-Plus, RISE matches the pure-shell DCI baseline at 78 percent accuracy with gpt-5.4-mini at roughly one quarter of the per-query cost.","At one million documents, RISE-BM25 reaches 81 percent accuracy on gpt-5.4-mini while DCI on gpt-5.4-nano degrades to 60 percent with 33 of 100 wall-clock failures.","Pre-processing documents for shell-style navigation during indexing allows the agent to use grep and file reads without scanning the whole corpus on every query."],"fun_headline_variants":["Retriever builds bounded spaces for agent corpus interaction","Bounded spaces match full scan accuracy at quarter cost","RISE retrieves interaction spaces for efficient agent tools","Retrieval scales agent search via bounded interaction spaces"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"A standard first-stage retriever such as BM25 can reliably supply a boundary that still contains all information the agent needs to solve the task without the need for iterative re-retrieval or expansion of the space.","fun_headline_variants_meta":{"raw":{"variants":["Retriever builds bounded spaces for agent corpus interaction","Bounded spaces match full scan accuracy at quarter cost","RISE retrieves interaction spaces for efficient agent tools","Retrieval scales agent search via bounded interaction spaces"]},"model":"grok-4.3","cost_usd":0.00886,"raw_usage":{"total_tokens":3993,"prompt_tokens":683,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":88599500,"prompt_tokens_details":{"text_tokens":683,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3254,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":683,"tokens_out":56,"duration_ms":22440,"temperature":1.0,"reasoning_tokens":3254,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T21:04:41.696617+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A set of queries where the information required to reach the correct answer lies outside the BM25-retrieved interaction space, causing the agent to fail even when given unlimited exploration time inside the space.","supporting_citations":[],"review_version":1}