{"id":"56dbc8b2-78b9-43e5-bc47-ebcd57a020b4","arxiv_id":"2608.11386","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Tool architecture alone, with capabilities held similar, changes coding-agent consistency, exploration, and token efficiency across three models and 11,700 trajectories.","lead":"This paper tests whether the way coding-agent tools are packaged and presented changes how well the agents work, even when the underlying abilities are the same. It finds that interface design alone shifts consistency, how widely agents search, and how much they cost to run.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NLSearch confounds architecture with extra LLM sub-agent capability: exploration gains may reflect added semantic search compute, not interface design, so the capability-equivalence premise in §2.1 is not secured.","rationale":"The reader's weakest_assumption is essentially the same: capability equivalence. I agree, but sharpen it to NLSearch. The Python concern is mitigated by the Appendix B.5 audit; Atomic and the cognitive scaffolds are more plausibly capability-preserving. NLSearch introduces an LLM subagent, which is an additional computational component, and no audit demonstrates equivalence. The paper's exploration claim is one of four headline findings, so the central causal claim is conditional on resolving this confound. Statistical uncertainty is a secondary issue: pass^k with n=10 yields coarse per-instance values and no confidence intervals are reported, but the 30-repeat robustness check in Appendix B.2 partly mitigates this for two actors. The capability-equivalence concern is more fundamental because it bears directly on the paper's stated premise. I therefore keep the CONDITIONAL verdict, with the specific condition that NLSearch's subagent be ablated or audited as described.","tokens_in":25868,"tokens_out":6367,"duration_ms":61622,"concrete_test":"Ablate NLSearch on the same 65-instance subset and three actors with three variants: (1) the published NLSearch subagent; (2) a deterministic keyword-to-grep NLSearch that extracts keywords from the natural-language query using a fixed rule (no LLM) and returns raw grep matches; (3) BashOnly plus a search tool that applies the same deterministic keyword extraction and grep. Compare read diversity and relevant-file recall (Tables 7 and 13). If (2) still shows gains over (3), the NL interface drives the effect; if only (1) shows gains, the effect depends on the LLM subagent's extra reasoning, so capability is not held constant. Also run a manual audit of 100 NLSearch subagent behaviors, analogous to Appendix B.5 Table 11, to verify that the subagent performs only operations a BashOnly actor could perform in a single grep call.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing assumption is that the six setups hold tool capability fixed, stated in §2.1 ('we design these setups to keep their capabilities as similar as possible'). The NLSearch setup violates this premise. §2.1 implements NLSearch as 'a subagent with the same actor model, which takes a natural-language query and uses bash commands such as grep to iteratively search for and return potentially relevant snippets.' This gives the NLSearch actor a second LLM with a multi-turn reasoning loop that BashOnly actors do not have. That is not just a different way to expose grep; it is an additional semantic-retrieval capability: an LLM subagent can interpret the query, disambiguate terms, follow up on partial results, and summarize snippets without consuming the actor's main context. The paper argues only that NLSearch avoids 'semantic indexing or embedding-based retrieval,' but the subagent itself is a learned semantic search component whose extra inference is a real capability difference. No equivalent of the Appendix B.5 Python-action audit (97/100 mapped to BashOnly) is provided for the subagent's search behavior. Consequently, the observed increase in read diversity and relevant-file recall in §3.3 and Table 7 could be caused by the additional LLM search compute rather than by the natural-language architecture. The causal attribution to architecture alone is therefore not secured for the exploration finding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how the organization and exposure of tool capabilities ('tool architecture') affect coding-agent behavior while attempting to hold underlying capabilities fixed. Six architectures are compared on repository-level issue fixing: BashOnly as baseline; Atomic (structured low-level tools); NLSearch (natural-language search via a subagent); Python (CodeAct-style executable code); and two lightweight cognitive scaffolds, HypoTrack and Scratchpad. Using 65 SWE-bench Live instances, three actor models, and 10 rollouts per actor–setup pair (11,700 trajectories total), the paper reports that Atomic improves repeated-attempt consistency (pass^k) across all three actors, NLSearch increases read diversity and relevant-file recall, Python achieves similar resolve rates with fewer steps and lower token cost, and the cognitive scaffolds have little effect. A smaller extension study on additional SWE-bench tasks is reported in Appendix B.6.","tokens_in":26121,"tokens_out":5973,"duration_ms":55449,"significance":"If the central isolation claim holds, this is a valuable empirical contribution: it provides one of the first large-scale, multi-actor controlled comparisons of tool-interface architecture per se, and it makes the effects concrete through repeated-rollout consistency, exploration, and efficiency metrics. The study is unusually careful in several respects: the Python-to-bash capability mapping is manually audited (97/100 actions, Table 11); the relevance proxy is human-validated (Table 6); and the main consistency result is checked with 30 repeats for the two open-weight actors (Table 4). The paper also states and tests falsifiable predictions (e.g., that Atomic reduces low-level interaction errors, that NLSearch increases early-search diversity). The principal weakness is that the capability-equivalence premise is not actually achieved for NLSearch, and the headline deltas are reported without uncertainty quantification; both issues bear directly on the paper's causal claims.","major_comments":[{"comment":"The capability-equivalence premise stated in §2.1 (“we design these setups to keep their capabilities as similar as possible”) is not secured for NLSearch. The implementation is “a subagent with the same actor model, which takes a natural-language query and uses bash commands such as grep to iteratively search for and return potentially relevant snippets.” This gives the NLSearch agent an additional LLM-backed, multi-turn retrieval component whose search reasoning is executed outside the actor's context; BashOnly actors have no comparable resource. Because the abstract and Finding 2 attribute the >11% relevant-file recall gain and the read-diversity gains to natural-language interface architecture rather than to added retrieval compute, this is a load-bearing confound. Please add a control condition that provides the BashOnly actor with a similarly capable search subagent under a structured or non-NL interface, or re-implement NLSearch without an LLM subagent (e.g., a deterministic NL-to-grep mapper), and re-run the exploration analysis.","section":"§2.1 (NLSearch); §3.3; Table 7"},{"comment":"The headline consistency and exploration comparisons are reported without confidence intervals or significance tests. In Table 2, the Atomic gains for Kimi-K2.5 are +0.013–0.014 and for Sonnet-4.5 +0.017–0.031 on absolute pass^k values around 0.25–0.31; Fig. 3 shows read-diversity changes of +13% to +28% without error bars. With 65 instances and 10 rollouts per cell, these differences may be within sampling noise. Table 4 provides a 30-repeat robustness check only for the two open-weight actors, not for Sonnet-4.5. Please add per-instance paired bootstrap confidence intervals (or an equivalent test) for the pass^k deltas and read-diversity deltas, and state which claimed effects survive once uncertainty is accounted for.","section":"§3.2 (Table 2); §3.3 (Fig. 3)"}],"minor_comments":[{"comment":"The abstract's “56.3% lower token usage” should specify that the large reduction is in input tokens; Table 8 shows output and observation tokens are not uniformly lower under Python, so “token usage” is ambiguous.","section":"Abstract; §3.4"},{"comment":"The definition of “high-relevant” files (“read by at least three successful trajectories from every setup”) is ambiguous about whether the count is pooled across setups or required within each setup; please clarify, since this affects how the recall comparison across setups should be interpreted.","section":"Appendix B.4, Table 7"},{"comment":"The human validation of the relevance proxy is described as LLM-assisted, but the number of annotators and inter-annotator agreement are not reported; adding these would strengthen the claim that the proxy is independent of the compared setups.","section":"Appendix B.4, Table 6"},{"comment":"The y-axis label “% Difference from BashOnly” is easy to misread as percentage points; please clarify in the caption and add error bars or confidence intervals so the reader can judge the stability of the 13–28% values.","section":"§3.3, Fig. 3"},{"comment":"The sampling procedure for the 65-instance subset (“randomly sampling 25 of its 100 repositories... yielding 65 problem instances”) should state the random seed or a reproducible sampling protocol so that the exact instance set can be reconstructed.","section":"§2.2"},{"comment":"Please state explicitly that the reported pass^k values are averages over instances, since the formal definition in Appendix A is per-instance and the aggregation step is only described in the text.","section":"Table 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the NLSearch confound: as written, the exploration finding is not a pure test of interface architecture because the NLSearch condition adds an LLM subagent. If the authors add a control that matches the extra retrieval compute under a different interface, or re-implement NLSearch without the subagent, the paper would be much stronger. The lack of confidence intervals is also notable given how small some of the headline deltas are; the 30-repeat robustness table helps for two actors, but the Sonnet results are unprotected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This paper does something genuinely useful: it isolates tool interface choices in coding agents with a large controlled corpus (11,700 trajectories, three actors, six setups). The core design—holding underlying actions and information fixed while varying how they're exposed—is careful, and the consistency and efficiency findings look credible. The Atomic result (structured primitives improve pass^k, especially for the weakest actor) is backed by an error analysis and a 30-repeat robustness check. The Python efficiency result is also convincing; the gain comes from fewer steps and the capability-equivalence audit, though limited to 100 actions, supports the mapping.\n\nThe soft spot is the NLSearch exploration claim, and the stress-test note is right. NLSearch is not capability-equivalent to BashOnly: it adds a second LLM subagent that can interpret natural-language queries, iterate with grep, and summarize snippets. That's an extra semantic-search capability, not just a different interface. Saying no embeddings are used does not remove the confound. So the 11% relevant-file recall gain and the read-diversity increase may reflect extra subagent compute rather than architecture. This matters because exploration is one of the three headline findings. The authors should either control for this (e.g., give BashOnly a similar subagent that searches with keywords rather than NL, or measure the subagent's own compute) or reframe the finding as 'adding an LLM-based search tool improves exploration,' which is less novel.\n\nOther issues are smaller. No significance tests or confidence intervals on the pass^k deltas; the stronger actors' gains are numerically small (Kimi +0.014 pass^5), though the 30-repeat check helps for Atomic. The Python action audit is a spot check, not a guarantee. These are minor relative to the core contribution.\n\nOverall, this deserves a serious referee. It's a well-organized empirical paper that gives coding-agent builders evidence they don't currently have. I'd send it to review, but ask the authors to address the NLSearch confound before it prints. For a reader, the consistency and efficiency sections are the useful parts; treat the exploration result as directional.","headline":"A genuinely controlled study of coding-agent tool interfaces with credible consistency and efficiency results; the exploration claim, however, is confounded by NLSearch's extra LLM subagent.","tokens_in":26653,"tokens_out":3308,"would_cite":true,"duration_ms":30804,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tool architecture alone — how capabilities are organized and exposed to a coding agent — shifts consistency, exploration, and efficiency even when the underlying information and actions are kept equivalent.","keywords":["tool architecture","coding agents","agent consistency","repository exploration","CodeAct","natural-language search","agent efficiency","cognitive scaffolding"],"falsifier":"A decisive test is to log the exact effect of every tool call in the Atomic, NLSearch, and Python setups and replay the same effect through BashOnly with identical output text; if $\\text{pass}^k$, read-diversity, and step-count gaps disappear when the underlying effects are identical, the architecture explanation is wrong. A simpler version replaces the NLSearch subagent with a deterministic grep wrapper and checks whether the 11% relevant-file recall gain survives.","tokens_in":25701,"feed_emoji":"🔧","tokens_out":9716,"duration_ms":81142,"temperature":0.7,"pith_summary":"This paper asks whether the interface through which a coding agent reaches its tools changes behavior once the underlying information and actions are held fixed. It answers yes: comparing six capability-matched tool architectures across three actor models and 11,700 trajectories on repository-level issue fixing, the authors find reliable differences in consistency, exploration, and efficiency. Structured low-level tools improve repeated-attempt consistency by up to 4.7×; a natural-language search interface broadens repository exploration and raises access to relevant files by more than 11%; and a Python code-execution interface keeps task performance roughly equal while using 41.6% fewer steps and 56.3% fewer tokens. Lightweight text-based cognitive scaffolding has little effect. The paper's broader point is that tool architecture is a design variable distinct from tool capability, worth measuring in its own right.","feed_headline":"Interface design changes how coding agents behave","feed_subtitle":"Structured tools lift repeat success up to 4.7x; code-style interfaces cut steps by 41.6%.","key_machinery":"The machinery is a controlled taxonomy of six tool architectures built on two axes: level of abstraction (raw bash; Atomic low-level primitives; NLSearch natural-language retrieval; Python executable-code blocks) and cognitive scaffolding (HypoTrack hypothesis records; Scratchpad free-form notes). To make architecture the only variable, each setup is deliberately implemented so that its actions can be reproduced in BashOnly: NLSearch uses no embeddings and runs grep-style commands through a subagent, atomic tools map onto documented shell equivalents, and 97 of 100 sampled Python actions map to bash-only operations. The argument then runs through trajectory-level measurements on repeated rollouts: $\\text{pass}^k$ for consistency, Jaccard distance over read files and CodeBLEU distance over patches for exploration, and steps plus input/output tokens for efficiency. The repeated-rollout protocol is what lets the paper separate architecture effects from aggregate task performance.","core_discovery":"The central discovery is that the same repository information and the same action capabilities, delivered through different interfaces, produce reproducible differences in how coding agents behave. The authors report that Atomic, a small set of constrained tools for search, viewing, targeted string replacement, and file creation layered on bash, is the only architecture that improves $\\text{pass}^k$ consistency for all three actor models; for the weakest actor the gain is 0.059–0.074 across $\\text{pass}^5$ to $\\text{pass}^9$, and the mechanism they identify is a sharp drop in malformed commands and broken edits. NLSearch, a natural-language query interface implemented as a subagent over grep-style search without embeddings, is the only architecture that raises file-read diversity for all three actors, increasing access to high-relevance files by more than 11% while also lowering precision. Python, a CodeAct-style interface in which the agent writes executable code blocks instead of issuing tool calls, achieves similar task performance to BashOnly with 41.6% fewer steps and 56.3% lower token usage, mainly because actors bundle several operations into each interaction. HypoTrack and Scratchpad, lightweight text tools for recording hypotheses and reasoning, do not change actor behavior in this setting, because actors mostly restate reasoning they already produce.","pith_inferences":["Editorial inference: because the Atomic consistency gain tracks interaction-error rates, the same effect should appear for any tool-using agent whose failures are dominated by command-syntax mistakes; a cheap test is to repackage a general-purpose shell as constrained primitives in a non-coding agent benchmark and compare rerun consistency.","Editorial inference: if NLSearch's exploration gain comes from query diversity rather than from embeddings, then a cheaper interface that diversifies the agent's own search formulations could reproduce part of the relevant-file recall increase without a subagent.","Editorial inference: the paper's matched-capability recipe — require every non-baseline tool to have a documented baseline equivalent, then spot-check the mapping — is itself a reusable test for deciding whether any agent-tool improvement is caused by architecture or by added capability."],"forward_implications":["If the results hold, coding-agent builders can improve repeated-run reliability by exporting common shell actions as constrained tools, with the largest gains for weaker models whose failures are dominated by malformed commands and broken edits.","A natural-language search interface is a reliable way to broaden repository coverage: it raises recall of relevant files for every actor tested, though deployers should expect lower precision and little change in final patch diversity.","CodeAct-style interfaces offer an efficiency win without a resolve-rate penalty: the same task is solved in fewer steps and with lower cumulative input cost across all three actors.","Lightweight text-based scratchpads and hypothesis trackers are not sufficient to change reasoning behavior on their own, since actors tend to project existing reasoning into them.","Overall resolve rate stays broadly similar across architectures, so the effects of tool architecture appear mainly in non-functional properties rather than in simple success rates."],"supporting_citations":[{"why":"Provides the SWE-Agent design whose structured low-level primitives motivate the Atomic setup and whose agent-computer-interface framing grounds the paper's notion of tool architecture.","marker":"Yang et al., 2024"},{"why":"Supplies SWE-bench Live, the contamination-controlled benchmark from which the main 65-instance evaluation subset is sampled.","marker":"Zhang et al., 2025"},{"why":"Supplies SWE-bench and SWE-bench Verified, used as the generalization corpus for issue-resolving tasks and as the origin of the bash-style coding-agent evaluation tradition.","marker":"Jimenez et al., 2023"},{"why":"Motivates the Python/CodeAct setup and the executable-code interaction style whose step and token efficiency is a central finding.","marker":"Wang et al., 2024b"},{"why":"Supplies the pass^k repeated-attempt consistency framing and notation used to measure reliability across rollouts.","marker":"Yao et al., 2024"},{"why":"Provides the fault-error-failure taxonomy the paper uses to explain Atomic's consistency gains as reductions in low-level interaction errors.","marker":"Avizienis et al., 2004"},{"why":"Supplies CodeBLEU, the code-similarity measure used to compute solution diversity across repeated attempts.","marker":"Ren et al., 2020"},{"why":"Supplies the Jaccard distance used to measure file-read diversity across repeated attempts.","marker":"Jaccard, 1901"},{"why":"Motivates the exploration dimension by arguing that coding agents often under-explore repository-level context.","marker":"Li et al., 2026"}],"fun_headline_variants":["Tool design reshapes coding agent behavior","Structured tools lift agent consistency up to 4.7x","Code interfaces cut agent steps by 41.6%","Same tools, new interfaces: coding agents change","How tool architecture steers coding agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The case rests on the assumption that the six tool setups give the agent the same information and the same actions, so that any behavioral change is caused by the interface rather than by a hidden difference in capability.","fun_headline_variants_meta":{"raw":{"variants":["Tool design reshapes coding agent behavior","Structured tools lift agent consistency up to 4.7x","Code interfaces cut agent steps by 41.6%","Same tools, new interfaces: coding agents change","How tool architecture steers coding agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1488,"prompt_tokens":1044,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":371}},"tokens_in":660,"tokens_out":444,"duration_ms":19363,"temperature":1.0,"reasoning_tokens":371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:02.504859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to log the exact effect of every tool call in the Atomic, NLSearch, and Python setups and replay the same effect through BashOnly with identical output text; if $\\text{pass}^k$, read-diversity, and step-count gaps disappear when the underlying effects are identical, the architecture explanation is wrong. A simpler version replaces the NLSearch subagent with a deterministic grep wrapper and checks whether the 11% relevant-file recall gain survives.","supporting_citations":[{"cited_title":"Basic concepts and taxonomy of dependable and secure computing","cited_arxiv_id":null,"evidence_quote":"Provides the fault-error-failure taxonomy the paper uses to explain Atomic's consistency gains as reductions in low-level interaction errors."},{"cited_title":"\\'E tude comparative de la distribution florale dans une portion des alpes et des jura","cited_arxiv_id":null,"evidence_quote":"Supplies the Jaccard distance used to measure file-read diversity across repeated attempts."}],"review_version":1}