{"id":"2834c90c-c006-404f-9bee-7aaffa217ecd","arxiv_id":"2607.02873","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"For a fixed DeepSeek model, the MCP client alone produced a 2.1× solve-rate gap on HexStrike-AI CTF trials, and bundled tool/behavior fixes lifted overall success from 55.4% to 72.0%.","lead":"An empirical study of HexStrike-AI finds that the client driving an LLM security agent can matter as much as the model: two DeepSeek clients differed by 2.1× on the same 86 picoCTF challenges. Targeted tool and behavior fixes raised overall solve rate from 55.4% to 72.0%, with residual failures mostly reasoning- or environment-bound.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"The client-gap claim is cleanly measured for DeepSeek but rests on an untested assumption that the 2.1× harness effect is not an artifact of 5ire’s soft constraint leakage and default sampling.","rationale":"The Reader correctly identifies same-set design and one-directional re-runs as the main threat to the intervention gains and residual taxonomy; those threats are real and already disclosed in §6.1. They do not, however, undercut the baseline client contrast that the Reader (and the paper) treat as the strongest claim. That claim is statistically supported on the reported numbers, yet §4.6 supplies direct evidence that constraint adherence is configuration-dependent and heaviest on the weaker client. Because the 2.1× figure aggregates across all three regimes, leakage can inflate the gap without any difference in how the two clients orchestrate HexStrike tools. Isolating Exp 1 (or hard-disabling natives) is a cheap, decisive check that the paper does not report. If the gap survives, the client-as-first-order-factor result is strengthened; if it collapses, the methodological implication (“a model’s solve rate is meaningful only when the client is named”) still holds but the magnitude and mechanism need revision. This keeps the verdict CONDITIONAL with medium correctness risk, only shifting emphasis from the intervention bundle to the cleanliness of the primary isolation. No stronger objection is warranted: the authors already label multi-model generality a hypothesis, disclose the soft constraint, and supply a stability sub-study.","tokens_in":12482,"tokens_out":852,"duration_ms":8772,"concrete_test":"Recompute the DeepSeek RooCode-vs-5ire baseline rates on Exp 1 only (free-solve, no tool-access instruction) and, separately, after hard-disabling native tools at the client or proxy layer rather than by prompt. If either contrast falls below ~1.3× or loses significance, the published 2.1× client effect is partly an artifact of soft-constraint leakage rather than harness quality.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper’s strongest claim (Abstract; §4.1, §5.1) is that, with DeepSeek fixed, the client is a first-order factor: RooCode 59.7% vs 5ire 27.9% (2.1×, two-proportion z p≈3×10⁻¹³). That contrast is the cleanest isolation in the design. Its load-bearing vulnerability is not the same-set intervention the Reader flags (which mainly weakens the secondary 55.4%→72.0% and residual-taxonomy claims), but whether the baseline gap itself is driven by genuine harness differences (agent loop, tool-call repair, context/retry policy) versus 5ire’s documented failure to honor the soft tool-access instruction. §4.6 reports 2,908 native execute_command + 717 Bash calls under Exp 2/3, “concentrated in the weaker 5ire and RooCode,” and notes that 5ire’s largest post-fix lift occurs precisely under the strict regime that most curbs its native fallback. If a large fraction of 5ire’s baseline failures are simply unconstrained native-tool thrashing or unpinned sampling noise rather than inferior orchestration of HexStrike tools, the 2.1× figure overstates the client-as-first-order-factor result that the paper treats as its methodological contribution. The authors correctly call multi-model generality a hypothesis; the more immediate concern is whether the single-model contrast is already confounded by constraint adherence.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper evaluates HexStrike-AI, an MCP-based security-tool orchestrator with 150+ tools, as a testbed for what bounds LLM agent success on offensive tasks. Across 86 picoCTF challenges, three tool-access regimes, and three model/client configurations (774 trials), it reports a large client effect for a fixed DeepSeek model (RooCode 59.7% vs 5ire 27.9%, ~2.1×), a monotonic difficulty gradient, and a bundled intervention (tool fixes, prompt/behavior changes, eleven new capability tools) that raises overall solve rate from 55.4% to 72.0% with significant paired gains. Residual failures after intervention are characterized as reasoning- or environment-bound rather than missing-tool. A 60-run stability sub-study supports single-run reproducibility. The authors are explicit that the study uses one benchmark, that fixes were tuned on the same challenges, and that multi-model generality of the client effect remains a hypothesis.","tokens_in":12885,"tokens_out":1699,"duration_ms":26990,"significance":"If the results hold, the paper makes a concrete methodological contribution to LLM-agent evaluation in security: a model’s CTF/orchestrator solve rate is not interpretable without naming the driving client/harness, and soft tool-access constraints must be measured rather than assumed. The 774-trial matrix, Wilson intervals, McNemar tests, native-tool leakage logging, recovery attribution (79 behavioral vs 49 specialized-tool), and stability sub-study are strengths relative to single-rate CTF reports. The evaluate–diagnose–intervene design and public fork further support reuse. The work is timely as MCP-style security orchestrators proliferate, even though external validity beyond picoCTF and one fixed-model client pair is limited by design.","major_comments":[{"comment":"RQ3 residual claim and same-set intervention (§3.4–3.5, Table 1, §4.3, §6.1): The conclusion that residual failures are “reasoning- or environment-bound rather than missing-tool” is load-bearing for RQ3, but the eleven capability tools, redirects, and prompt rules were motivated by and tuned on the same 86 challenges later re-scored, with 56 trials pre-judged “out of reach” and excluded from re-run while still counted as failures. On this design, remaining failures are partly residual by construction. The manuscript already flags same-set tuning; it should either (a) reframe the residual taxonomy as exploratory on this benchmark rather than a general bound, or (b) add a hold-out / leave-some-challenges-untuned analysis, or (c) report how many residual failures would still be tool-addressable under a pre-registered tool budget. Without one of these, the “where capability runs out” claim o","section":"§3.4–3.5, Table 1, §4.3, §6.1"},{"comment":"Client-effect isolation vs. constraint leakage and sampling (§4.1 Tables 2–3, §4.6, §5.1): The 2.1× DeepSeek client gap is the paper’s strongest isolation and is already visible under Exp 1 free-solve (59.3% vs 30.2%), which is important because it is not solely an artifact of soft HexStrike-only instructions. However, §5.1 attributes the gap to agent loop, tool-call repair, context/retry, and constraint enforcement without decomposing how much of the overall 2.1× is carried by Exp 1 versus Exp 2/3, where 2,908 native execute_command and 717 Bash calls are concentrated on the weaker clients. Temperature/top-p are also unpinned client defaults. For the central “name the client” claim, please report the client contrast with 95% CIs and tests separately by regime (especially Exp 1), and state more carefully what fraction of the gap could still be soft-constraint non-adherence or default sam","section":"§4.1, Tables 2–3, §4.6, §5.1"},{"comment":"RQ2 attribution under a bundled, one-directional intervention (§3.4–3.5, §4.2, §6.1): Recoveries are attributed 79 to behavioral/default fixes vs 49 to specialized tools, but the intervention is a single bundle (minimal baseline prompt → hypothesis-first rules + decomposition + tool nudges + new tools), and only prior non-successes were re-run (b=0 in McNemar by construction). The 79/49 split therefore cannot cleanly separate prompt upgrade from tooling, and post-fix rates cannot detect regressions among baseline successes. The paper notes both points; still, for RQ2 (“does recovery come more from specialized tools or default behavior?”) the claim should be stated as an upper/lower bound under a one-way re-run of a non-ablated bundle, or an ablation (prompt-only vs tools-only on a subset) should be added. As written, “behavior > tools” is suggestive but not identified.","section":"§3.4–3.5, §4.2, §6.1"}],"minor_comments":[{"comment":"Figure 1 caption packs Phase I/II numbers densely; a small table of trial counts (baseline 774, excluded 56, re-run 289, recoveries 128) would help readers verify the 72.0% post-fix arithmetic.","section":"Figure 1"},{"comment":"PARTIAL is defined and then folded into non-success for all rates; consider reporting PARTIAL rates in an appendix so partial-credit progress is not discarded for secondary analysis.","section":"§3.3"},{"comment":"Equation (1) Wilson interval and (2)–(3) McNemar/z-tests are standard; stating software/package used for the exact binomial reduction when b=0 would aid reproducibility.","section":"§3.6"},{"comment":"DeepSeek is a rolling deepseek-chat alias whose snapshot may have shifted (§6.2); if any run dates or API version pins are available, list them; otherwise keep the caveat more visible near Table 2.","section":"§6.2"},{"comment":"Minor typos/spacing: “HexStrikeAI” vs “HexStrike-AI” in the abstract; “McNemarp” missing space in abstract; “T able” line breaks in table titles; “wetesttheclienteffect” spacing glitch in §3.6.","section":"Abstract, §3.6, Tables"},{"comment":"Related work is strong on harness/ACI literature; a one-sentence contrast with Yu et al. (ref. 23), who found HexStrike strongest among five frameworks, would clarify why this paper treats it as testbed rather than competitor.","section":"§2"}],"recommendation":"minor_revision","confidential_remarks":"Fit for a solid empirical SE/security venue is good. The client-gap result is the publishable core and is cleaner than the skeptic note implies once Exp 1 is taken seriously; the same-set residual taxonomy is the main overclaim risk and should be toned or bounded in revision. I would not reject for lack of multi-model client generality given the authors already call it a hypothesis. No ethics red flags: picoCTF-only, no live targets, no flags shipped."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The result that matters is the DeepSeek contrast: RooCode 59.7% vs 5ire 27.9% on the same model and challenge set, and that separation already shows under free-solve (Exp 1: 59.3% vs 30.2%). So the worry that the whole 2.1× is just soft-constraint thrashing does not hold—the free regime still splits them. Naming the client when you report a model solve rate is the real takeaway, and it is earned.\n\nWhat they did well: full 86×3×3 matrix (774 trials), Wilson CIs, McNemar, a 60-run stability check (17/20 unanimous), and they logged native-tool leakage under “HexStrike-only” instead of pretending the instruction was hard. Recovery attribution is useful—79 from behavioral/default fixes vs 49 from specialized tools, with the pwntools redirect alone beating all eleven new tools. Authors are unusually explicit about same-set tuning, one-directional re-runs, and single-model client generality.\n\nSoft spots, in proportion: the 55.4%→72.0% lift and residual “reasoning-bound” taxonomy are engineered on the same 86 challenges, so treat them as existence proofs of recovery levers, not transfer claims. The intervention is a bundle (prompt upgrade + tools), so prompt and tooling cannot be fully separated. Soft constraints with thousands of native calls mean Exp 2/3 are not clean partitions—they measure that and use it to explain 5ire’s strict-regime lift. None of this sinks the client-gap result.\n\nStats are standard and correctly applied; citations cover PentestGPT, Cybench, SWE-agent ACI, and harness-effect work. This is for people building or evaluating MCP security agents and CTF harnesses—subfield-methodological, not a general AI claim. I’d send it to peer review and cite the client-isolation point. Engage.","headline":"Clean fixed-model client gap (already visible under free-solve); improvement numbers are weaker than the methodological point, and the authors mostly say so.","tokens_in":13499,"tokens_out":502,"would_cite":true,"duration_ms":12489,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"For a fixed LLM, the client that drives a security-tool orchestrator can swing CTF solve rates by a factor of two, and leftover failures are reasoning-bound rather than missing tools.","keywords":["LLM agents","penetration testing","capture the flag","tool orchestration","Model Context Protocol","agent harness","empirical evaluation","security tools"],"falsifier":"Re-run the same three model/client configurations on a held-out CTF suite or authorized real targets without re-tuning tools or prompts; if the client gap collapses, the post-fix lift disappears, or residual failures become missing-tool again, the central claims fail.","tokens_in":13365,"feed_emoji":"🔐","tokens_out":951,"duration_ms":17474,"temperature":0.7,"pith_summary":"This paper treats an open-source security-tool orchestrator (150+ tools over the Model Context Protocol) as a testbed and asks what actually bounds LLM agents that drive it. Across 86 picoCTF challenges and 774 trials, the same DeepSeek model solves 59.7% of challenges with one client and 27.9% with another—a 2.1× gap—so a model’s score is only meaningful when the client is named. A bundled intervention (tool corrections, behavior changes, eleven new capability tools) lifts overall success from 55.4% to 72.0%, with more recoveries from default-behavior fixes than from new specialized tools. What still fails is classified as reasoning- or environment-bound, not a missing-tool gap. Anyone who builds or benchmarks such agents should treat the harness as a first-order factor and measure constraint leakage rather than assume it.","feed_headline":"Same LLM, two clients: a 2.1× gap in CTF solve rates","feed_subtitle":"After tool and prompt fixes, leftover failures are reasoning-bound, not missing tools.","key_machinery":"The evaluate–diagnose–intervene workflow: a 774-trial baseline (86 challenges × three tool-access regimes × three model/client configurations), failure-mode characterization, then re-run of only prior non-successes under a bundled fix, with each recovery attributed to specialized tools (49) or to behavioral and default fixes (79).","core_discovery":"Holding the DeepSeek model fixed, two clients produce a 2.1× baseline solve-rate gap (59.7% vs 27.9%). After tool corrections, agent-behavior changes, and eleven new capability tools, overall success rises from 55.4% to 72.0%, every configuration improves significantly, and residual failures are reasoning- or environment-bound rather than missing-tool. A model’s solve rate is therefore meaningful only when the client that produced it is named.","pith_inferences":["Offensive-agent leaderboards will keep flipping until harness and client are treated as first-class experimental factors alongside the model.","Soft instruction-only tool gates systematically favor stronger clients; fair “orchestrator-only” comparisons may need hard technical disablement of native tools.","The finding that behavioral redirects recovered more trials than new specialized tools may generalize to other tool-heavy agent domains beyond CTF.","Real engagements will likely enlarge the environment-bound failure class that flag-centric CTF settings understate."],"forward_implications":["A model’s CTF or pentest solve rate is only interpretable when the client (harness) that produced it is named.","Benchmarks that vary models while fixing an unnamed harness may be measuring the harness as much as the model.","Instruction-only tool constraints leak native calls and must be measured, not assumed, especially for weaker clients.","Once core tooling is filled in, further gains come more from client orchestration and reasoning than from catalog breadth.","Mid-tier challenges absorb the largest intervention gains; hard multi-step and environment-bound tasks remain the floor."],"fun_headline_variants":["Same DeepSeek, two clients: 2.1× CTF solve-rate gap","Client choice alone yields 2.1× LLM CTF performance swing","After fixes, CTF failures stay reasoning-bound not tool-bound","LLM security solve rate rises 55% to 72% with client fixed","Name the client: model alone does not set CTF agent limits"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The fixes, prompts, and new tools were designed and tested on the same 86 challenges used to score them, so the measured lift and the “not missing-tool” residual taxonomy may not hold on held-out or non-CTF targets.","fun_headline_variants_meta":{"raw":{"variants":["Same DeepSeek, two clients: 2.1× CTF solve-rate gap","Client choice alone yields 2.1× LLM CTF performance swing","After fixes, CTF failures stay reasoning-bound not tool-bound","LLM security solve rate rises 55% to 72% with client fixed","Name the client: model alone does not set CTF agent limits"]},"model":"grok-4.5","effort":"low","cost_usd":0.004156,"raw_usage":{"total_tokens":1368,"prompt_tokens":917,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":41560000,"prompt_tokens_details":{"text_tokens":917,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":368,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":917,"tokens_out":83,"duration_ms":3574,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T06:27:13.116350+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same three model/client configurations on a held-out CTF suite or authorized real targets without re-tuning tools or prompts; if the client gap collapses, the post-fix lift disappears, or residual failures become missing-tool again, the central claims fail.","supporting_citations":[],"review_version":1}