{"id":"d2c85c9c-a006-461c-a35a-b65780976097","arxiv_id":"2607.15263","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Security-agent success changes differently with budget: offensive CTF tasks improve with more compute, while defensive SOC work depends more on tool discipline than spend.","lead":"This paper compares AI security agents by how much they achieve per dollar of compute and tool use, rather than by peak success alone. It finds that offensive hacking tasks improve with more computing budget, while defensive security work depends more on disciplined tool use than on spending more.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrospective cap replay assumes budget-unaware behavior transfers to real budgets; without prospective validation, the offensive scaling deltas may overstate headroom.","rationale":"The paper is a careful, well-caveated evaluation with real strengths: a common harness for both offensive and defensive tasks, explicit cost accounting, decontamination controls, paired bootstrap intervals for the main contrasts, and transparent limitations. The central claim—that offensive CTF performance scales with test-time compute while defensive SOC investigation does not—is plausible and supported by several complementary analyses, including a prospective budget comparison for DeepSeek v4 Flash on BOTS v1. However, the most load-bearing assumption is the validity of the retrospective cap analysis in Section 6. This is exactly what the reader identified as the weakest assumption. The concern is not that the authors are unaware of it—they explicitly call it 'descriptive replay'—but that the headline scaling conclusion depends on this replay being equivalent to true budget-constrained operation. A real budget would give the agent the opportunity to adapt its strategy, and such adaptation could easily change the shape of the scaling curves. This is an empirical question, not a theoretical flaw. The proposed test—prospective capped runs—would settle it. If the retrospective and prospective results match, the concern evaporates; if they differ, the scaling claims would need to be downgraded to 'success under budget-unaware operation' rather than 'success under realistic budget constraints.' Given the paper's transparency and the plausibility of the qualitative distinction, the reader's CONDITIONAL verdict remains appropriate, with the added condition that the retrospective method should be prospectively validated or explicitly framed as a measure of budget-unaware agent performance.","tokens_in":15675,"tokens_out":4897,"duration_ms":46866,"concrete_test":"Run prospective budget-capped evaluations for the same models on Cybench (and, if feasible, BOTS v1) with a $0.80 per-sample cap enforced during the run, using the same harness and agent but with the agent aware that total spend must stay under the cap (or at least with the run aborted at $0.80). Compare the resulting success rates and cost-scaling deltas to the retrospective estimates in Table 6. If the prospective success rates are within the reported bootstrap intervals of the retrospective rates for each model, the retrospective method is validated and the scaling claims stand. If they diverge (e.g., by more than a few percentage points), the retrospective analysis is not a valid measure of budget-constrained success and the scaling conclusions need to be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central scaling distinction in Section 6 rests on 'retrospective budget caps' applied to traces from agents that were never actually constrained by the cap. The paper itself states agents are 'unaware of cost or token budgets' (Section 3) and that these are 'descriptive replay analyses rather than new benchmark runs' (Section 6). This is a valid way to ask 'what success would this budget-unaware agent have achieved if stopped at this spend?' but it does not measure how an agent would behave if it knew it was operating under a $0.80 cap. A budget-aware agent could make different decisions: it might avoid long exploration chains on hard challenges, skip expensive enrichment, or prioritize easier tasks early, changing both the success level and the slope of the scaling curve. Consequently, the Cybench deltas (+18.8 pp for Claude Opus 4.8, +10.3 pp for DeepSeek v4 Flash) and the BOTS v1 deltas, which are the principal evidence for the offensive-vs-defensive scaling asymmetry, may not transfer to real constrained deployments. The claim that 'offensive CTF performance improves with additional test-time compute' is thus conditional on the retrospective replay being a faithful proxy for true budget-constrained performance—an assumption that is plausible but untested. Without prospective confirmation, the paper's headline scaling finding is not yet a robust empirical result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cost-aware evaluation methodology for offensive and defensive security agents, measuring success at fixed per-sample cost budgets rather than at peak capability. It applies this to Cybench CTF challenges and Splunk BOTS v1 SOC investigation tasks, decomposes spend into model inference and priced tool use, and reports full operating points for multiple frontier and open models. The principal claim is an asymmetry in scaling: offensive CTF success improves with additional test-time compute, while defensive SOC investigation does not scale with raw budget and instead depends on disciplined tool use, telemetry navigation, and selective enrichment. The paper also contributes refusal accounting, no-tools contamination controls for public BOTS v1, and a cost-per-point perspective.","tokens_in":16000,"tokens_out":7762,"duration_ms":68952,"significance":"If the scaling asymmetry is robust, this is a useful and actionable contribution: security-agent evaluations should report cost-success operating points rather than single peak-success leaderboards. The paper has several genuine strengths: a shared harness across offensive and defensive tasks, paired retrospective replays that control for model and trace identity, explicitly reported run dates and cost assumptions, bootstrap intervals for several key rows, and a rare same-model prospective budget comparison for the defensive side. The BOTS v1 no-tools controls are an honest and important check on a public benchmark. However, the load-bearing evidence for the offensive scaling claim is retrospective replay, and the defensive conclusion is weakened by the paper's own message-limit and contamination disclosures. The contribution is significant but the central empirical claim needs stronger prospective and decontamination support.","major_comments":[{"comment":"The retrospective budget-cap replay is the principal evidence for offensive scaling. The paper states in Section 3 that agents are 'unaware of cost or token budgets', and in Section 6 that the $0.80 cap analysis is a 'descriptive replay' rather than a new run. Re-scoring a completed trace under a budget measures the cost-sensitivity of a budget-unaware policy, not the behavior of a policy that knows it must finish under $0.80. The headline deltas (+18.8 pp for Claude Opus 4.8, +10.3 pp for DeepSeek v4 Flash) may not transfer to real budget-constrained deployments. The only same-model prospective cap comparison on BOTS is inconclusive (Table 10, interval [−4.9, +6.9]), and no prospective Cybench comparison of this type exists. Please add at least one prospective budget-cap run for an offensive model, or explicitly restrict the abstract and conclusion claims to budget-unaware agents.","section":"Section 6 / Table 6 / Section 3"},{"comment":"The defensive non-scaling conclusion is based on absolute full-agent BOTS v1 scores, but the paper's own contamination controls show that no-tools plus official prerequisite Q&A recovers 50.5% to 77.2% of points for several models (e.g., GPT-5.6 Sol 77.2%, Claude Opus 4.8 74.8%, GPT-5.5 62.1%). Because BOTS v1 is public and old, high direct-answer recovery means memorization can inflate the very scores whose flat scaling is interpreted as 'tool discipline, not budget'. The paper acknowledges in Section 7 that absolute scores require decontamination checks, but it still uses those scores as the basis for the defensive scaling claim. A sensitivity analysis that removes or downweights questions recoverable without tools, or a private/perturbed holdout, is needed before the defensive asymmetry can be accepted.","section":"Section 5 / Table 5"},{"comment":"Section 6 calls the DeepSeek v4 Flash $2.10 vs $4.20 comparison the 'cleanest direct budget check' for BOTS, but Section 7 reports that the 250-message limit was hit by 8/93 sample-epochs at the $2.10 cap and 14/93 at the $4.20 cap. At the higher budget, more sample-epochs were truncated by message count rather than by budget. The flat 0.9 pp gain may therefore reflect the message limit, not the absence of budget-sensitivity. This directly weakens the 'clean check' characterization and the defensive non-scaling claim. Please either rerun with a higher message limit or report results conditional on non-truncated epochs.","section":"Section 6 / Section 7 / Table 3"},{"comment":"The quantitative scaling and bootstrap analyses exclude the GPT-5.6, Fable, and latest July Opus rows, as stated in Appendix B. This is not an ancillary omission: Table 4 shows GPT-5.6 Sol moving from 9.4% to 87.2% on Cybench after the account-state change, and GPT-5.6 Terra and Luna also shift by 10–20 pp. With the newest frontier models excluded from the bootstrap support, the abstract's general statement that 'offensive CTF performance improves with additional test-time compute' should be scoped to the audited subset, or the analyses must be extended to the new rows.","section":"Appendix B / Table 6 / Table 4"}],"minor_comments":[{"comment":"Typo: 'DeepSeke' should be 'DeepSeek'.","section":"Table 2 caption"},{"comment":"The 'solved equiv.' column shows values like '36.77/117'. If these are challenge equivalents, the denominator should be 39 (the number of Cybench challenges); if they are sample-epochs, the label should not say 'equiv.' Please clarify the unit.","section":"Table 2"},{"comment":"The 250-message limit is mentioned only in Limitations, but it materially affects the BOTS v1 runs. Please state this limit in the evaluation design, alongside the cost caps and tool limits.","section":"Section 3.2"},{"comment":"Figures 4 and 5 include July Kimi K3 and Qwen 3.7 Max traces, while Appendix B says the bootstrap/scaling analyses exclude the new rows. State explicitly whether the July traces in the scaling figures are illustrative or are included in the headroom deltas.","section":"Section 6 / Appendix B"},{"comment":"The no-tools controls are single-epoch runs while the full-agent baselines are three-epoch means. The comparison would be cleaner if the no-tools probes used the same number of epochs, since sampling variance could affect the reported recovery percentages.","section":"Table 5 / Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent and the evaluation harness is a useful contribution, but the central scaling asymmetry is currently supported by retrospective replay rather than prospective budget-constrained behavior, and the defensive conclusion is further complicated by message-limit truncation and high no-tools answer recovery. A revision that adds prospective runs, addresses the DeepSeek truncation issue, and connects the contamination controls to the scaling analysis would substantially strengthen the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid measurement paper. It adds cost-success operating points to security-agent evaluation, plus a genuinely useful no-tools contamination control for BOTS v1. The central finding—offensive CTF success keeps climbing with test-time compute while defensive SOC scores flatten—is plausible and well-caveated. The main soft spot is the retrospective budget-cap replay in Section 6. The stress-test note is right that the agents never operated under the cap, so the deltas describe what a budget-unaware agent would have done if stopped earlier, not how a budget-aware agent would behave. But the authors label these as descriptive replay analyses, and they include one prospective contrast (DeepSeek v4 Flash at $2.10 vs $4.20) that shows no meaningful defensive gain. So the concern is real but bounded; it does not sink the paper.\n\nWhat's new: the paired budget-headroom analysis, priced-tool cost accounting, and the no-tools contamination probes. The contamination controls are a genuine service—showing that no-tools BOTS v1 scores vary from 13% to 77% depending on model and context should be required reading for anyone using public SOC benchmarks. The custom BOTS v1 harness is also a contribution, even if not released.\n\nSoft spots: the abstract's \"cost-competitive\" overstates what Table 2 shows—DeepSeek v4 Flash is cheaper per solve than Claude Opus but not than GPT-5.5. The newest GPT-5.6 and Fable rows lack the bootstrap intervals older rows get, and the July runs are observational with acknowledged account-state changes. The harness and logs aren't released, which limits reproducibility but isn't fatal.\n\nBottom line: this deserves a serious referee. The measurement framework is useful, the limitations are honestly stated, and the central scaling claim is conditional but clearly presented as such. I'd bring it to reading group and likely cite it. My recommendation: engage, and push for prospective budget runs and artifact release in revision.","headline":"Cost-aware evaluation is a real step forward for security-agent benchmarks; the retrospective replay is a genuine caveat, but the paper handles it honestly.","tokens_in":16456,"tokens_out":2042,"would_cite":true,"duration_ms":17879,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Security-agent usefulness is a cost-success question: offensive CTF performance climbs with extra test-time compute, while defensive SOC investigation does not scale with budget in the same way.","keywords":["security agents","cost-aware evaluation","offensive security","defensive security","test-time compute","tool use discipline","benchmark contamination","operating points"],"falsifier":"Run the same models under a true prospective budget equal to the retrospective cap (e.g., $0.80 per sample on both benchmark families) and compare scores to the replayed-trace results. If prospective runs diverge substantially—because agents choose cheaper actions or stop earlier—the paper's scaling deltas and its offensive-versus-defensive distinction would not transfer to real constrained deployments.","tokens_in":15570,"feed_emoji":"🛡️","tokens_out":3879,"duration_ms":31978,"temperature":0.7,"pith_summary":"The paper argues that measuring security agents by peak success under generous budgets is incomplete; operational security consumes budget at every step. It evaluates agents on offensive capture-the-flag challenges and a defensive security-operations-center investigation under fixed cost caps, reporting success at given spend rather than only best-case results. The central finding: offensive task performance improves with additional test-time compute, and scaled open-weight models can be cost-competitive with frontier proprietary systems, whereas defensive investigation success depends more on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget. The paper proposes cost-aware evaluation with operating points, budget-headroom analysis, and decontamination controls as the standard.","feed_headline":"Offense scales with compute; defense needs discipline, not budget","feed_subtitle":"A cost-aware evaluation shows SOC agents gain little from extra spend, while CTF agents convert budget into solves.","key_machinery":"The central mechanism is the cost-success operating point: a per-sample budget cap that turns success rate into an economic question of how much capability a model buys per dollar of inference and tooling. Paired with this is retrospective budget-cap replay, where a completed trace is re-scored under a lower cap to measure headroom while holding the agent's behavior fixed. The harness also separates inference spend from priced-tool spend (e.g., external enrichment lookups) and records tool-call counts, so scaling curves can be plotted against dollars, tokens, and interaction volume.","core_discovery":"The paper's core claim is that red- and blue-team security agents live in different scaling regimes. On offensive CTF workloads, extra budget buys extra solves: replaying completed traces under a lower per-sample cost cap shows large success-rate drops for some models and near-zero drops for others, indicating real headroom. On defensive SOC investigation, higher spend and more tool calls do not reliably buy higher scores; the strongest model achieves the highest score with roughly an order of magnitude fewer tool calls than high-volume models. The paper therefore argues that security-agent benchmarks should report cost-success operating points—success rate at a fixed cost limit—and decompos","pith_inferences":["If the offensive-versus-defensive scaling distinction holds, then simply scaling reasoning effort or model size may not improve SOC agents; investment should shift toward tool selection, telemetry navigation, and selective enrichment strategies.","A testable extension: run agents under true prospective low budgets and compare to the paper's retrospective replays; divergence would show whether the scaling curves transfer to real constrained deployments.","The contamination spread across models suggests that old public SOC benchmarks may partly reward memorization; a private fresh-incident benchmark would give a cleaner signal of live investigation capability.","The cost-per-1,000-points metric generalizes naturally to other agent domains with priced tools, making economic efficiency a standard evaluation axis beyond security."],"forward_implications":["Reporting only peak success overstates offensive usefulness; a model that needs a huge budget is less operationally valuable than one that solves at low spend.","Defensive SOC evaluation should report tool-call efficiency and evidence quality alongside final score, not just accuracy.","Open-weight models can be cost-competitive with frontier proprietary systems on offensive tasks when compared at equal cost.","Budget-headroom analysis reveals which models genuinely benefit from extra compute and which have already hit their practical ceiling.","Public SOC benchmarks need decontamination controls before absolute scores are interpreted as live investigation skill."],"fun_headline_variants":["Red teams scale with compute, blue teams don't","For security agents, budget buys offense but not defense","CTF agents spend to win, SOC agents win by saving","Cost-aware eval: offense scales, defense plateaus","Security AI: more compute helps attackers, not defenders"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that replaying a finished trace under a lower budget tells you how the agent would do if it had actually been running under that budget—but a real budget-constrained agent could choose different, cheaper actions, so the measured scaling gaps may not hold in practice.","fun_headline_variants_meta":{"raw":{"variants":["Red teams scale with compute, blue teams don't","For security agents, budget buys offense but not defense","CTF agents spend to win, SOC agents win by saving","Cost-aware eval: offense scales, defense plateaus","Security AI: more compute helps attackers, not defenders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000148,"raw_usage":{"total_tokens":1022,"prompt_tokens":734,"completion_tokens":288,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":210}},"tokens_in":478,"tokens_out":288,"duration_ms":3261,"temperature":1.0,"reasoning_tokens":210,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:41:08.724123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same models under a true prospective budget equal to the retrospective cap (e.g., $0.80 per sample on both benchmark families) and compare scores to the replayed-trace results. If prospective runs diverge substantially—because agents choose cheaper actions or stop earlier—the paper's scaling deltas and its offensive-versus-defensive distinction would not transfer to real constrained deployments.","supporting_citations":[],"review_version":1}