Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Security-agent usefulness is a cost-success question: offensive CTF performance climbs with extra test-time compute, while defensive SOC investigation does not scale with budget in the same way.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:41 UTC pith:5BZ22R6T

load-bearing objection Cost-aware evaluation is a real step forward for security-agent benchmarks; the retrospective replay is a genuine caveat, but the paper handles it honestly. the 4 major comments →

arxiv 2607.15263 v3 pith:5BZ22R6T submitted 2026-07-16 cs.CR cs.AI

Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Paul Kassianik , Blaine Nelson , Yaron Singer This is my paper
classification cs.CR cs.AI
keywords security agentscost-aware evaluationoffensive securitydefensive securitytest-time computetool use disciplinebenchmark contaminationoperating points
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that measuring security agents by peak success under generous budgets is incomplete; operational security consumes budget at every step. It evaluates agents on offensive capture-the-flag challenges and a defensive security-operations-center investigation under fixed cost caps, reporting success at given spend rather than only best-case results. The central finding: offensive task performance improves with additional test-time compute, and scaled open-weight models can be cost-competitive with frontier proprietary systems, whereas defensive investigation success depends more on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget. The paper proposes cost-aware evaluation with operating points, budget-headroom analysis, and decontamination controls as the standard.

Core claim

The paper's core claim is that red- and blue-team security agents live in different scaling regimes. On offensive CTF workloads, extra budget buys extra solves: replaying completed traces under a lower per-sample cost cap shows large success-rate drops for some models and near-zero drops for others, indicating real headroom. On defensive SOC investigation, higher spend and more tool calls do not reliably buy higher scores; the strongest model achieves the highest score with roughly an order of magnitude fewer tool calls than high-volume models. The paper therefore argues that security-agent benchmarks should report cost-success operating points—success rate at a fixed cost limit—and decompos

What carries the argument

The central mechanism is the cost-success operating point: a per-sample budget cap that turns success rate into an economic question of how much capability a model buys per dollar of inference and tooling. Paired with this is retrospective budget-cap replay, where a completed trace is re-scored under a lower cap to measure headroom while holding the agent's behavior fixed. The harness also separates inference spend from priced-tool spend (e.g., external enrichment lookups) and records tool-call counts, so scaling curves can be plotted against dollars, tokens, and interaction volume.

Load-bearing premise

The load-bearing assumption is that replaying a finished trace under a lower budget tells you how the agent would do if it had actually been running under that budget—but a real budget-constrained agent could choose different, cheaper actions, so the measured scaling gaps may not hold in practice.

What would settle it

Run the same models under a true prospective budget equal to the retrospective cap (e.g., $0.80 per sample on both benchmark families) and compare scores to the replayed-trace results. If prospective runs diverge substantially—because agents choose cheaper actions or stop earlier—the paper's scaling deltas and its offensive-versus-defensive distinction would not transfer to real constrained deployments.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reporting only peak success overstates offensive usefulness; a model that needs a huge budget is less operationally valuable than one that solves at low spend.
  • Defensive SOC evaluation should report tool-call efficiency and evidence quality alongside final score, not just accuracy.
  • Open-weight models can be cost-competitive with frontier proprietary systems on offensive tasks when compared at equal cost.
  • Budget-headroom analysis reveals which models genuinely benefit from extra compute and which have already hit their practical ceiling.
  • Public SOC benchmarks need decontamination controls before absolute scores are interpreted as live investigation skill.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the offensive-versus-defensive scaling distinction holds, then simply scaling reasoning effort or model size may not improve SOC agents; investment should shift toward tool selection, telemetry navigation, and selective enrichment strategies.
  • A testable extension: run agents under true prospective low budgets and compare to the paper's retrospective replays; divergence would show whether the scaling curves transfer to real constrained deployments.
  • The contamination spread across models suggests that old public SOC benchmarks may partly reward memorization; a private fresh-incident benchmark would give a cleaner signal of live investigation capability.
  • The cost-per-1,000-points metric generalizes naturally to other agent domains with priced tools, making economic efficiency a standard evaluation axis beyond security.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a cost-aware evaluation methodology for offensive and defensive security agents, measuring success at fixed per-sample cost budgets rather than at peak capability. It applies this to Cybench CTF challenges and Splunk BOTS v1 SOC investigation tasks, decomposes spend into model inference and priced tool use, and reports full operating points for multiple frontier and open models. The principal claim is an asymmetry in scaling: offensive CTF success improves with additional test-time compute, while defensive SOC investigation does not scale with raw budget and instead depends on disciplined tool use, telemetry navigation, and selective enrichment. The paper also contributes refusal accounting, no-tools contamination controls for public BOTS v1, and a cost-per-point perspective.

Significance. If the scaling asymmetry is robust, this is a useful and actionable contribution: security-agent evaluations should report cost-success operating points rather than single peak-success leaderboards. The paper has several genuine strengths: a shared harness across offensive and defensive tasks, paired retrospective replays that control for model and trace identity, explicitly reported run dates and cost assumptions, bootstrap intervals for several key rows, and a rare same-model prospective budget comparison for the defensive side. The BOTS v1 no-tools controls are an honest and important check on a public benchmark. However, the load-bearing evidence for the offensive scaling claim is retrospective replay, and the defensive conclusion is weakened by the paper's own message-limit and contamination disclosures. The contribution is significant but the central empirical claim needs stronger prospective and decontamination support.

major comments (4)
  1. [Section 6 / Table 6 / Section 3] The retrospective budget-cap replay is the principal evidence for offensive scaling. The paper states in Section 3 that agents are 'unaware of cost or token budgets', and in Section 6 that the $0.80 cap analysis is a 'descriptive replay' rather than a new run. Re-scoring a completed trace under a budget measures the cost-sensitivity of a budget-unaware policy, not the behavior of a policy that knows it must finish under $0.80. The headline deltas (+18.8 pp for Claude Opus 4.8, +10.3 pp for DeepSeek v4 Flash) may not transfer to real budget-constrained deployments. The only same-model prospective cap comparison on BOTS is inconclusive (Table 10, interval [−4.9, +6.9]), and no prospective Cybench comparison of this type exists. Please add at least one prospective budget-cap run for an offensive model, or explicitly restrict the abstract and conclusion claims to budget-unaware agents.
  2. [Section 5 / Table 5] The defensive non-scaling conclusion is based on absolute full-agent BOTS v1 scores, but the paper's own contamination controls show that no-tools plus official prerequisite Q&A recovers 50.5% to 77.2% of points for several models (e.g., GPT-5.6 Sol 77.2%, Claude Opus 4.8 74.8%, GPT-5.5 62.1%). Because BOTS v1 is public and old, high direct-answer recovery means memorization can inflate the very scores whose flat scaling is interpreted as 'tool discipline, not budget'. The paper acknowledges in Section 7 that absolute scores require decontamination checks, but it still uses those scores as the basis for the defensive scaling claim. A sensitivity analysis that removes or downweights questions recoverable without tools, or a private/perturbed holdout, is needed before the defensive asymmetry can be accepted.
  3. [Section 6 / Section 7 / Table 3] Section 6 calls the DeepSeek v4 Flash $2.10 vs $4.20 comparison the 'cleanest direct budget check' for BOTS, but Section 7 reports that the 250-message limit was hit by 8/93 sample-epochs at the $2.10 cap and 14/93 at the $4.20 cap. At the higher budget, more sample-epochs were truncated by message count rather than by budget. The flat 0.9 pp gain may therefore reflect the message limit, not the absence of budget-sensitivity. This directly weakens the 'clean check' characterization and the defensive non-scaling claim. Please either rerun with a higher message limit or report results conditional on non-truncated epochs.
  4. [Appendix B / Table 6 / Table 4] The quantitative scaling and bootstrap analyses exclude the GPT-5.6, Fable, and latest July Opus rows, as stated in Appendix B. This is not an ancillary omission: Table 4 shows GPT-5.6 Sol moving from 9.4% to 87.2% on Cybench after the account-state change, and GPT-5.6 Terra and Luna also shift by 10–20 pp. With the newest frontier models excluded from the bootstrap support, the abstract's general statement that 'offensive CTF performance improves with additional test-time compute' should be scoped to the audited subset, or the analyses must be extended to the new rows.
minor comments (5)
  1. [Table 2 caption] Typo: 'DeepSeke' should be 'DeepSeek'.
  2. [Table 2] The 'solved equiv.' column shows values like '36.77/117'. If these are challenge equivalents, the denominator should be 39 (the number of Cybench challenges); if they are sample-epochs, the label should not say 'equiv.' Please clarify the unit.
  3. [Section 3.2] The 250-message limit is mentioned only in Limitations, but it materially affects the BOTS v1 runs. Please state this limit in the evaluation design, alongside the cost caps and tool limits.
  4. [Section 6 / Appendix B] Figures 4 and 5 include July Kimi K3 and Qwen 3.7 Max traces, while Appendix B says the bootstrap/scaling analyses exclude the new rows. State explicitly whether the July traces in the scaling figures are illustrative or are included in the headroom deltas.
  5. [Table 5 / Figure 3] The no-tools controls are single-epoch runs while the full-agent baselines are three-epoch means. The comparison would be cleaner if the no-tools probes used the same number of epochs, since sampling variance could affect the reported recovery percentages.

Circularity Check

0 steps flagged

No significant circularity: the paper is an external-benchmark evaluation with no fitted parameters, no load-bearing self-citations, and its one self-referential element (retrospective cap replay) is explicitly labeled descriptive rather than predictive.

full rationale

The paper's claims are measurements against external benchmarks (Cybench and Splunk BOTS v1) with openly stated scoring rules, so no claimed result is equivalent to its input by construction. There are no fitted parameters: all cost figures come from provider token ledgers and published rate sheets, success rates are scored by official flag matching and official BOTS points, and the headline offensive-vs-defensive scaling contrast rests on empirical magnitudes (e.g., Claude Opus 4.8's +18.8 pp versus GPT-5.5's +2.6 pp Cybench headroom, and DeepSeek v4 Flash's 0.0 pp BOTS delta), not on any equation that forces the conclusion. The only self-referential element is the Section 6 retrospective budget-cap analysis, which re-scores completed traces at a $0.80 cap; the paper explicitly states, 'These are descriptive replay analyses rather than new benchmark runs' (Section 6) and cautions that the DeepSeek v4 Pro zero delta 'should not be read as evidence that the model cannot scale: the run was capped at a low prospective limit.' While the sign of a capped-vs-full delta is monotone by construction, the paper's substantive claims concern the size and model-dependence of the deltas, which are empirical. The skeptic's concern (budget-unaware traces may not transfer to truly budget-constrained deployments) is an external-validity limitation, not circularity, and the paper repeatedly hedges the claim as holding 'under retrospective caps' (Section 9) and presents the SOC conclusion as 'an evaluation-design result rather than a final claim about production SOC readiness' (Section 7). No self-citations appear in the reference list; citations to ReAct, Inspect, Cybench, and BOTS are external and independently checkable. The decontamination controls (Section 5) are additional evidence, not inputs to the main claim. Therefore no circular step can be exhibited, and the honest finding is score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The paper's results rest on evaluation-design choices (cost caps, tool limits, refusal policy) and framework/pricing assumptions. No new physical or mathematical entities are introduced, and no parameters are fitted to data.

free parameters (4)
  • Retrospective budget cap $0.80 per sample = $0.80
    Hand-chosen threshold in Section 6/Table 6; the magnitudes of scaling deltas and 'headroom' conclusions depend on this choice.
  • Prospective per-sample cost caps = $2.10 and $4.20
    Main budget caps for Tables 2-4; selected by the authors, not derived; samples overrunning the cap are aborted and scored as failures.
  • Priced-tool call limits = 5 web searches, 3 VirusTotal/WhoisXML total calls
    Hand-set constraints in Section 3.2; they shape tool-spend and defensive scores, especially for high-volume models.
  • Refusal limit of one per sample-epoch = 1
    Section 3.3; a single heuristic-detected refusal aborts the Cybench sample and affects refusal-dominated rows such as GPT-5.6 Sol and Fable 5.
axioms (6)
  • domain assumption Inspect evaluation framework correctly records model-token costs and tool-call statistics
    Section 3 relies on Inspect for cost, token, and tool metrics; if ledger data are inaccurate, all dollar-denominated comparisons fail.
  • domain assumption Public BOTS v1 official scoring (points, hint penalties, prerequisite structure) is correctly transcribed and applied
    Section 3.2 and Appendix E assume the official competition scoring and prerequisite chart are faithfully reproduced.
  • domain assumption Three independent epochs with mean reduction give a valid estimate of model performance
    Sections 3.1-3.2; the paper does not analyze epoch-level variance or agreement, and bootstraps resample epochs but assume homogeneity.
  • domain assumption Retrospective replay of a completed trace under a lower cap is a valid descriptive measure of budget-constrained success
    Section 6 explicitly calls these 'descriptive replay analyses rather than new benchmark runs'; interpreting them as scaling evidence depends on this premise.
  • domain assumption OpenRouter average pricing across one primary and two fallback providers approximates true deployment cost
    Section 3 states average pricing and provider cache-miss variation; cost comparisons are only as good as these approximations.
  • domain assumption Observed BOTS v1 responses are not substantially contaminated beyond what the no-tools controls reveal
    Section 5's no-tools probes provide a partial control but do not establish the mechanism or full extent of contamination; absolute scores remain uncertain.

pith-pipeline@v1.3.0-alltime-deepseek · 15434 in / 13053 out tokens · 122597 ms · 2026-08-01T23:41:08.724123+00:00 · methodology

0 comments
read the original abstract

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.

Figures

Figures reproduced from arXiv: 2607.15263 by Blaine Nelson, Paul Kassianik, Yaron Singer.

Figure 1
Figure 1. Figure 1: Resource scaling for various models on Cybench and BOTS v1. Left: Cybench success rate versus a per-sample cost budget cap: a challenge counts as solved at budget x only if its model-token spend was at most x. Right: BOTS v1 binary answer accuracy versus a per-sample cap on tool calls, shown on a symmetric-log axis. Claude Fable 5 appears only BOTS v1 due to refusals on Cybench. These GPT-5.6, Fable, and K… view at source ↗
Figure 2
Figure 2. Figure 2: Performance and policy refusals for selected current models, including Kimi K3 and Qwen 3.7 Max. Hatched segments show refusal events in both panels. Left: BOTS v1 refusal rates overlay the performance bars because affected sample-epochs may still earn points when a later attempt succeeds. Right: Cybench refusal outcomes are failures and therefore stack beyond, rather than overlap, solved share. dominates … view at source ↗
Figure 3
Figure 3. Figure 3: BOTS v1 no-tools contamination probe. No-tools rows are single-epoch controls with zero non-submit tool events; full-agent baselines are three-epoch mean runs. Prior Q&A denotes official prerequisite question-answer context [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cybench scaling curves for the previously audited subset plus the July Kimi K3 and Qwen 3.7 Max traces. Left: success versus retrospective per-sample cost budget (USD, log scale). Right: success versus cumulative tokens (input, output, cached, and reasoning). Success keeps rising toward the run limits: extra compute buys additional solves, most visibly for DeepSeek v4 Flash and Claude Opus 4.8, while GPT-5… view at source ↗
Figure 5
Figure 5. Figure 5: BOTS v1 scaling curves for the previously audited full-agent subset plus Kimi K3 and Qwen 3.7 Max. Left: official BOTS points (% of 10,300) versus a retrospective per-sample budget covering model-token plus priced-tool spend; dotted companions show the model-token-only budget, while solid curves include priced Brave Search and WhoisXMLAPI calls. Right: points versus cumulative tokens. Unlike Cybench, highe… view at source ↗
Figure 6
Figure 6. Figure 6: Tool-call scaling: success at cap x counts a sample-epoch only if it succeeded within x non-submit tool calls. Top: Cybench success on a linear 0–200-call axis, with the inset continuing the DeepSeek v4 Flash trajectory over the 200–930-call long tail; most models saturate within ∼50 calls, but DeepSeek v4 Flash keeps extracting solves past 800 — on offense, interaction volume can substitute for per-call q… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 14 linked inside Pith

  1. [2]

    org/abs/2404.13161

    URLhttps://arxiv. org/abs/2404.13161. Brave. Brave search API documentation,

    Pith/arXiv arXiv

  2. [4]

    Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg

    URLhttps://arxiv.org/abs/2410.09247. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. InProceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 5075–5084. Association for Computational...

    Pith/arXiv arXiv 2023

  3. [8]

    Xiaoqun Liu et al

    URLhttps://arxiv.org/abs/2406.04244. Xiaoqun Liu et al. Benchmarking LLMs in an embodied environment for blue team threat hunting,

    Pith/arXiv arXiv

  4. [9]

    Suryadipta Majumdar et al

    URLhttps://arxiv.org/abs/2505.11901. Suryadipta Majumdar et al. Before you hand over the wheel: Evaluating LLMs for security incident analysis,

    Pith/arXiv arXiv

  5. [10]

    Paul Mockapetris

    URLhttps://arxiv.org/abs/2603.06422. Paul Mockapetris. Domain names – implementation and specification. RFC 1035,

    arXiv

  6. [11]

    URLhttps://arxiv.org/abs/2406.05590. Splunk. Boss of the SOC dataset version 1,

    Pith/arXiv arXiv

  7. [12]

    UK AI Security Institute

    URLhttps://arxiv.org/abs/2412.02776. UK AI Security Institute. Inspect AI: Framework for large language model evaluations,

    Pith/arXiv arXiv

  8. [13]

    Zengzhi Wang et al

    URLhttps://arxiv.org/abs/ 2408.01605. Zengzhi Wang et al. Benchmarking benchmark leakage in large language models,

    Pith/arXiv arXiv

  9. [14]

    WhoisXML API

    URL https://arxiv.org/abs/2404.18824. WhoisXML API. WHOIS History API documentation,

    Pith/arXiv arXiv

  10. [15]

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao

    URLhttps://arxiv.org/abs/2507.14201. John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. InterCode: Stan- dardizing and benchmarking interactive coding with execution feedback. InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2023a. URL https://openreview.net/forum?id=fvKaLF1ns8. John Yang, Akshara Prabhaka...

    Pith/arXiv arXiv

  11. [16]

    URL https://arxiv.org/abs/2210.03629. Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models,

    Pith/arXiv arXiv

  12. [17]

    URLhttps://arxiv.org/abs/2408.08926. Andy K. Zhang et al. BountyBench: Dollar impact of AI agent attackers and defenders on real-world cybersecurity systems. InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,

    Pith/arXiv arXiv

  13. [18]

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, et al

    URLhttps://arxiv.org/abs/2505.15216. Yuxuan Zhu, Antony Kellermann, Dylan Bowman, et al. CVE-Bench: A benchmark for AI agents’ ability to exploit real-world web application vulnerabilities,

    arXiv

  14. [19]

    Appendix A

    URL https://arxiv.org/abs/2503.17332. Appendix A. Evaluation Run Dates Tables 7 and 8 report the UTC creation timestamp recorded in each source Inspect log used by the paper tables. Rows that appear in multiple result, scaling, or robustness tables are listed once. Where a row aggregates retry logs into the reported cost, the retry timestamps are included...

    Pith/arXiv arXiv 2026

  15. [2004]

    rfc-editor.org/rfc/rfc3912

    URL https://www. rfc-editor.org/rfc/rfc3912. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. In- vestigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 870...

    doi:10.18653/v1/2024.naacl-long.482 2024

  16. [2023]

    URL https://aclanthology.org/2023

    doi: 10.18653/v1/2023.emnlp-main.308. URL https://aclanthology.org/2023. emnlp-main.308/. Pengfei Jing et al. SecBench: A comprehensive multi-dimensional benchmarking dataset for LLMs in cybersecurity,

    doi:10.18653/v1/2023.emnlp-main.308 2023

  17. [2024]

    URLhttps://aclanthology.org/2024.eacl-long.5/

    doi: 10.18653/v1/2024.eacl-long.5. URLhttps://aclanthology.org/2024.eacl-long.5/. Manish Bhatt, Sahana Chennabasappa, Yue Li, et al. CyberSecEval 2: A wide-ranging cybersecurity evaluation suite for large language models,

    doi:10.18653/v1/2024.eacl-long.5 2024

  18. [2025]

    15 Kassianik Nelson Singer Ambuj Kumar et al

    URLhttps://arxiv.org/abs/2412.20787. 15 Kassianik Nelson Singer Ambuj Kumar et al. Cyber defense benchmark: Agentic threat hunting evaluation for LLMs in SecOps,

    Pith/arXiv arXiv

  19. [2026]

    Yucheng Li et al

    URLhttps://arxiv.org/abs/2604.19533. Yucheng Li et al. Benchmark data contamination of large language models: A survey,

    Pith/arXiv arXiv