Pith. sign in

REVIEW 3 major objections 6 minor 15 references

Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing

T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Default coding agents may match security-specific pentesting systems once models are matched

desk verdict The baseline-first message is right and well supported by the internal runs; the cross-study residuals and 'beats 91%' claim are not controlled measurements, but the paper is honest about that and deserves a serious referee. read the letter →

arxiv 2607.13085 v1 pith:66E7RM4O submitted 2026-07-13 cs.CR cs.AI

Ananda Dhakal , Krish Neupane , Aarjan Chaudhary This is my paper · ORCID
classification cs.CRcs.AI
keywords autonomouspenetrationtestingXBOWbenchmarkcodingagentsLLMagentbaselinesmodelscalingarchitectureattributionpromptengineeringsecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the field of autonomous penetration testing cannot attribute benchmark gains to security-specific architecture unless it first measures a strong, model-matched plain-agent baseline. Running default coding CLI agents on the 104-task XBOW benchmark, it finds that Codex with GPT-5 already solves 67.3% of tasks on average, and with GPT-5.5 reaches 92.3% pass@1 and 95.2% two-pass coverage, exceeding the 91% headline of a purpose-built system. Security-specific prompt variants did not improve over the default prompt. The paper concludes that architecture can add a few points on single-attempt efficiency, but model progress alone closes most of the gap.

What carries the argument

The central object is the 'architecture residual' — the difference between a published purpose-built system's solve rate and the closest model-matched plain coding-agent solve rate — computed under a controlled harness that varies only one factor at a time (agent CLI, prompt, model) while holding model, budget, target interface, and scoring rule fixed. The plain-agent baseline is a default coding CLI (Codex, OpenCode, Pi) with no security-specific modifications, whose general loop of write-code, execute, observe, and revise is argued to already instantiate the iterative probing mechanics of web exploitation.

What would settle it

Rerun MAPTA and PentestGPT V2 inside the paper's harness with identical model, budget cap, and target images; if the residuals vanish or reverse, the architecture contribution is smaller than reported. Conversely, run plain Codex on the original unrepaired benchmark; if scores drop substantially, the local repairs rather than agent ability explain the baseline.

Watch

Extended reading notes

Core claim

On the 104-task XBOW benchmark, a default, unmodified Codex CLI agent is a strong baseline for autonomous penetration testing. Across two full passes, Codex with GPT-5 averages 67.3% pass@1, GPT-5.2 averages 79.8%, and GPT-5.5 averages 92.3%, with two-pass union coverage of 77.9%, 88.5%, and 95.2% respectively. The GPT-5.5 plain-agent pass@1 exceeds the 91% headline reported for PentestGPT V2 with Opus 4.5, and the two-pass union exceeds it by 4.2 points. Against model-matched published results, MAPTA retains a 9.6-point residual over GPT-5 Codex, and PentestGPT V2 retains a 5.2-point residual over GPT-5.2 Codex, but both residuals are smaller than headline system-to-system gaps suggest, and

Load-bearing premise

That the published MAPTA and PentestGPT V2 scores are directly comparable to this paper's runs, despite the paper's fork repairing about 40 benchmark targets, different cost caps, and unknown inference settings, and despite those systems not being rerun in the same harness.

Editorial extensions

If this is right

  • Future autonomous-pentesting papers should report a model-matched plain coding-agent baseline before attributing benchmark gains to architecture.
  • Repeated plain-agent runs (pass@2) can match or exceed some published architecture scores, so single-pass comparisons overstate harness value.
  • Backbone model progress alone substantially lifts a fixed minimal scaffold, meaning headline scores conflate model and architecture gains.
  • Security-specific prompt text did not help in this study; prompt design should be compared against the agent's shipped default.
  • Architecture can still deliver value on single-attempt efficiency and cost per solved challenge, even when union coverage is matched.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The results suggest that some reported advantages of multi-agent security harnesses may shrink or invert when re-tested with the same model generation and a strong generic agent loop, a hypothesis testable on the same benchmark.
  • The Codex advantage over OpenCode and Pi may be partly explained by OpenAI-model integration maturity; a replication with non-GPT models could reveal whether the ranking is model-agnostic.
  • The benchmark repair of about 40 targets introduces a possible difficulty shift; an ablation on the original unrepaired targets would clarify whether plain agents benefit from the repairs.
  • The methodology transfers to other agentic security tasks, such as CTF-style challenges or bug bounty reproduction, where the same model/architecture confound exists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies the attribution of XBOW benchmark performance to security-specific architecture versus backbone model capability. It runs three default coding CLIs (Codex, OpenCode, Pi) under a fixed GPT-5 model on the 104-task XBOW benchmark with fresh flags, matched interfaces, and a fixed cost cap; finds Codex the strongest baseline; tests whether security-specific prompt variants help (they do not); compares the default Codex scaffold against published MAPTA and PentestGPT V2 results under the closest claimed model matches; and scales the same scaffold to GPT-5.2 and GPT-5.5. The central recommendation is that future penetration-testing-agent evaluations should report model-matched plain-agent baselines before attributing gains to architecture. The paper also releases a reproducibility artifact.

Significance. The baseline-first message is timely and valuable: it directly addresses a real confound in the recent autonomous-pentesting literature, and the controlled internal comparisons (RQ1/RQ2/RQ4) are designed carefully, with fresh flags, a fixed target interface, and budget-censored analyses. The released artifact with per-attempt tables is a genuine strength. If the paper limited its quantitative claims to its own runs, the contribution would be solid. However, the headline architecture-residual and 'exceeds 91%' claims depend on published scores from a different benchmark checkout, and the residual estimates are reported without uncertainty intervals. The methodological recommendation survives, but the specific quantitative architecture gap is not yet established.

major comments (3)
  1. [V-C, Eq. (1), Table VII, VII-D] The architecture residuals are computed as published architecture score minus plain-agent score, using MAPTA and PentestGPT V2 results that were not rerun in this infrastructure. Section VII-D states that roughly 40 of 104 targets required local image/build repairs, so the fork is not byte-identical to cross-paper benchmark checkouts; Table VIII shows different cost caps ($0.75 vs $0.206 average per task for GPT-5; $1.50 vs $0.18 median for GPT-5.2) and unknown inference settings. These differences shift both the absolute plain-agent scores and the residuals. The paper is transparent about this, but it still presents +9.6pp and +5.2pp as results rather than as illustrative cross-study observations. Please either rerun the published systems in the same harness, restrict the comparison to the subset of unchanged targets, or explicitly label these residuals as non-quantitative and move them
  2. [VI-B, Tables V and IX] Only two passes per condition are reported, and the paper itself notes 22 challenges flip between solved and unsolved across the two GPT-5 passes. Yet no confidence intervals or per-challenge variance estimates are provided for the key pass@1 means (67.3%, 79.8%, 92.3%) or for the residuals in Table VII. For a 104-task binary outcome, the single-pass binomial standard error at 67.3% is roughly 4.6 pp, so the 9.6 pp MAPTA residual and the 5.2 pp PentestGPT V2 residual are not clearly separated from noise. Please report bootstrap intervals, per-task regression estimates, or at least explicitly state that the residuals are not statistically distinguishable from zero under the current trial count. This is load-bearing because the quantitative force of RQ3 depends on these gaps.
  3. [V-D, VII-C, IX] The claim that plain Codex with GPT-5.5 'exceeds PentestGPT V2's 91% headline' compares a different, newer model (GPT-5.5) against Opus 4.5 and compares a two-pass union (95.2%) or pass@1 mean (92.3%) to a single-run published score. This is a model-scaling observation, not an architecture-attribution result, and presenting it in the conclusion as 'exceeding that headline' overstates the comparison. Reframe all such statements as illustrative of why current plain baselines matter, and separate them from the internal RQ1/RQ2/RQ4 results. This is especially important because the abstract and conclusion lean on the strong plain-agent performance to motivate the methodology.
minor comments (6)
  1. [II-A, II-B, References [8],[10]] 'OW ASP' should be 'OWASP' in the text and references.
  2. [II-C] 'A WE' should be 'AWE'.
  3. [VI-A, Eq. (1)] The sentence before Eq. (1), 'For each comparison systems, let P_s be ...' has a grammar error; also define P_s and C_s explicitly as percentages or fractions to avoid ambiguity.
  4. [Table VIII] The 'Run $' cell for PentestGPT V2 is blank; consider writing 'not reported' rather than leaving it empty.
  5. [V-C] 'The MAPTA residual is about ten percentage points' is imprecise; Table VII reports +9.6 pp. Use the exact number.
  6. [Figure 1] The caption states 'Plain P@2 is a two-pass union, not the residual baseline.' This is helpful, but consider adding a visual marker to distinguish union from single-run averages in the figure itself.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: plain-agent measurements and published comparisons are independent inputs, not derivations of each other.

full rationale

The paper's derivation chain is an empirical benchmark comparison rather than a formal derivation. RQ1/RQ2/RQ4 are internal controlled runs: Codex, OpenCode, Pi, prompt variants, and model generations are evaluated under stated conditions, and the reported pass@1/pass@2 values are direct observations, not quantities fitted to or defined by the conclusions. RQ3's architecture residual (Eq. 1: Delta_s = P_s - C_s) is an explicit difference between two independently obtained scores, so the residual is by construction the comparison being reported, not a hidden reuse of the outcome. The comparison systems MAPTA and PentestGPT V2 are external published results, not self-citations, and the paper explicitly disclaims that they were not rerun in its infrastructure (Section VII-D). That cross-infrastructure comparability issue is a validity threat (benchmark repairs, cost caps, unknown inference settings), not circularity. Public-benchmark contamination is likewise an external-generalizability concern, explicitly acknowledged in Section VII-D, and does not make any derived quantity equivalent to its input. No step satisfies the standard of exhibiting Eq. X = Eq. Y by construction or a fitted parameter renamed as prediction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The study is empirical; its load-bearing assumptions are about benchmark validity, cross-paper comparability, and containment. The cost caps and reasoning effort are hand-chosen controls, not fitted parameters. No new entities are introduced.

free parameters (2)
  • per-challenge cost cap = $0.75 (GPT-5), $1.50 (GPT-5.2), $5.00 (GPT-5.5)
    Chosen per model generation; caps spending but confounds model scaling with budget in RQ4. The GPT-5.2 cap was also enforced with a placeholder price estimate.
  • reasoning effort = medium for all GPT-5 rows
    Chosen by hand; not varied, but affects absolute solve rates and the comparability with published systems that may use higher thinking effort.
assumptions (4)
  • domain assumption XBOW benchmark solve rate is a meaningful measure of web-penetration-testing capability
    Used throughout; the paper itself notes it is not a substitute for real-world pentesting (Section II-E).
  • domain assumption Published MAPTA and PentestGPT V2 scores are accurately reported and are comparable to our runs despite different benchmark checkouts and settings
    Load-bearing for RQ3 architecture residuals; acknowledged as a threat in Section VII-D Cross-study attribution.
  • domain assumption Fresh random flags prevent the agents from solving via memorized answer strings
    Relied on in Section IV-B; the paper notes in Section VII-D that public challenge structure may still be known.
  • domain assumption The jail environment prevents the agents from accessing host or Docker sockets and confines them to the target
    Stated in Section IV-B; egress audit is limited (Section VII-D).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing." pith.science (2026). https://pith.science/paper/66E7RM4O

@misc{pith2026260713085,
  author       = {Pith},
  title        = {Pith review of: Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/66E7RM4O}},
  note         = {Machine review of arXiv:2607.13085}
}
read the original abstract

Recent autonomous penetration testing papers report high benchmark scores while adding multi-component security harnesses around frontier LLMs. Because these systems often change both architecture and backbone model, it is difficult to tell how much performance comes from the harness rather than from the underlying model. This paper presents a controlled study on the 104-task XBOW benchmark using default coding CLI agents as plain-agent baselines. We first run Codex, OpenCode, and Pi with the same GPT-5 model, budget, target interface, and scoring rule. This phase identifies the strongest same-model baseline and tests whether security-specific prompt variants improve its observed score. We then compare the default Codex scaffold with published MAPTA and PentestGPT V2 results under the closest available model matches. Finally, we repeat the plain-agent experiment with GPT-5.2 and GPT-5.5 to measure model scaling inside the same scaffold. The results show a mixed but practical picture. Specialised harnesses can add measurable benchmark lift and may improve cost efficiency, but plain coding agents already solve a large share of the benchmark; repeated plain-agent runs can match or exceed some published architecture scores in union coverage, and newer models substantially improve the same scaffold. Future evaluations should report model-matched plain-agent baselines before attributing benchmark gains to architecture design alone.

Figures

Figures reproduced from arXiv: 2607.13085 by the authors.

Figure 1
Figure 1. RQ3 matched-model score comparison. Plain P@2 is a two-pass [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 1 linked inside Pith

  1. [1]

    Multi-Agent Penetration Testing AI for the Web,

    I. David and A. Gervais, “Multi-Agent Penetration Testing AI for the Web,”arXiv:2508.20816, 2025

    arXiv 2025

  2. [2]

    What Makes a Good LLM Agent for Real-world Penetration Testing?

    G. Deng, Y . Liu, Y . Li, R. Yang, X. Xie, J. Zhang, H. Qiu, and T. Zhang, “What Makes a Good LLM Agent for Real-world Penetration Testing?”arXiv:2602.17622, 2026

    arXiv 2026

  3. [3]

    AWE: Adaptive Agents for Dynamic Web Penetration Testing,

    A. S. Jaswal and A. Baghel, “AWE: Adaptive Agents for Dynamic Web Penetration Testing,”arXiv:2603.00960, 2026

    arXiv 2026

  4. [4]

    Red-MIRROR: Agentic LLM-based Autonomous Penetration Testing with Reflective Verification and Knowledge-augmented Interaction,

    T. V . Khang, N. D. N. Khang, N. H. Khoa, D. T. T. Hien, V .-H. Pham, and P. T. Duy, “Red-MIRROR: Agentic LLM-based Autonomous Penetration Testing with Reflective Verification and Knowledge-augmented Interaction,” arXiv:2603.27127, 2026

    arXiv 2026

  5. [5]

    PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing,

    G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing,” inProc. 33rd USENIX Security Symposium, pp. 847–864, 2024

    2024

  6. [6]

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,

    J. Yang, C. E. Jimenez, O. Press, and K. Narasimhan, “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  7. [7]

    XBOW Validation Benchmarks,

    XBOW Engineering, “XBOW Validation Benchmarks,” https://github.com/xbow-engineering/validation-benchmarks, Accessed: Dec. 2024

    2024

  8. [8]

    OW ASP Top 10,

    OW ASP Foundation, “OW ASP Top 10,” https://owasp.org/Top10/, 2021

    2021

Show all 15 references
  1. [9]

    Burp Suite Web Vulnerability Scanner,

    PortSwigger, “Burp Suite Web Vulnerability Scanner,” https://portswigger.net/burp

  2. [10]

    OW ASP Zed Attack Proxy (ZAP),

    OW ASP Foundation, “OW ASP Zed Attack Proxy (ZAP),” https://www.zaproxy.org/

  3. [11]

    Nuclei: Fast and Customizable Vulnerability Scanner,

    ProjectDiscovery, “Nuclei: Fast and Customizable Vulnerability Scanner,” https://github.com/projectdiscovery/nuclei

  4. [12]

    sqlmap: Automatic SQL Injection and Database Takeover Tool,

    D. Stamatiset al., “sqlmap: Automatic SQL Injection and Database Takeover Tool,” https://sqlmap.org/

  5. [13]

    Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java,

    K. Li, S. Chen, L. Fan, R. Feng, H. Liu, C. Liu, Y . Liu, and Y . Chen, “Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java,” inProc. ESEC/FSE, 2023

    2023

  6. [14]

    Technical Guide to Information Security Testing and Assessment,

    K. Scarfone, M. Souppaya, A. Cody, and A. Orebaugh, “Technical Guide to Information Security Testing and Assessment,”NIST Special Publication 800-115, 2008

    2008

  7. [15]

    Introducing GPT-5.2,

    OpenAI, “Introducing GPT-5.2,” https://openai.com/index/introducing-gpt-5-2/, 2026. APPENDIXA REPRODUCIBILITYARTIFACT The accompanying GitHub artifact, https://github.com/ krodalabs/coding-agent-research-artifact, contains the material needed to inspect or regenerate the repor...

    2026

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.