{"id":"e6d5da1d-ae12-4170-b049-078140b06e4b","arxiv_id":"2508.20816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-agent LLM system with tool execution and proof-of-concept validation claims 76.9% success on the XBOW web-pentesting benchmark and 19 real-world vulnerabilities across six open-source projects.","lead":"MAPTA is a multi-agent AI system that autonomously hunts web vulnerabilities, solving 80 of 104 XBOW hacking challenges and finding 19 issues in six popular open-source apps. Its reported cost, about $0.12 per CTF challenge and $3.67 per app review, makes continuous automated security testing look affordable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 76.9% success rate rests on 43 manually repaired XBOW images, but no equivalence analysis is provided; without diff-level evidence the reported score may measure a different benchmark than XBOW's 84.6%.","rationale":"The reader's weakest_assumption identifies exactly the concern I consider most load-bearing: the integrity of the modified XBOW benchmark. The 76.9% headline and the comparison to XBOW's 84.6% are only meaningful if the 43 manually fixed Docker images preserve the original challenges' vulnerability semantics and difficulty. The paper provides no diff, no equivalence analysis, and no separate statistics for the repaired subset, so the claim is conditional on an unverified assumption. I considered other concerns—single-run statistics without variance, post-hoc early-stopping thresholds, and anonymized real-world findings that cannot be independently checked—but none is as fundamental as benchmark integrity, because all CTF-based conclusions inherit it. The paper otherwise has real strengths: open-source code, detailed cost accounting, per-challenge tool/token breakdowns, and acknowledgement of limitations. I am not moving the verdict because the reader already marked the paper CONDITIONAL; my check would either validate the repaired benchmark and allow acceptance, or require revising the headline numbers. Therefore the appropriate verdict remains CONDITIONAL, and my read does not change the reader's verdict.","tokens_in":16209,"tokens_out":4709,"duration_ms":51971,"concrete_test":"Publish a line-level diff between the upstream XBOW validation-benchmarks repository and the authors' fork for all 43 repaired images, classifying each changed file as dependency/environment (Dockerfile, package manifest, version pin) versus application source/route/flag file. If any application-source diff exists, independently re-run those challenges on an image reconstructed from the original application source where feasible and report per-challenge outcomes. In all cases, report success rates separately for the 61 untouched and 43 repaired challenges; if the repaired-subset success rate differs substantially from the untouched subset, recompute the aggregate 76.9% and qualify the XBOW comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is MAPTA's 76.9% (80/104) success on the XBOW benchmark and the comparison to XBOW's reported 84.6%. However, Section 3 states that '43 of the original 104 XBOW Docker images required manual fixes due to deprecated software versions,' and the Open Science section describes a forked 'updated XBOW 104 Challenge Evaluation Framework.' The paper gives no pre/post equivalence analysis: no diffs of the 43 fixes, no classification of whether changes were limited to dependency/version updates or touched application logic, no check that flags, routes, and exploit paths were preserved, and no separate success-rate reporting for the 61 untouched versus 43 repaired challenges. If a fix altered a controller, disabled an unintended hardening mechanism, changed a flag location, or made a challenge easier/harder, the aggregate 76.9% and the comparison to XBOW's own run on the original images are not measuring the same object. This is an evidence gap, not an accusation; the authors' engineering effort is commendable, but the load-bearing assumption of semantic equivalence is simply asserted. Every downstream conclusion—per-category success, cost per challenge, early-stopping thresholds, and the 'competitive with commercial' claim—inherits this uncertainty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes MAPTA, an LLM-based multi-agent system for autonomous web penetration testing. Three roles (Coordinator, Sandbox, Validation) share a per-job Docker container, and a bounded loop performs hypothesis synthesis, tool execution, PoC assembly, and validation. Evaluation is in two modes: blackbox CTF on the 104-challenge XBOW benchmark, where MAPTA reportedly solves 80 challenges (76.9%) with total LLM cost $21.38 and median $0.117 per challenge; and whitebox assessment of ten open-source repositories, where it reports 19 vulnerabilities with 10 pending CVEs. The paper also reports negative correlations between resource usage and success and proposes early-stopping thresholds of about 40 tool calls, $0.30, and 300 seconds. Code and a modified XBOW framework are released; 43 of the 104 XBOW Docker images were manually repaired to fix deprecated software versions.","tokens_in":16618,"tokens_out":7155,"duration_ms":68484,"significance":"The paper has real strengths: open-source release, a binary flag-based success metric that avoids false-positive ambiguity, detailed token-level cost accounting, and a Validation agent that executes candidate PoCs. If the empirical claims are supported, MAPTA would be a useful open baseline for autonomous web pentesting and a valuable data point on cost-performance tradeoffs. The reported 76.9% on XBOW is appealing, but it currently rests on an unverified modified benchmark and a single run per challenge; the cost-threshold analysis is in-sample; and the real-world findings are anonymized and unverifiable. The potential contribution is significant, but the evidence as presented does not yet justify the headline comparison or the actionable early-stopping claims.","major_comments":[{"comment":"The central claim of 76.9% (80/104) is computed on a benchmark in which 43 of the 104 XBOW Docker images were manually repaired. No diff-level equivalence analysis is provided. Without evidence that the fixes preserve vulnerability semantics, difficulty, flags, and exploit paths, the comparison with XBOW's 84.6% and all downstream per-category and cost conclusions are not measuring the same object. Please provide (a) a per-fix diff or classification (dependency-only vs application-logic changes), (b) success rates separately for the untouched and repaired subsets, and (c) a validation protocol showing preserved exploitability on the repaired images.","section":"Section 3 (Benchmark modification)"},{"comment":"The thresholds of about 40 tool calls, $0.30, and 300 seconds are derived from the 95th percentile of successful challenges on the same 104-challenge data and then presented as practical early-stopping thresholds. This is in-sample fitting rather than prediction. Please re-label these as descriptive statistics or validate them out-of-sample (e.g., held-out challenges, repeated runs, or a separate benchmark); as written, the actionable contribution is circular.","section":"Section 3.3 (Early-stopping thresholds)"},{"comment":"The evaluation appears to consist of a single run per challenge. Given the stochastic LLM controller, the 76.9% point estimate and category-level rates (e.g., 57% XSS, 0% blind SQLi) carry no uncertainty. Please report multiple runs for at least a random subset, or provide confidence intervals, and clarify whether the reported cost/time values come from the same single run. The p<0.001 correlations also assume independent observations, which a single-run design does not establish.","section":"Section 3.2, Table 2 (Run variance)"},{"comment":"The real-world findings are anonymized (OSN-XX), with no CVE identifiers, patch status, or disclosure artifacts for the 10 findings said to be under CVE review. This makes the practical-impact claims unverifiable. Please provide a reproducible disclosure package (e.g., redacted PoCs after maintainer response) or clearly separate verified and pending findings, and state the verification status of each reported vulnerability.","section":"Section 4.1, Table 3 (Real-world validation)"},{"comment":"The competitive-with-commercial claim rests on XBOW's self-reported 84.6% from a blog post with no published protocol, and on the modified 104-image benchmark. Even if the image fixes are shown to be semantically neutral, the evaluation protocols may differ. Please add a direct comparison run under a documented protocol on the same images, or qualify the claim as comparable to XBOW's reported result under a partially updated benchmark.","section":"Section 3.4 (XBOW comparison)"},{"comment":"The claim that MAPTA's solutions are genuine discovery because no online CTF solutions were found is an argument from absence, with no search methodology or contamination probe. Since GPT-5 training data may include public benchmark repositories, please provide a contamination check (e.g., query the model with challenge descriptions and tools disabled, or test for exact flag/solution memorization) and disclose the search procedure; otherwise remove or weaken the claim.","section":"Section 3 (Memorization)"}],"minor_comments":[{"comment":"Typo: 'OW ASP' appears in Section 2.3, Section 5.1, and reference [17]; should be 'OWASP'.","section":"Throughout"},{"comment":"CTF mode uses a single agent, while the abstract and introduction emphasize multi-agent coordination and a Validation agent. Clarify how end-to-end PoC validation applies to the CTF results, since validation reduces to flag extraction in that mode.","section":"Section 2.4"},{"comment":"The claim of being the first open-source multi-agent penetration testing AI should be carefully scoped relative to open-source projects such as PentestGPT; clarify the precise novelty (multi-agent vs multi-stage, web-focused, validation-oriented).","section":"Section 1.1 / Related Work"},{"comment":"The Sankey diagram is visually dense and difficult to read. Provide a companion table with exact counts per vulnerability category and outcome.","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising open-source system and a well-motivated cost-accounting methodology, but the headline comparison to XBOW is not yet supported because of the unverified modified benchmark, single-run evaluation, and in-sample threshold derivation. These gaps are addressable with additional analysis (diff reports, repeated runs, contamination checks, disclosure artifacts), so I recommend major revision rather than rejection. The promotional framing in the abstract and contributions should also be tightened to match the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take: MAPTA is a competent piece of systems work with the most detailed cost model I've seen for LLM pentesting, but I don't trust the 76.9% headline until the 43 repaired XBOW images are shown to be equivalent to the originals.\n\nThe validation agent that forces every finding through a concrete PoC is a real improvement over earlier agents, and the token-level accounting across 104 challenges is genuinely useful. The fact that they're releasing code and fixing the deprecated benchmark images is a community contribution. I'd give them credit for that.\n\nThe soft spots are concentrated in the benchmark methodology. The paper says 43 of 104 Docker images required manual fixes, then compares MAPTA's score on the repaired set to XBOW's score on the original set. There is no diff-level analysis, no classification of whether fixes touched application logic, no separate success rate for the 61 untouched versus 43 repaired challenges. That's the load-bearing assumption, and it's simply asserted. The fixes might well be trivial dependency bumps, but we can't tell. This single issue makes the 'competitive with commercial' claim unverifiable as stated.\n\nA few other things: each challenge was run once, so there's no variance estimate. The early-stopping thresholds (40 tool calls, $0.30, 300 seconds) are computed from the same dataset they're then presented as actionable—they're fitted values, not validated predictions. The real-world findings are anonymized, so nobody can independently confirm them, and the whitebox mode is a different task from the blackbox CTF, so the two evaluations shouldn't be blended in the abstract. The correlation analysis is okay, but inferring difficulty directly from resource use is a stretch. And the 'first open-source' claim is arguable given prior open-source multi-agent systems like PentestGPT.\n\nWho is this for? Security people who want to see what an open-source LLM agent can do, and researchers working on agent tool orchestration. It deserves a serious referee—the architecture and cost model are worth discussing—but a referee should push hard for the equivalence analysis and ideally for at least a second run of the benchmark. If the XBOW fixes check out, this becomes a solid empirical paper; if not, the headline number shouldn't stand.\n\nRecommendation: send it to peer review with the expectation that the benchmark integrity issue is addressed before acceptance.","headline":"Solid engineering and a useful cost model, but the headline 76.9% is not trustworthy until the 43 repaired XBOW images are shown to preserve the original challenges.","tokens_in":17004,"tokens_out":3567,"would_cite":false,"duration_ms":32961,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source multi-agent LLM system solves 80 of 104 web pentest challenges, matching commercial performance at a median $0.12 per target.","keywords":["multi-agent systems","LLM agents","web penetration testing","XSS","SQL injection","SSRF","cost analysis","XBOW benchmark"],"falsifier":"Compare MAPTA's success rate on the 43 repaired challenges against the same challenges reproduced from pristine originals (or archived source) with identical flags and exploit paths; any mismatch in success or flag value would show the fixes altered the benchmark. Alternatively, a manual audit of pre- and post-fix differential behavior (route responses, error messages, flag location) that finds a behavioral change in even a handful of the 43 images would invalidate the headline comparison.","tokens_in":16170,"feed_emoji":"🛡️","tokens_out":4800,"duration_ms":43202,"temperature":0.7,"pith_summary":"MAPTA is an open-source multi-agent system that combines an LLM coordinator, sandboxed tool-execution agents, and a separate validation agent to autonomously find and prove web vulnerabilities. On the 104-challenge XBOW benchmark it solves 80 challenges (76.9%), close to the commercial system's reported 84.6%, at a median cost of $0.117 per challenge. It achieves perfect scores on SSRF and misconfiguration, strong scores on injection and authorization, and weak scores on XSS and blind SQL injection. The paper argues this shows tool-grounded, verify-by-execution multi-agent design is a viable path to continuous, cost-predictable security assessment.","feed_headline":"AI pentest agent: 76.9% success, $0.12 per challenge","feed_subtitle":"Multi-agent LLM design cracks 80 of 104 benchmark targets and uncovers real-world critical bugs for $3.67 per app.","key_machinery":"The load-bearing mechanism is the three-role agent architecture with a single-pass controller: a Coordinator agent reasons about attack paths and delegates through tools, one or more Sandbox agents execute commands and Python in a shared per-job Docker container, and a Validation agent consumes each candidate finding and verifies it by concrete execution, returning pass/fail evidence (a captured flag in CTF mode, side-effect evidence in real-world mode). The per-job container gives stateful reuse of artifacts across subtasks while keeping LLM contexts isolated, and the Validation step is what converts theoretical findings into verified exploits.","core_discovery":"The central claim is that a multi-agent LLM architecture with mandatory end-to-end exploit validation can perform autonomous black-box web penetration testing at near-commercial success rates while remaining fully open and reproducible. MAPTA separates strategic planning (Coordinator), tactical execution (Sandbox agents sharing one Docker container per job), and verification (Validation agent that executes candidate proofs-of-concept and returns pass/fail evidence). This design turns vulnerability assessment from hypothesis generation into empirical validation, eliminating most false positives. The paper supports the claim with an 80/104 success rate on the 104-challenge benchmark, a $21.38","pith_inferences":["The negative resource-success correlation suggests the agent is not 'persisting' but thrashing: adding a second, differently configured agent (e.g., a payload-generation specialist for XSS) might convert some of the 24 failures without raising median cost.","The 43 repaired Docker images are unvalidated; a reader wanting to trust the 76.9% comparison should require before/after equivalence checks (same flag, same exploit path, same intended difficulty) on those images.","If the real-world results hold up under CVE review, open-source projects with modest budgets could obtain near-commercial-grade continuous security assessment, shifting vulnerability discovery from scheduled audits toward always-on monitoring.","Blind SQL injection (0/3) is likely a timing-analysis gap, not a reasoning gap; existing blind-injection techniques (conditional delays) could be added as a specialist tool inside sandbox agents."],"forward_implications":["Early stopping at roughly 40 tool calls, $0.30, or 300 seconds would prune most failing attempts, cutting wasted spend on targets the agent cannot crack.","Operating cost of $0.073 per successful benchmark challenge and $3.67 per open-source app assessment makes continuous, scheduled pentesting economically plausible at scale.","Perfect SSRF and misconfiguration scores plus 83% authorization success suggest the strongest near-term use is access-control and server-side request forgery review, not XSS.","Findings are ready-to-use proofs-of-concept with evidence, not pattern-match alerts, so organizations can feed them straight into triage or CVE processes.","Because 43 of the 104 challenge images had to be repaired, the benchmark's longevity depends on community maintenance of those fixes."],"supporting_citations":[{"why":"Supplies the 104-challenge benchmark and the commercial system whose reported 84.6% success is the comparison target.","marker":"[25]"},{"why":"Prior LLM pentesting system whose evaluation methods and cost reporting this paper extends; its benchmark was planned but unavailable.","marker":"[8]"},{"why":"Two-stage LLM framework coupling discovery with remediation, used as prior state of the art that motivates verification-by-execution.","marker":"[13]"},{"why":"Establishes the reasoning-plus-acting loop that underpins the coordinator's tool use and feedback-driven exploration.","marker":"[28]"},{"why":"Provides the motivating statistic that up to 40% of AI-generated code contains vulnerabilities, framing the scalability crisis.","marker":"[21]"},{"why":"Public description of the commercial system, cited as lacking reproducibility and as the source of the claimed 84.6% baseline.","marker":"[10]"},{"why":"Justifies the exclusive use of GPT-5 for the evaluation based on reported performance for autonomous penetration testing.","marker":"[26]"}],"fun_headline_variants":["Multi-agent AI pentest: 77% success, $0.12 per challenge","AI agent team passes 80 of 104 web pentest challenges","LLM orchestration reaches 76.9% success in web security","Autonomous web pentest: 77% success, $21 total cost","Multi-agent AI finds critical bugs for $3.67 per app"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The 43 repaired Docker images must preserve the original challenges' behavior and difficulty; if the fixes changed how a vulnerability can be exploited, the 76.9% score and the comparison to the commercial baseline no longer measure the same benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent AI pentest: 77% success, $0.12 per challenge","AI agent team passes 80 of 104 web pentest challenges","LLM orchestration reaches 76.9% success in web security","Autonomous web pentest: 77% success, $21 total cost","Multi-agent AI finds critical bugs for $3.67 per app"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000765,"raw_usage":{"total_tokens":3257,"prompt_tokens":799,"completion_tokens":2458,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":2360}},"tokens_in":543,"tokens_out":2458,"duration_ms":17276,"temperature":1.0,"reasoning_tokens":2360,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:46:38.255573+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare MAPTA's success rate on the 43 repaired challenges against the same challenges reproduced from pristine originals (or archived source) with identical flags and exploit paths; any mismatch in success or flag value would show the fixes altered the benchmark. Alternatively, a manual audit of pre- and post-fix differential behavior (route responses, error messages, flag location) that finds a behavioral change in even a handful of the 43 images would invalidate the headline comparison.","supporting_citations":[{"cited_title":"Xbow validation bench- marks","cited_arxiv_id":null,"evidence_quote":"Supplies the 104-challenge benchmark and the commercial system whose reported 84.6% success is the comparison target."},{"cited_title":"Pentestgpt: Evaluating and harnessing large language models for automated pene- tration testing","cited_arxiv_id":null,"evidence_quote":"Prior LLM pentesting system whose evaluation methods and cost reporting this paper extends; its benchmark was planned but unavailable."},{"cited_title":"Penheal: A two-stage llm framework for automated pentesting and optimal remediation","cited_arxiv_id":null,"evidence_quote":"Two-stage LLM framework coupling discovery with remediation, used as prior state of the art that motivates verification-by-execution."},{"cited_title":"React: Synergizing reasoning and acting in language models, 2022","cited_arxiv_id":null,"evidence_quote":"Establishes the reasoning-plus-acting loop that underpins the coordinator's tool use and feedback-driven exploration."},{"cited_title":"Asleep at the key- board? assessing the security of github copilot’s code contributions","cited_arxiv_id":null,"evidence_quote":"Provides the motivating statistic that up to 40% of AI-generated code contains vulnerabilities, framing the scalability crisis."},{"cited_title":"Ai agents for offsec with zero false positives, 2025","cited_arxiv_id":null,"evidence_quote":"Public description of the commercial system, cited as lacking reproducibility and as the source of the claimed 84.6% baseline."},{"cited_title":"Gpt-5 performance analysis for autonomous penetration testing","cited_arxiv_id":null,"evidence_quote":"Justifies the exclusive use of GPT-5 for the evaluation based on reported performance for autonomous penetration testing."}],"review_version":1}