{"id":"8c6841b2-a848-4067-8d0f-47cc9a533806","arxiv_id":"2505.10321","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AutoPentest, an autonomous GPT-4o multi-agent pentesting tool, completes 15-26% of subtasks on three Hack The Box machines, mildly outperforming manual ChatGPT-4o at API cost of $96.20.","lead":"This paper presents AutoPentest, a system that uses GPT-4o and LangChain to run black-box penetration tests with little human help, and tests it on three Hack The Box machines. The system and a manual ChatGPT-4o baseline both completed only 15 to 26 percent of benchmark subtasks, with AutoPentest slightly ahead on one machine and about $96 in API costs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Release date after cutoff does not rule out memorization: the three machines are solved via pre-cutoff public CVEs, so the 15-26% completion may reflect recall of known exploits rather than autonomous reasoning.","rationale":"The central claim is descriptive of an observed comparison, so the largest threat to its validity is that the measurement may be contaminated by the model's prior exposure to the attack paths. The paper's own Sections V and VII-B acknowledge only that release after the cutoff 'minimises' leakage; they do not test recall of the underlying CVEs. The three machines are rated easy precisely because they use well-known public exploits, and those exploits predate GPT-4o's training cutoff. Since the system also has RAG over offensive-security documents and live web search, failing to rule out retrieval of solution-specific content would weaken the 'black-box autonomy' reading of the results. The reader's weakest_assumption identifies the same issue, and I agree with it. Other concerns, such as small sample size and best-run reporting, are real but affect generalizability and statistical strength rather than the internal validity of the reported completion counts as directly as contamination does. The proposed probe is cheap, uses only the model API, and would distinguish recall of known exploit knowledge from genuine on-the-fly pentesting behavior.","tokens_in":21698,"tokens_out":5483,"duration_ms":56458,"concrete_test":"Run a contamination probe against the GPT-4o API: provide the exact nmap/service-discovery output observed at the start of each machine, with no tools, no RAG, and temperature 0, and ask for concrete next exploitation steps. Compare the generated steps to the official subtask lists in Appendix C-A; if the model independently names and details CVE-2023-46604 for Broker, the vm2 escape for Codify, or the Joomla configuration leak for Devvortex before any tool use, then the benchmark does not separate trained knowledge from autonomous discovery. A complementary check would be to rerun AutoPentest with Tavily search and the RAG vector database disabled on the same three machines and measure how much of the completion rate depends on external retrieval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V selects machines released after GPT-4o's October 2023 cutoff and says this 'minimises the chance of public solutions being used in the training data.' That mitigation is weaker than the paper's use of it suggests: all three machines are solved through CVEs that were public before the cutoff (Joomla CVE-2023-23752, Apache ActiveMQ CVE-2023-46604, vm2 CVE-2023-29199), and the subtask benchmark is derived from official write-ups whose techniques are likely represented in the model. The release-date argument therefore does not establish that the observed 15-26% reflects novel multi-step reasoning rather than recall of known exploit recipes. Moreover, the system's RAG corpus contains public offensive-security resources and the agents have live Tavily search; the paper's only safeguard is a manual check of search queries ('No such behaviour was ever found'), with no raw logs or query history published. If either training-data recall or live retrieval supplied the decisive exploit steps, the central claim about autonomous black-box capability is overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AutoPentest is a multi-agent penetration-testing system built on GPT-4o and LangChain. The paper reviews prior LLM-based pentesting work, identifies common evaluation issues, and presents a hierarchical agent architecture (planner, supervisor, and specialised workers) augmented with RAG and external tools. It evaluates AutoPentest on three Hack The Box machines released in November 2023, after the GPT-4o training cutoff, by decomposing official walkthroughs into subtasks (26, 10, and 27 subtasks for Devvortex, Broker, and Codify, respectively). The study compares best-run subtask completion rates of AutoPentest with a manually operated ChatGPT-4o baseline and reports token costs. The headline results are 15.4%, 20.0%, and 25.9% subtask completion for AutoPentest versus 15.4%, 20.0%, and 22.2% for ChatGPT-4o on the three machines, with a total AutoPentest API cost of $96.20.","tokens_in":21885,"tokens_out":9547,"duration_ms":99390,"significance":"The paper's related-work review is useful, particularly its emphasis on training-data leakage and on methodological omissions in prior studies. The implementation is open source, the prompts are listed in full in the appendix, and the cost accounting is transparent; these are genuine strengths for reproducibility. However, the empirical basis is narrow (three easy Linux machines, best-of-run scoring, no repeated-run statistics), and the claimed advantage over the baseline rests on a one-subtask difference on a single machine. The post-cutoff release date is an imperfect guarantee against leakage, and the search-log safeguard is not independently verifiable. If revised to address these issues, the paper would be a credible early empirical datapoint for autonomous LLM-based penetration testing; in its current form the accuracy and comparative claims are only partially supported.","major_comments":[{"comment":"The central comparison is based on the best run within a two-hour window on only three machines, with no repeated-run statistics. Section V states that an approach was restarted when no subtask had been completed for 20 minutes and that two to four runs were typical, but Section VI-B reports only the highest-scoring run. With this protocol, a one-subtask difference on Codify (25.93% vs. 22.22%) is within run-to-run noise, so the claim that AutoPentest 'slightly outperforms' ChatGPT-4o is not statistically supported. Please report all runs, per-run counts, and per-subtask completion, and base the comparison on a statistic that reflects the small sample.","section":"Section V and Section VI-B (Table III)"},{"comment":"The post-cutoff release date does not by itself rule out leakage of the route to completion. The three machines are solved via CVEs that were public before the October 2023 cutoff (Joomla CVE-2023-23752, Apache ActiveMQ CVE-2023-46604, vm2 CVE-2023-29199), and the subtask benchmark is derived from official write-ups. Moreover, Specialised Workers have live Tavily search and RAG access to public offensive-security documents (Sections IV-C and IV-D), so machine-specific walkthroughs could in principle be retrieved during a run. The manual check of search behaviour is a reasonable first step, but no raw query logs are published, so the safeguard is not independently verifiable. I request a direct probe, such as an ablation with web search and RAG disabled, or a test in which the model receives only generic CVE descriptions, and that the external-validity claims be calibrated to the strength of that evidence.","section":"Section V, Section VII-B, Appendix C-A"},{"comment":"RQ2 asks how accurately the system identifies and exploits vulnerabilities, but Table III only reports aggregate completed-subtask counts, with no separate identification metric and no per-subtask breakdown. As a result, the statement in Section VI-B that enumeration and identification 'is working well' is not directly supported by the tabulated data. A subtask-level table showing which steps were completed by each approach in each run would separate identification failures from exploitation failures and make the evaluation interpretable.","section":"Section VI-B and RQ2 (Section I-A)"}],"minor_comments":[{"comment":"The abstract states completion rates of '15-25 %', but Table III reports 25.93% for AutoPentest on Codify. The abstract should say 15-26% or be reconciled with Table III.","section":"Abstract and Table III"},{"comment":"The table uses the label 'ChatGPT-4.0' while the text and other tables use 'ChatGPT-4o'. The labels should be made consistent throughout.","section":"Table III"},{"comment":"The machine name is spelled 'Devortex' in the table, whereas the text and Table II use 'Devvortex'. Please correct the spelling.","section":"Table IV"},{"comment":"The cost comparison contrasts $96.20 in metered API usage with a $20 ChatGPT Plus subscription. These are different purchasing and usage models; the comparison should state explicitly that the subscription is rate-limited and that API costs scale with usage, aspects the text only partially acknowledges.","section":"Section VI-C"},{"comment":"The paper states that all online searches were manually checked and that no leakage behaviour was found, but no search-query history or agent logs are published. Releasing the raw logs or a query summary alongside the repository would substantially strengthen the verification of this claim.","section":"Section VII-B"}],"recommendation":"major_revision","confidential_remarks":"This is, in my view, a borderline major revision rather than a reject: the engineering contribution and the open-source release are real, and the leakage concern can plausibly be addressed by reformulating the claims and adding a focused ablation plus query-log publication. The single-author master's-thesis origin shows in the small scale of the evaluation; the paper would be considerably stronger if the claims were scaled to the evidence and if the baseline comparison were reported with run-level variation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AutoPentest is a honest incremental contribution: it open-sources a multi-agent LLM pentesting system and evaluates it on three HTB machines with better contamination hygiene than much of the prior work. The measured capability is modest—15-26% of subtasks, no full compromise—and the cost comparison in the paper actually favors the $20 ChatGPT subscription over the $96.20 API spend. The paper deserves a serious referee, but the evaluation is too small and the contamination argument is weaker than the paper's language suggests.\n\nWhat's new: the open-source implementation, the explicit comparison against a manually-operated ChatGPT-4o baseline, and the careful reporting of token costs per run. The architecture (planner, supervisor, specialized workers, RAG, Tavily search, NVD lookup) is a combination of known pieces, and the paper says so. The evaluation is better than most prior work because the machines were released after the model's stated cutoff, and the subtask benchmark comes from official walkthroughs, not from the model's own output. The paper also reports its own limitations in Section VII—task repetition, assumed shell context, unreported observations—which is more than most such papers do.\n\nThe soft spots, in order of importance. First, the cutoff argument only 'minimises' contamination, and the stress-test note is right: all three machines are solved through CVEs that were public before the cutoff (Joomla CVE-2023-23752, ActiveMQ CVE-2023-46604, vm2 CVE-2023-29199). Release date of the machine doesn't make the underlying exploit recipes novel to the model, and the agents have live Tavily search plus a RAG corpus of public offensive-security resources. The manual check of search queries is not a substitute for publishing the query logs. Second, the evaluation is three machines, best run within a two-hour window, no variance or error bars. That's enough for a pilot, not for a strong claim. Third, the baseline is human-in-the-loop: a person copies commands and outputs. That conflates human execution with ChatGPT's reasoning, though the paper tries to keep the human unbiased. Fourth, the cost comparison is incomplete: $96.20 vs $20 sounds bad for AutoPentest, and the paper's scaling argument (rate limits) is plausible but not a full cost-benefit analysis.\n\nNone of these are fatal. The paper's actual claims are modest and supported by the tables. The central result—both approaches complete 15-25% of subtasks, with AutoPentest slightly ahead—holds up. The paper is for people building LLM pentesting agents and for benchmark designers who care about contamination. It's a useful baseline and a dataset point. I would not cite it as evidence that autonomous pentesting works, but I would cite it as a cautious, reproducible negative result with open code.\n\nRecommendation: engage with it. Send it to peer review, ask for per-run results, more machines, and a leak check on the specific CVEs. The code is public, so the reproducibility bar is met; the claims need better statistical support.","headline":"Honest incremental contribution with open-source code and better evaluation hygiene than most prior work, but the contamination defense is weaker than the paper suggests and the evaluation is too small to carry much weight.","tokens_in":22392,"tokens_out":2335,"would_cite":true,"duration_ms":21785,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent GPT-4o system can autonomously execute black-box penetration testing steps, completing 15–26% of subtasks on three Hack The Box machines released after the model's training cutoff.","keywords":["LLM agents","penetration testing","GPT-4o","LangChain","retrieval-augmented generation","Hack The Box","vulnerability management","autonomous security testing"],"falsifier":"Prompt a no-tools GPT-4o instance with each machine name and ask how to get user and root access; if it can reliably reproduce the exact attack paths (the Joomla configuration disclosure on Devvortex, the ActiveMQ CVE on Broker, the vm2 escape on Codify) without any tool output, the post-cutoff novelty claim is falsified. Alternatively, run AutoPentest on a set of fresh easy Linux HTB machines that have no public writeups at all; a collapse in completion rates would indicate the reported 15–26% was largely memorisation.","tokens_in":21490,"feed_emoji":"🛡️","tokens_out":7282,"duration_ms":66837,"temperature":0.7,"pith_summary":"This paper tries to establish that a mostly autonomous multi-agent LLM system can carry out meaningful portions of a black-box penetration test. It reports that AutoPentest, built on GPT-4o and LangChain, completed 15–26% of the subtasks on three Hack The Box machines released after GPT-4o's training cutoff, slightly outperforming a manual ChatGPT-4o baseline that completed 22% on one machine and matched elsewhere. The author argues this matters because automating enumeration, vulnerability identification, and exploitation could make security testing cheaper and more frequent within vulnerability management. The measured API cost was $96.20 across all AutoPentest runs, against $20 for a month of ChatGPT Plus, with AutoPentest offering greater query throughput per hour.","feed_headline":"Autonomous GPT-4o agents complete up to 26% of pentest subtasks","feed_subtitle":"Multi-agent system beat human-piloted ChatGPT on post-cutoff Hack The Box machines for $96.20 in API calls.","key_machinery":"The load-bearing mechanism is a hierarchical multi-agent loop: a Planner agent produces a step-by-step plan, a Supervisor delegates the next step to a Specialised Worker, and the worker executes it with a fixed toolset, after which the Planner re-plans based on the worker's observations. The plan-and-solve idea comes from Wang et al., the multi-agent division of labour from the AutoGen concept, and the worker's context is enriched by retrieval-augmented generation (RAG) over a vector database of pentest documentation, so each step is informed by relevant OWASP, CWE, HackTricks, and PortSwigger material. Deterministic service discovery with nmap feeds NVD CVE lookups into the Planner's initial context, and temperature is set to 0 as recommended by related work on AutoAttacker. This machinery lets the system chain many small autonomous decisions into a longer attack path while keeping the LLM's context bounded.","core_discovery":"On the paper's own terms, the discovery is that a GPT-4o agent framework can autonomously run a black-box penetration test from an initial IP address through enumeration, vulnerability analysis, and attempted exploitation, and that it finishes a non-trivial minority of the attack chain. AutoPentest uses a Planner agent to build and repeatedly revise a high-level plan, a Supervisor that delegates each step, and Specialised Workers organised around OWASP Top 10 categories plus privilege escalation, all augmented with retrieval-augmented generation from a vector database and tools for shell commands, Python execution, headless browsing, web search, and NVD lookups. The paper's experiments on Devvortex, Broker, and Codify measure subtask completion rates of 15.38%, 20%, and 25.93% for AutoPentest, versus 15.38%, 20%, and 22.22% for a human manually executing ChatGPT-4o's suggested commands. The author interprets the results as showing that these systems are already good at enumeration and vulnerability identification but still struggle with reliable exploitation, and that the approach is viable enough that stronger future models and implementation work could make it a practical part of vulnerability management.","pith_inferences":["The cleanest test of the novelty claim would be to run AutoPentest against machines released after the cutoff that have never been publicly solved; the current design relies on the absence of writeups rather than proving their absence.","Comparing $96.20 in API costs against $20 in subscription cost understates the API route's advantage for repeated, unattended scans, since the subscription's hourly message cap is the binding constraint.","The task-repetition and assumed-shell-context failures suggest that state tracking, not model intelligence, is the next bottleneck; checkpointing and explicit shell-state summaries, both mentioned in the paper, are testable fixes."],"forward_implications":["If the results hold, autonomous LLM agents can already serve as a first-pass reconnaissance and vulnerability triage layer, flagging target services and known CVEs before a human tester takes over.","Because the biggest failures were in exploitation and long-horizon execution, fixing agent memory, shell-state tracking, and interactive tool use should yield larger gains than prompt tweaks.","The cost data imply that per-token API usage, while more expensive than a flat subscription in small experiments, scales past ChatGPT's per-hour message cap, making continuous or scheduled scans feasible.","Future evaluations should follow the same post-training-cutoff rule and add automated monitoring of web searches, since public solutions online would otherwise contaminate results."],"supporting_citations":[{"why":"Supplies the subtask-based benchmarking method: scoring an LLM's progress against essential steps extracted from public solution write-ups.","marker":"[13]"},{"why":"Motivates the Planner's plan-then-execute prompting, which the paper uses to decompose the pentest into high-level steps.","marker":"[19]"},{"why":"Provides the multi-agent conversation concept used to divide labour between Planner, Supervisor, and Specialised Workers.","marker":"[20]"},{"why":"Grounds the RAG design that augments each worker's context with retrieved pentest documentation.","marker":"[21]"},{"why":"Contributes the temperature-0 setting and an autonomous multi-agent attack system whose results are compared throughout.","marker":"[2]"},{"why":"Provides an autonomous privilege-escalation benchmark and success-rate comparisons that frame the paper's results.","marker":"[3]"},{"why":"Justifies using Hack The Box CTF machines as an evaluation platform by linking CTF skills to professional pentesting work.","marker":"[15]"},{"why":"Proposes the original ChatGPT pentesting subtask benchmark idea that the paper adapts for its evaluation.","marker":"[22]"}],"fun_headline_variants":["AutoPentest: Multi-agent LLM autonomously probes for vulnerabilities","GPT-4o agent system completes up to 26% of pentest subtasks","LLM agents auto-pentest for $96, outperform manual ChatGPT","Autonomous LLM pentester beats manual ChatGPT on HTB","AutoPentest: AI agents run black-box pentests on their own"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the assumption that the three Hack The Box machines, released in November 2023, are not effectively represented in GPT-4o's October 2023 training data; if the model has memorised public walkthroughs of these machines, the measured subtask completions reflect recall rather than novel reasoning.","fun_headline_variants_meta":{"raw":{"variants":["AutoPentest: Multi-agent LLM autonomously probes for vulnerabilities","GPT-4o agent system completes up to 26% of pentest subtasks","LLM agents auto-pentest for $96, outperform manual ChatGPT","Autonomous LLM pentester beats manual ChatGPT on HTB","AutoPentest: AI agents run black-box pentests on their own"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001568,"raw_usage":{"total_tokens":6293,"prompt_tokens":1012,"completion_tokens":5281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":5185}},"tokens_in":628,"tokens_out":5281,"duration_ms":41378,"temperature":1.0,"reasoning_tokens":5185,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:10:52.993739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Prompt a no-tools GPT-4o instance with each machine name and ask how to get user and root access; if it can reliably reproduce the exact attack paths (the Joomla configuration disclosure on Devvortex, the ActiveMQ CVE on Broker, the vm2 escape on Codify) without any tool output, the post-cutoff novelty claim is falsified. Alternatively, run AutoPentest on a set of fresh easy Linux HTB machines that have no public writeups at all; a collapse in completion rates would indicate the reported 15–26% was largely memorisation.","supporting_citations":[{"cited_title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,","cited_arxiv_id":null,"evidence_quote":"Grounds the RAG design that augments each worker's context with retrieved pentest documentation."},{"cited_title":"Understanding Hackers' Work: An Empirical Study of Offensive Security Practitioners","cited_arxiv_id":"2308.07057","evidence_quote":"Justifies using Hack The Box CTF machines as an evaluation platform by linking CTF skills to professional pentesting work."},{"cited_title":"The Convergence of AI and Cybersecurity: An Examination of ChatGPT’s Role in Penetration Testing and its Ethical and Legal Implications,","cited_arxiv_id":null,"evidence_quote":"Proposes the original ChatGPT pentesting subtask benchmark idea that the paper adapts for its evaluation."}],"review_version":1}