{"id":"2b835915-e055-4e8f-81c2-af80ca374ddf","arxiv_id":"2412.12745","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ADAPT, a MAPE-K-based automated penetration testing tool, autonomously compromised Metasploitable2, Metasploitable3, and a realistic virtual lab network using pre-populated attack and scan repertoires.","lead":"This paper presents a formal model of automated penetration testing and a self-adapting architecture, implemented in a tool called ADAPT that runs scans, exploits, and post-exploitation steps without human intervention. The tool successfully compromised intentionally vulnerable systems and a realistic training network, suggesting that more of the labor-intensive security testing workflow can be automated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §7 claim of full automation is undercut by §6.2's target-tailored repertoires; an out-of-sample evaluation is required to support it.","rationale":"The reader's weakest-assumption analysis is spot on: the evaluation's success is enabled by hand-crafted, target-specific repertoires. My stress-test does not surface a different, more severe flaw. The formalization in Section 4 is coherent and maps cleanly onto the MAPE-K architecture; the implementation appears genuine and the authors are candid about the repertoire's role and about not inventing new attack techniques. The withheld intrusive plugins and the private VM network limit reproducibility, but the decisive gap is external validity: the Section 7 claim generalizes from an experiment in which the authors already knew the targets' vulnerabilities. This is not an accusation of dishonesty; it is an evidence-quality issue. A held-out evaluation with a non-tailored repertoire would settle whether the tool actually automates the discovery-and-exploitation loop or merely executes a pre-solved plan. Until such evidence is produced, CONDITIONAL remains the right verdict, with the condition being the out-of-sample demonstration and full disclosure of the repertoires.","tokens_in":25914,"tokens_out":8878,"duration_ms":87474,"concrete_test":"Out-of-sample evaluation: take a vulnerable target not used in the paper (e.g., a VulnHub VM or a newly built lab network with different OSes and services) and run ADAPT with a fixed repertoire built solely from generic, publicly documented attack techniques — the same set used for Metasploitable2/3 — with no additions or parameter changes informed by the new target. Record whether ADAPT achieves full compromise (all intended flags/root access). If it does, the Section 7 claim survives; if it fails because a required exploit or parameter is absent from the generic repertoire, then the Section 6.3 success was attributable to target-specific repertoire pre-population. To make the test reproducible, the full repertoire and its mapping to CVEs/techniques must be published.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 7 — that ADAPT can 'automate all parts of the penetration testing process including planning, dynamic at runtime decision making and attack automation' against realistic environments — rests on the evaluation in Section 6.3, whose success depends on the repertoires described in Section 6.2. Section 6.2 states: 'we opted to use repertoires that contain the required capabilities to successfully complete the individual exploits against the hosts of the VM network.' This is an explicit admission that the attack and scan libraries were pre-populated with exactly the exploits needed for the specific targets, based on the authors' prior knowledge of those systems. Section 5.1 confirms the tool 'automate[s] the selection and application of previously known, generic and widely applicable attacks rather than invent new attack techniques.' Consequently, the hard target-specific knowledge — which vulnerabilities exist and which exploit/parameter combination succeeds — was supplied by the test designers, not discovered by ADAPT. This is in tension with Section 2's claim that the approach 'requires no knowledge of the system architecture, vulnerabilities or functionality.' At runtime ADAPT indeed receives only the network range and flag identifiers, but constructing the evaluation repertoire required that knowledge. Because the VM network is the authors' own course lab, the single successful run demonstrates orchestration of a hand-selected exploit set, not autonomous penetration testing of an arbitrary realistic environment. The load-bearing assumption is therefore that a generic, pre-existing repertoire can achieve the same results; the paper provides no evidence for this.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes penetration testing at the architectural level using a security-informed architecture and a labeled transition system, and it proposes a generic autonomic-computing architecture (MAPE-K) with utility-based decision making for automated penetration testing. The authors implement this architecture in the ADAPT tool and report successful, unattended penetration tests against Metasploitable2, Metasploitable3, and a university lab network with 11 flags. The central claim is that ADAPT automates all parts of the penetration testing process, including planning, runtime decision making, and attack automation, against realistic environments.","tokens_in":26142,"tokens_out":3351,"duration_ms":32180,"significance":"If the evaluation were fully convincing, the paper would be a useful step toward end-to-end automated penetration testing: the architectural formalization is reasonably general and appears internally consistent, the MAPE-K instantiation is sensible, and the implementation is released as open source. The exploitation graphs and runtime statistics are valuable qualitative evidence that the architecture can coordinate a real scanner/exploitation stack. However, the principal empirical claim is weakened by a circularity in the evaluation design: the attack and scan repertoires were deliberately populated with the exact capabilities needed to compromise the targets, so the reported success demonstrates orchestration of a hand-selected exploit set rather than autonomous discovery or planning against unknown systems. This does not invalidate the formal or architectural contributions, but it materially narrows what the evaluation establishes.","major_comments":[{"comment":"The central claim in Section 7 that ADAPT is 'the fist tool that is able to automate all parts of the penetration testing process including planning, dynamic at runtime decision making and attack automation that can be employed against realistic environments' is not supported by the evaluation as designed. Section 6.2 explicitly states: 'we opted to use repertoires that contain the required capabilities to successfully complete the individual exploits against the hosts of the VM network.' Because the repertoires were constructed using prior knowledge of the target systems, the successful runs in Section 6.3 show that ADAPT can select and execute a prespecified set of known exploits, not that it can operate against previously unknown systems. To support the full-automation claim, the paper needs an out-of-sample evaluation in which the repertoires are not tailored to the targets, or at least an ablation that separates the contribution of repertoire coverage from the contribution of the decision-making engine.","section":"Section 6.2, Section 7"},{"comment":"The related-work section claims that the approach 'requires no knowledge of the system architecture, vulnerabilities or functionality.' This is accurate only if 'the approach' refers strictly to the runtime operation of ADAPT after the repertoires and utility parameters have been created. Section 6.2 shows that the designers needed substantial target-specific knowledge to build the attack and scan repertoires: the repertoires had to contain the specific techniques, parameters, and ordering that would compromise the hosts. The paper should clarify this boundary and rephrase the claim so it does not imply that the overall method, including its configuration, requires no knowledge of the target. As written, the claim in Section 2 is in direct tension with the evaluation setup in Section 6.2.","section":"Section 2, Section 6.2"},{"comment":"For the VM-network case study, the paper reports a single run of the tool (the exploitation graph in Figure 6 is described as 'a single run'). No repeated runs at the same concurrency threshold are reported, so the success rate and variance across runs are unknown. Since the same repertoires are used and the utility scores converge in Figure 8, the authors should state how many runs were performed and, if only one, acknowledge that the demonstration is a single data point.","section":"Section 6.3"}],"minor_comments":[{"comment":"There is a typo in the conclusion: 'the fist tool' should be 'the first tool.'","section":"Section 7"},{"comment":"In the definition of a strategy, the notation S_N : S_M × I → T uses I, but I was previously defined as a set of interfaces in Definition 2; the intended domain is the set of penetration test states, which is denoted M in Definition 5. Please correct this notation for clarity.","section":"Definition 5 (Section 4.2)"},{"comment":"The text after Definition 4 reads 'After a a scan's completion'; the duplicated article is a typo.","section":"Definition 4 (Section 4.1)"},{"comment":"The name 'Metaspoitable3' appears once in the setup text; it should be 'Metasploitable3' for consistency with the rest of the paper.","section":"Section 6.1"},{"comment":"In the VM-network exploitation graph, one node is labeled 'H4∶1'; this appears to be a typo for 'H4∶I1'.","section":"Figure 6"},{"comment":"The statement that the concrete implementation 'outperforms them in the number of services exploited, running time and scalability' with respect to LLM-based tools is not substantiated by any experimental data or quantitative comparison in the evaluation section. Since no comparative experiments are reported, this claim should be removed or supported with a dedicated comparison.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The formalization and architecture are likely solid contributions, but the evaluation's circularity is a serious concern for a journal that will judge the paper on its empirical claims. If the authors can add an out-of-sample evaluation or substantially soften the claims in Sections 2 and 7, the paper could be acceptable. The incomplete plugin release (intrusive plugins withheld) also limits reproducibility; the authors should at least provide a detailed environment replication package for the non-intrusive parts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a solid systems paper with an overclaimed headline. The MAPE-K instantiation for automated pentesting is real, and the evaluation against real VMs is valuable, but the §7 claim that ADAPT automates 'all parts' of pentesting is not supported by the §6.2 admission that the repertoires were populated with exactly the exploits needed. The tool is orchestrating a hand-curated attack library, not discovering attacks.\n\nWhat's genuinely new: the end-to-end architecture, the runtime post-exploitation decisions, and the working implementation. The formalization is a labeled transition system plus strategy, which is standard for RL/planning, but the paper is honest about the mapping. The exploitation graphs are a nice way to show the tool's behavior. The evaluation on Metasploitable2/3 and their own lab network is a step beyond simulation-only results.\n\nThe soft spot is the evaluation's circularity. Section 6.2 explicitly says the repertoires were chosen to contain the required capabilities. So the success demonstrates that the orchestrator can chain known attacks and adapt at runtime, which is real, but it does not demonstrate autonomous discovery of vulnerabilities. The §2 statement that the approach 'requires no knowledge of the system architecture, vulnerabilities or functionality' is only true at runtime, not for the construction of the repertoire. That's a meaningful gap between the marketing and the method.\n\nA second issue: the VM network is their own course lab, and the code is open only for the non-intrusive parts. So independent replication is hard.\n\nThat said, the paper is not sloppy. It is internally consistent, and the limitation is stated, albeit buried. The weights in the utility functions are hand-tuned, but that is typical for this kind of system.\n\nWho is this for? Researchers working on automated pentesting, especially those building on MAPE-K or trying to move from simulation to real systems. It deserves a serious peer review, but I would push for major revisions: an out-of-sample evaluation (e.g., on a target the authors did not pre-analyze) and a rewritten §7 that matches §6.2's scope.\n\nRecommendation: engage with it, but with the claims trimmed.","headline":"A real architecture and implementation, but the full-automation claim is undercut by the hand-curated repertoires; worth reviewing with revisions.","tokens_in":26733,"tokens_out":2457,"would_cite":true,"duration_ms":22822,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that its self-organizing ADAPT tool can automate the whole penetration-testing process—planning, runtime decisions, and attack execution—and reports capturing all 11 flags and root on every host in a realistic VM network.","keywords":["Automated Penetration Testing","Attack Planning","Attack Automation","Formal Model","Autonomic Computing","MAPE-K","Utility-Based Decision Making","Security-Informed Architecture"],"falsifier":"A concrete check is to watch a full ADAPT run against the VM network and record every decision point: if any target choice, exploit selection, parameter value, or pivot is supplied or corrected by a human after the run starts, the 'all parts automated' claim fails. The paper reports no such interventions, and the run logs would settle it; a stronger test would strip from the repertoire every exploit tied to the lab's known CVEs and observe whether any path to the 11 flags remains.","tokens_in":25682,"feed_emoji":"🛡️","tokens_out":10137,"duration_ms":81938,"temperature":0.7,"pith_summary":"Penetration testing today is done largely by scarce human experts, and while many tools automate attack planning or individual exploits, the paper argues that no existing approach covers the whole process against realistic systems. This paper tries to change that by formalizing penetration testing at an architectural level—modeling the target as a security-informed architecture of components and interfaces, and the test as a labeled transition system with an explicit strategy—so the problem can be studied independently of any particular planner or learning method. It then instantiates that formalism in ADAPT, an autonomic tool built on a MAPE-K loop, and claims that ADAPT can run a complete penetration test without human interaction when given a sufficient repertoire of scans and attacks. The reported results are concrete: against Metasploitable2, Metasploitable3, and a realistic virtual network used for ethical-hacking training, ADAPT reached the intended compromise goals, including all 11 flags and root on every host in the VM network. If true, this establishes that end-to-end automation of penetration testing is achievable with current tooling, shifting the practical bottleneck from runtime expertise to the construction of the attack repertoire.","feed_headline":"Automated pentest tool cracks all 11 flags without a human","feed_subtitle":"An autonomic MAPE-K architecture plans, decides, and executes an end-to-end pentest on real target networks.","key_machinery":"The load-bearing mechanism is the pairing of a formal model with a control loop. Formally, the target is a security-informed architecture and the penetration test is a labeled transition system plus a strategy; the state records which capabilities the tester holds and which components and interfaces are known, the transition relation is populated by attacks and scans, and the strategy's next-move function $S_N$ selects what to try next. Operationally, the MAPE-K loop (Monitor, Analyze, Plan, Execute, sharing a Knowledge Base) carries the argument: monitoring feeds scan and exploit results into the knowledge base, analysis computes utility rankings over targets, attacks, and scans using $U(o)=\\sum_{f\\in F} w_f u_f(o)$, planning allocates the next moves under concurrency thresholds, and execution launches the selected tools and installs post-exploitation probes on compromised hosts. The strategy's memory-update function $S_U$ is realized as a re-ranking of targets, attacks, and scans after every adaptation step. The result is an architecture that can keep choosing targets and attacks until the formal goal predicates are satisfied.","core_discovery":"On the paper's own terms, the central discovery is that a complete penetration test can be modeled and then run autonomously. Given an unknown target modeled as a security-informed architecture—components with interfaces, capabilities, vulnerabilities, interactions, and security properties—the paper defines a penetration test state as $P = \\langle K, C, I\\rangle$ (tester capabilities, known components, known interfaces), attacks as sequences of attack steps that add capabilities, scans as actions that add known components and interfaces, and a penetration test as a labeled transition system $T = \\langle \\mathcal{S}, \\mathcal{A}, \\Sigma, \\rightarrow, P_0, \\mathcal{S}_F\\rangle$ together with a strategy $S = \\langle S_M, S_{M0}, S_U, S_N\\rangle$ that updates memory and picks the next action. The paper argues this formalization is general enough to cover AI-planning, POMDP, and reinforcement-learning formulations, and that a MAPE-K autonomic loop provides a concrete way to realize it. ADAPT is that realization for host-and-service networks: it uses utility-based scoring of targets, attacks, and scans to decide what to do next, operates scanners, exploitation frameworks, and post-exploitation tools, and adapts at runtime when scans complete, attacks fail, or new hosts and interfaces surface. The evaluative claim is that ADAPT is, to the authors' knowledge, the first tool able to automate all parts of the penetration-testing process—planning, runtime decision making, and attack automation—against realistic environments, demonstrated by capturing all 11 flags and obtaining root on all hosts in the VM-network case study.","pith_inferences":["Editorial inference: the evaluation suggests the hard part of autonomy is not the control loop but the repertoire; if so, the next measurable step is automated repertoire generation, which the authors themselves list as future work.","Editorial inference: because the formal definitions are stated over components and interfaces rather than hosts and IP addresses, the same architecture could be instantiated for API surfaces, containerized services, or component-based software, not just networks of machines.","Editorial inference: a direct test of the generality claim would be to run ADAPT against a freshly built target network whose vulnerabilities are covered only by a newly composed repertoire, or to compare its time-to-compromise against human students working the same lab, using flag-capture times as the metric."],"forward_implications":["If the central claim is right, a penetration test can be run end-to-end without a human in the loop, provided a library of attack tactics and scans covering the target's weaknesses exists.","The architectural formalization gives a common language for comparing AI-planning, POMDP, and reinforcement-learning pentest methods, since each can be mapped to the same labeled-transition-system-plus-strategy formulation.","Runtime decision making based on utility rather than a fixed attack graph lets the tool adapt to failed attacks, newly discovered services, and lateral-movement opportunities during a single run.","Automating the operation of standard tools such as nmap, Metasploit, sqlmap, password crackers, and post-exploitation scripts means human effort shifts from executing commands to curating the repertoire and interpreting the final report.","Concurrency thresholds for targets and attacks give the operator a direct trade-off between speed and thoroughness, with the reported VM-network runs showing diminishing returns beyond two concurrent paths."],"supporting_citations":[{"why":"Defines the MAPE-K autonomic manager that the architecture uses as its control loop.","marker":"[19]"},{"why":"Attack-planning approach over PDDL that the paper maps onto its labeled-transition-system formulation.","marker":"[23]"},{"why":"Closest black-box reinforcement-learning pentest baseline; the paper encodes its MDP in the proposed formalism and compares against it.","marker":"[33]"},{"why":"Metasploit framework, the exploitation tooling whose operation ADAPT automates.","marker":"[16]"},{"why":"The Cyber Grand Challenge, cited as prior attack automation at autonomy level 2, the baseline this work aims beyond.","marker":"[21]"},{"why":"Labeled transition system theory that underlies the formal definition of a penetration test.","marker":"[13]"},{"why":"Logical attack graphs, used for the exploitation-graph representation that presents ADAPT's case-study results.","marker":"[24]"}],"fun_headline_variants":["Automated pentesting goes full auto, captures 11 flags","Autonomic loop automates pentest from scan to root","ADAPT formalizes and automates pentesting, wins 11 flags","Pentest automation achieved: ADAPT clears 11 flags solo","Formal model enables fully autonomous pentesting on real nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole demonstration rests on the authors supplying, in advance, libraries of attacks and scans that already include the exact exploits and parameters needed to defeat each target; a target requiring any technique outside those libraries would not be compromised, because ADAPT selects and applies known attacks rather than inventing new ones.","fun_headline_variants_meta":{"raw":{"variants":["Automated pentesting goes full auto, captures 11 flags","Autonomic loop automates pentest from scan to root","ADAPT formalizes and automates pentesting, wins 11 flags","Pentest automation achieved: ADAPT clears 11 flags solo","Formal model enables fully autonomous pentesting on real nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000397,"raw_usage":{"total_tokens":2133,"prompt_tokens":1051,"completion_tokens":1082,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":996}},"tokens_in":667,"tokens_out":1082,"duration_ms":9012,"temperature":1.0,"reasoning_tokens":996,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:45:57.974294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to watch a full ADAPT run against the VM network and record every decision point: if any target choice, exploit selection, parameter value, or pivot is supplied or corrected by a human after the run starts, the 'all parts automated' claim fails. The paper reports no such interventions, and the run logs would settle it; a stronger test would strip from the repertoire every exploit tied to the lab's known CVEs and observe whether any path to the 11 flags remains.","supporting_citations":[{"cited_title":"Attack Planning in the Real World","cited_arxiv_id":"1306.4044","evidence_quote":"Attack-planning approach over PDDL that the paper maps onto its labeled-transition-system formulation."},{"cited_title":"Frontiers of Information Technol- ogy & Electronic Engineering20(9), 1277–1288 (Sep 2019),https: //doi.org/10.1631/FITEE.1800532 C","cited_arxiv_id":null,"evidence_quote":"Closest black-box reinforcement-learning pentest baseline; the paper encodes its MDP in the proposed formalism and compares against it."},{"cited_title":"In: 2014 IEEE 15th International Symposium on Computa- tional Intelligence and Informatics (CINTI)","cited_arxiv_id":null,"evidence_quote":"Metasploit framework, the exploitation tooling whose operation ADAPT automates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Cyber Grand Challenge, cited as prior attack automation at autonomy level 2, the baseline this work aims beyond."}],"review_version":1}