Vulnerability Mitigation System (VMS): LLM Agent and Evaluation Framework for Autonomous Penetration Testing

Farzana Abdulzada

REVIEW 5 major objections 7 minor 1 cited by

Vulnerability Mitigation System (VMS): LLM Agent and Evaluation Framework for Autonomous Penetration Testing

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM agent called VMS, built from a Planner and a Summarizer, can run penetration-testing tasks autonomously, with GPT-4o solving 41 of 120 PicoCTF and 32 of 80 OverTheWire challenges.

desk verdict A plausible LLM agent and useful CTF benchmark, but the evaluation is too weak to support the 'autonomous penetration testing' and 'vast majority' claims, and the lack of contamination controls leaves the solve rates unexplained. read the letter →

arxiv 2507.21113 v1 pith:3EV5OCDI submitted 2025-07-14 cs.CR

Farzana Abdulzada This is my paper · ORCID

classification cs.CR

keywords AutonomousPenetrationTestingAgentsCaptureTheFlagChallengesLargeLanguageModelsCybersecurityAutomationBenchmarksPlanner-SummarizerArchitecturePicoCTFOverWire

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a large language model, wrapped in a two-module agent called VMS, can carry out penetration-testing tasks without a human in the loop. To support this, the paper built two new Capture the Flag benchmarks, 120 PicoCTF challenges and 80 OverTheWire challenges, and ran eight LLMs through them inside an isolated container. The best model, GPT-4o, solved 41 of the 120 PicoCTF challenges and 32 of the 80 OverTheWire challenges at temperature 1 and top-p 0.9. If the claim holds, LLM agents could make security testing faster and more scalable while containerization limits the damage of a bad command.

What carries the argument

The load-bearing mechanism is the Planner–Summarizer loop. The Planner uses a system prompt that treats the LLM as an expert penetration tester and outputs shell commands; the Summarizer processes each command's output, truncating it to an observation window and updating the action history; and the loop repeats inside a Kali Linux container protected by firewall rules until a flag is captured or a step limit is reached. The evaluation also tunes observation window size, temperature, and top-p, with the best results at window 250 for PicoCTF, 500 for OverTheWire, temperature 1, and top-p 0.9.

What would settle it

Run VMS on a realistic, deliberately vulnerable network where no flag is predefined and no human hints are allowed; if the agent cannot discover and exploit a vulnerability, the CTF results would not establish autonomous penetration testing.

Watch

Extended reading notes

Core claim

The central claim is that VMS is a completely autonomous penetration testing agent: a Planner module turns system prompts and observed context into executable commands, and a Summarizer module condenses command output and keeps an action history, so the loop can run without human intervention. The paper reports that GPT-4o solved 41 of 120 PicoCTF tasks and 32 of 80 OverTheWire tasks, outperforming the other seven models tested, and that the released benchmarks give the field a standardized way to measure such agents.

Load-bearing premise

The load-bearing premise is that solving curated CTF puzzles with known flags is a fair stand-in for autonomous penetration testing of real, live systems.

Editorial extensions

If this is right

A single LLM, prompted alternately as Planner and Summarizer, can replace the human in a closed command-execution loop for curated CTF tasks.
GPT-4o is the strongest of the eight tested models on these benchmarks, with optimal settings at temperature 1 and top-p 0.9.
The released 200-challenge benchmark suite gives later researchers a shared yardstick for comparing LLM-based penetration testing agents.
A containerized Kali Linux environment with firewall rules is a workable safety wrapper for letting an LLM issue arbitrary shell commands.
Raising temperature to 2.0 degrades reliability enough to render the system unusable, so sampling parameters act as a safety-relevant control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the benchmarks score flag capture rather than discovery and exploitation on live systems, the measured capability may not transfer to realistic multi-host networks; a natural next experiment is to run VMS against a vulnerable virtual network with no pre-known flag.
The same 200-challenge suite could also be used to compare agent architectures beyond LLM choice, since its category and difficulty metadata make failure-mode analysis straightforward.
The strong dependence on temperature suggests that production deployments would need to bound sampling parameters as a safety layer, not rely on container isolation alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. It builds a reasonable LLM agent and a useful CTF benchmark, but the reported solve rates do not support the central claims about autonomous penetration testing. The load-bearing problem is contamination: PicoCTF and OverTheWire are public and heavily documented, and the paper provides no evidence that GPT-4o isn't recalling memorized solutions.

What's actually new: the VMS architecture is a clean Planner/Summarizer design, and the 200-challenge benchmark with metadata and solver scripts is a genuinely useful artifact if the author actually releases it as promised. The model comparison, narrow though it is, is a real measurement. The paper also cites the relevant prior work (PentestGPT, InterCode, Abramovich et al.), so it is not ignoring the literature.

The soft spots are substantial. First, the overclaiming: 41/120 and 32/80 are 34% and 40%, not a "vast majority." Second, there are no baselines. The paper never compares against a plain LLM prompted to solve CTF challenges, nor against PentestGPT or an ablation without the Summarizer. So the attribution of these numbers to the VMS architecture is unestablished. Third, the contamination issue is load-bearing and unaddressed. OverTheWire's Bandit, Natas, Leviathan, and Krypton have fixed command sequences reproduced in countless writeups; GPT-4o almost certainly saw them in training. No holdout analysis, no perturbation test, nothing. Fourth, the parameter optimization (temperature 1, top-p 0.9, window sizes 250/500) is performed on the same data used to report final performance, and there are no error bars. Fifth, only three of the eight tested models appear in the results table. Sixth, the leap from CTF puzzles to "autonomous penetration testing" of real systems is unsupported by any evidence.

None of this makes the paper worthless. The benchmark alone could be useful to the community, and the narrow empirical fact that GPT-4o solves a third of these curated challenges is worth knowing. But as it stands, the paper's interpretation goes well beyond what the experiments show.

If it comes to peer review: yes, send it out. The topic is important, the artifacts are new, and the evaluation can be fixed in revision. But the author needs to add baselines, ablations, contamination controls, error bars, and a much more careful framing of what CTF success does and does not mean. A serious referee would be doing the field a service by pushing on those points.

Referee Report

5 major / 7 minor

Summary. The paper presents VMS, an LLM-based agent with two modules (Planner and Summarizer) that is claimed to perform autonomous penetration testing. The authors construct two CTF benchmarks from PicoCTF (120 challenges) and OverTheWire (80 challenges), test eight LLMs, and tune temperature, top-p, and observation window size. They report that GPT-4o solves 41/120 PicoCTF and 32/80 OverTheWire challenges, which they describe as a 'vast majority' and as evidence for completely autonomous penetration testing. The paper also claims to release the VMS and benchmarks to the public, but no repository link is given.

Significance. The two CTF benchmarks, if properly released, could be a useful community resource for evaluating LLM-based agents in cybersecurity settings. The containerized evaluation setup is a reasonable safety measure, and the demonstration that GPT-4o can solve a non-trivial fraction of curated CTF challenges in an automated loop is of interest. However, the central claims about 'completely autonomous penetration testing' and solving a 'vast majority' of challenges are not supported by the reported numbers (34% and 40%). The evaluation also lacks full reporting of model results, contamination controls, and ablations, so the empirical contribution is currently not established at the level claimed.

major comments (5)

[Conclusions] The Conclusions state that VMS 'was able to solve a vast majority of the Capture The Flag (CTF) challenges' and is a 'completely autonomous penetration testing agent.' With 41/120 (34%) and 32/80 (40%) solved, these are not vast majorities; the overstatement is load-bearing because it frames the entire contribution. The claims should be rewritten to match the reported evidence, and the term 'autonomous penetration testing' should be qualified as 'solving curated CTF challenges in a containerized environment.'
[Experimental Evaluation / Table 1.1] The paper states that eight LLMs were tested, but Table 1.1 reports results for only three (GPT-4o, Llama-3.1-70B, GPT-4o-mini). No results are given for Llama-3.1-8B, Qwen2-72B, Mixtral-8x7B, Phi-3-mini-4k, or Phi-3.5-MoE. Without the full comparison, the claim that 'GPT-4o performed best' among the evaluated models is not verifiable. The authors should either report the full results or clearly state that the other models were omitted for brevity and make the data available.
[Experimental Evaluation / Parameter optimization] The optimal parameters (observation window 250/500, temperature 1.0, top-p 0.9) are identified from experiments on the same benchmarks that produce the final results in Table 1.1. This is a selection-on-the-test-set procedure; no held-out validation or repeated-run statistics are reported. The solve counts may be optimistically biased, and the paper should specify whether the reported numbers use the tuned parameters and provide error bars or a sensitivity analysis.
[Benchmark Construction / Experimental Evaluation] The benchmarks are built from public PicoCTF and OverTheWire challenges whose flags are fixed and extensively documented on the web. For a model like GPT-4o with unknown training-data coverage, the reported solve counts could reflect memorized solution paths rather than the Planner+Summarizer reasoning loop. The paper provides no contamination analysis (e.g., holding out challenges released after the model's cutoff, modifying challenge details, or checking exact overlaps) and no ablation (e.g., Planner-only or plain-prompt baseline). Without these controls, the attribution of the results to the VMS architecture is not established.
[Introduction / Conclusions] The paper's title and abstract claim 'autonomous penetration testing,' but the evaluation measures flag capture on CTF puzzles with known flags. The gap between CTF performance and real-world penetration testing is never discussed, and no realistic or network-based scenario is included. A concrete test of transfer, or an explicit statement that CTF success is only a proxy with unknown external validity, is needed to support the stated goal.

minor comments (7)

[Abstract] 'todays' should be 'today's'.
[Methods] The phrase 'trained by Large Language Models' should be 'powered by' or 'based on', since VMS is an agent that uses LLMs, not a trained model itself.
[Results and Discussion] The reference to 'Figure 1.1' appears in the text, but no figure is present in the manuscript; the figure and caption should be added or the reference removed.
[Benchmark Construction] The statement 'Solver scripts were written to pull flags dynamically' is vague; the authors should clarify whether these scripts are used as ground truth for verification or as part of the agent's workflow.
[Conclusions] The claim that 'This research therefore becomes the first to propose the VMS and its benchmarks' is not supported by the references, which include PentestGPT and AutoAttacker; a related-work comparison is needed.
[References] Several reference entries are incomplete or contain typos (e.g., reference 4 lists 'Lie, Y.' likely for 'Liu, Y.'; references 12 and 15 lack full publication details).
[Experimental Evaluation] The paper states that both VMS and the benchmarks are publicly available, but no repository link or access instructions are given; include the URL for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central results are empirical solve counts on externally maintained CTF benchmarks, not predictions derived from fitted inputs.

full rationale

The paper does not present a derivation chain in which a predicted quantity is constructed from fitted inputs. The central empirical claims are solve counts (GPT-4o: 41/120 PicoCTF and 32/80 OverTheWire) measured against externally maintained CTF challenges, with benchmark content taken from PicoCTF and OverTheWire rather than generated from the VMS definition. The Planner/Summarizer architecture is described, not derived, and no self-citation is load-bearing; the reference list contains no work by the author. The only in-sample selection is the choice of temperature 1, top-p 0.9, and window sizes 250/500 based on the same benchmark runs, but the paper presents these as determined settings, not as an independent prediction validated after fitting; that is a test-set overfitting and validity concern, not definitional circularity. Likewise, the CTF-to-real-pentest generalization gap and possible training-data contamination are validity and correctness risks, not circularity. No circular step is identifiable.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The core evaluation rests on benchmark representativeness, tuned sampling parameters, and container safety; none of these are independently validated in the paper.

free parameters (4)

observation window size = 250 characters (PicoCTF), 500 characters (OverTheWire)
Selected by testing on the benchmarks; the paper reports these as optimal without held-out validation.
temperature = 1.0
Reported as optimal; tuned on the same benchmark runs, with higher temperatures increasing error rates.
top-p = 0.9
Reported as optimal; tuned alongside temperature on the evaluation benchmarks.
iteration step limit = 20 steps per challenge
Chosen as an evaluation limit; no sensitivity analysis is reported.

assumptions (5)

domain assumption Capturing a CTF flag is a valid measure of penetration testing ability.
Used throughout to equate CTF solve counts with autonomous pentesting capability (Introduction, Conclusions).
domain assumption The selected PicoCTF and OverTheWire challenges are representative of real-world vulnerabilities.
Benchmark Construction selects 120 and 80 challenges without a stated representativeness argument.
domain assumption Solver scripts retrieve flags without leaking solutions to the LLM.
Benchmark Construction says solver scripts pull flags dynamically, but no details ensure the LLM cannot access them.
domain assumption Containerization and firewall rules fully isolate the agent.
Methods state the agent runs in a Kali Linux container with firewall rules; no verification of isolation is provided.
domain assumption A 20-step limit is enough to assess a model's problem-solving ability.
Experimental Evaluation fixes 20 iterative steps per challenge without an ablation.

invented entities (2)

VMS agent
purpose: Autonomously plans and executes penetration testing commands via a Planner and Summarizer.
The paper provides no external validation, third-party reproduction, or public repository link in the text.
Summarizer module
purpose: Processes command outputs and maintains action history to guide the Planner.
This is a component of VMS; its contribution is not independently measured with an ablation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Vulnerability Mitigation System (VMS): LLM Agent and Evaluation Framework for Autonomous Penetration Testing." pith.science (2026). https://pith.science/paper/3EV5OCDI

@misc{pith2026250721113,
  author       = {Pith},
  title        = {Pith review of: Vulnerability Mitigation System (VMS): LLM Agent and Evaluation Framework for Autonomous Penetration Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3EV5OCDI}},
  note         = {Machine review of arXiv:2507.21113}
}

read the original abstract

As the frequency of cyber threats increases, conventional penetration testing is failing to capture the entirety of todays complex environments. To solve this problem, we propose the Vulnerability Mitigation System (VMS), a novel agent based on a Large Language Model (LLM) capable of performing penetration testing without human intervention. The VMS has a two-part architecture for planning and a Summarizer, which enable it to generate commands and process feedback. To standardize testing, we designed two new Capture the Flag (CTF) benchmarks based on the PicoCTF and OverTheWire platforms with 200 challenges. These benchmarks allow us to evaluate how effectively the system functions. We performed a number of experiments using various LLMs while tuning the temperature and top-p parameters and found that GPT-4o performed best, sometimes even better than expected. The results indicate that LLMs can be effectively applied to many cybersecurity tasks; however, there are risks. To ensure safe operation, we used a containerized environment. Both the VMS and the benchmarks are publicly available, advancing the creation of secure, autonomous cybersecurity tools.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI
cs.SE 2026-07 conditional novelty 6.0 of 10

For a fixed DeepSeek model, the MCP client alone produced a 2.1× solve-rate gap on HexStrike-AI CTF trials, and bundled tool/behavior fixes lifted overall success from 55.4% to 72.0%.

Reference graph

Works this paper leans on

11 extracted references · 5 canonical work pages · cited by 1 Pith paper

[2]

Cybercompetitions: A survey of competitions, tools, and systems to support cybersecurity education,

Balon, T., Baggili, I., "Cybercompetitions: A survey of competitions, tools, and systems to support cybersecurity education," Education and Information Technologies, vol. 28, no. 9, pp. 11759-11791, 2023

work page 2023
[3]

A Data-Driven Approach to Lightweight DVFS-Aware Counter-Based Power Modeling for Heterogeneous Platforms

CrowdStrike, "2024 Global Threat Report," Accessed: November 1, 2024. [Online]. Available: https://www.crowdstrike.com/global-threat-report/ 4. Deng, G., Lie, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Piazger, M., Rass, S., "PentestGPT: An LLM-empowered automatic penetration testing tool," arXiv preprint arXiv:2305.06782, 2023

work page Pith review arXiv 2024
[5]

Getting pwn’d by AI: Penetration testing with the language models,

Happe, A., Cito, J., "Getting pwn’d by AI: Penetration testing with the language models," in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE ’23, ACM, November 2023. [Online]. Available: http://dx.doi.org/10.1145/3611643.3613083 6. Leune, K., Petrilli Jr, S....

work page arXiv 2023
[7]

Large language models: A survey,

Minace, S., Mikolov, T., Nikrad, N., Chenaghlu, M., Socher, R., Amatriain, X., Gao, J., "Large language models: A survey," arXiv preprint arXiv:2402.06196, 2024

arXiv 2024
[8]

Large language models in cybersecurity: State-of-the-art,

Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., Meinel, C., "Large language models in cybersecurity: State-of-the-art," arXiv preprint arXiv:2402.00891, 2024

arXiv 2024
[9]

Can large language models find and fix vulnerable software?

Noever, D., "Can large language models find and fix vulnerable software?" arXiv preprint arXiv:2308.10345, 2023

arXiv 2023
[10]

The rise and potential of large language model based agents: A survey,

Tenable, Inc., "Nessus," Accessed: November 1, 2024. [Online]. Available: https://www.tenable.com/products/nessus 11. Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al., "The rise and potential of large language model based agents: A survey," arXiv preprint arXiv:2309.07864, 2023

arXiv 2024
[12]

AutoAttacker: A large language model guided system to implement automatic cyber-attacks,

Xu, J., Stokes, J. W., McDonald, G., Bai, X., Marshall, D., Wang, S., Swaminathan, A., Li, Z., "AutoAttacker: A large language model guided system to implement automatic cyber-attacks,"

work page

Show all 11 references

[14]

A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly,

Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., Zhang, Y., "A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly," High-Confidence Computing, p. 100211, 2024

2024
[15]

A survey of large language models,

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al., "A survey of large language models," arXiv preprint arXiv:2303.18223, 2023

2023 arXiv
[2024]

Intercode: Standardizing and benchmarking interactive coding with execution feedback,

[Online]. Available: https://arxiv.org/abs/2403.01038 13. Yang, J., Prabhakar, A., Narasimhan, K., Yao, S., "Intercode: Standardizing and benchmarking interactive coding with execution feedback," Advances in Neural Information Processing Systems, vol. 36, 2024

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2]

Cybercompetitions: A survey of competitions, tools, and systems to support cybersecurity education,

Balon, T., Baggili, I., "Cybercompetitions: A survey of competitions, tools, and systems to support cybersecurity education," Education and Information Technologies, vol. 28, no. 9, pp. 11759-11791, 2023

work page 2023

[2] [3]

A Data-Driven Approach to Lightweight DVFS-Aware Counter-Based Power Modeling for Heterogeneous Platforms

CrowdStrike, "2024 Global Threat Report," Accessed: November 1, 2024. [Online]. Available: https://www.crowdstrike.com/global-threat-report/ 4. Deng, G., Lie, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Piazger, M., Rass, S., "PentestGPT: An LLM-empowered automatic penetration testing tool," arXiv preprint arXiv:2305.06782, 2023

work page Pith review arXiv 2024

[3] [5]

Getting pwn’d by AI: Penetration testing with the language models,

Happe, A., Cito, J., "Getting pwn’d by AI: Penetration testing with the language models," in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE ’23, ACM, November 2023. [Online]. Available: http://dx.doi.org/10.1145/3611643.3613083 6. Leune, K., Petrilli Jr, S....

work page arXiv 2023

[4] [7]

Large language models: A survey,

Minace, S., Mikolov, T., Nikrad, N., Chenaghlu, M., Socher, R., Amatriain, X., Gao, J., "Large language models: A survey," arXiv preprint arXiv:2402.06196, 2024

arXiv 2024

[5] [8]

Large language models in cybersecurity: State-of-the-art,

Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., Meinel, C., "Large language models in cybersecurity: State-of-the-art," arXiv preprint arXiv:2402.00891, 2024

arXiv 2024

[6] [9]

Can large language models find and fix vulnerable software?

Noever, D., "Can large language models find and fix vulnerable software?" arXiv preprint arXiv:2308.10345, 2023

arXiv 2023

[7] [10]

The rise and potential of large language model based agents: A survey,

Tenable, Inc., "Nessus," Accessed: November 1, 2024. [Online]. Available: https://www.tenable.com/products/nessus 11. Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al., "The rise and potential of large language model based agents: A survey," arXiv preprint arXiv:2309.07864, 2023

arXiv 2024

[8] [12]

AutoAttacker: A large language model guided system to implement automatic cyber-attacks,

Xu, J., Stokes, J. W., McDonald, G., Bai, X., Marshall, D., Wang, S., Swaminathan, A., Li, Z., "AutoAttacker: A large language model guided system to implement automatic cyber-attacks,"

work page