Pith. sign in

REVIEW 3 major objections 6 minor 28 references

A two-agent LLM system can autonomously find and exploit IoT vulnerabilities with 95% success in controlled testbeds.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 01:25 UTC pith:XXPFNUMQ

load-bearing objection Solid empirical multi-agent IoT pentest system that reliably orchestrates known exploits on two lab images; useful domain extension, not a leap in discovery. the 3 major comments →

arxiv 2607.09653 v1 pith:XXPFNUMQ submitted 2026-07-10 cs.CR cs.AI

VEXAIoT: Autonomous IoT Vulnerability EXploitation using AI Agents

Katherine Swinea , Kshitiz Aryal , Lopamudra Praharaj , Maanak Gupta This is my paper
classification cs.CR cs.AI
keywords IoT SecurityLarge Language ModelsAutonomous AgentsPenetration TestingVulnerability ExploitationOWASP IoT
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

IoT devices stay hard to security-test at scale because of constrained hardware, outdated firmware, and insecure defaults, and conventional scanners still need heavy manual setup. This paper presents VEXAIoT, a multi-agent system in which one LLM agent performs reconnaissance and plans ordered attacks while a second agent selects tools, generates commands, and runs exploits. Across ten OWASP-mapped scenarios in IoTGoat and three related services in Metasploitable2, the system completed 260 attack trials at a 95% overall success rate, with most attacks finishing under two minutes and modest token use. The claim is that LLM-driven agents can automate IoT vulnerability assessment and offensive workflows that today still depend on human operators, at least inside isolated lab environments. That matters to anyone who needs repeatable IoT security testing without rewriting attack scripts for every firmware image.

Core claim

Across 260 autonomous attack executions, VEXAIoT—an LLM multi-agent framework that pairs a vulnerability-detection agent with an attack-execution agent—achieves a 95.0% overall success rate (94.5% in IoTGoat, 96.7% in Metasploitable2), with most attacks completing in under two minutes and low token overhead. The paper treats this as evidence that LLM-based agents can automate IoT vulnerability discovery and exploitation in controlled environments.

What carries the argument

VEXAIoT’s coordinated two-agent loop: a vulnerability detection agent that runs nmap and searchsploit, builds a dependency-aware attack sequence, validates outcomes, and orchestrates retries; and an attack execution agent that chooses tools or exploit scripts, generates commands, executes them, and returns status. That separation is what carries autonomous planning, adaptive retry, and optional parallel execution.

Load-bearing premise

The result hangs on a general-purpose LLM plus a fixed toolkit reliably emitting valid, non-refused commands for the exact services and versions in the lab images, and on the paper’s attack-success checks transferring to similar IoT firmware.

What would settle it

Re-run the same ten IoTGoat scenarios and three Metasploitable2 attacks for twenty trials each with the same prompts and tools; if overall success falls well below ~95% or most attacks no longer finish near the reported times under unchanged network conditions, the central performance claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Known OWASP-style IoT firmware flaws can be exploited end-to-end by agents without per-attack hand scripting.
  • Dependency-aware parallel runs can cut wall-clock campaign time roughly in half while token cost stays additive.
  • Failures cluster on model refusals and syntax errors for service-specific exploits, so command validation and retry design become first-order requirements.
  • The same scan–plan–execute pattern can transfer from IoTGoat firmware to adjacent vulnerable services such as those in Metasploitable2.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Moving past intentionally vulnerable lab images will likely need retrieval over live CVE and exploit corpora plus schema-constrained command generation to reduce refusals.
  • A defensive agent that watches the same reconnaissance and exploit signals could turn the setup into a closed red/blue training loop for IoT fleets.
  • Because token use remains additive under parallelization, continuous assessment cost—not only success rate—will gate real-world deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents VEXAIoT, a multi-agent LLM framework for autonomous IoT vulnerability discovery and exploitation. A vulnerability detection agent uses nmap and searchsploit to reconnoiter and plan ordered attacks; an attack execution agent selects tools/scripts, generates commands, and returns outcomes for validation and retry. The system is evaluated on IoTGoat (10 OWASP-mapped scenarios) and three Metasploitable2 services, with 20 trials per attack (260 total). Reported success is 94.5% on IoTGoat, 96.7% on Metasploitable2 (95.0% overall), with most attacks under two minutes and modest token use; parallel dependency-aware execution is also demonstrated. Failures are attributed to model refusals, syntax errors, and rare hallucination.

Significance. If the results hold under the stated scope, the work is a useful empirical demonstration that LLM agents can orchestrate known-exploit workflows against intentionally vulnerable IoT-like images with high reliability and low latency. Strengths include explicit per-attack success criteria (Table I), a failure taxonomy (Table IV), 20-trial consistency, secondary validation on Metasploitable2, and concrete timing/token measurements (Table V) plus parallel-execution data. The contribution is primarily systems/engineering rather than a new discovery method: it shows that coordinated agents plus standard offensive tools can automate lab-style IoT penetration testing. That is of practical interest for controlled security testing and education, provided claims remain scoped to known-vulnerable environments.

major comments (3)
  1. [Abstract; §I; §IV-B/C; Table II] Central claim framing vs. experimental scope (Abstract; §I contributions; §IV-B/C; Table II): The 95% success rate is measured exclusively on IoTGoat OpenWrt firmware and three Metasploitable2 services whose versions map directly into searchsploit/Exploit-DB (steps 1–2 of Fig. 1 and §III-A). The detection agent therefore receives ground-truth CVE/script matches rather than inventing or adapting exploits for unseen firmware. No trial is reported on a device whose nmap banner does not already yield a known exploit script. The abstract and contribution bullets should be revised to state clearly that the result demonstrates reliable orchestration of known exploits in controlled images, not autonomous discovery/exploitation of arbitrary IoT firmware. Without that scoping, the headline claim overreaches the evidence.
  2. [§IV-B; Tables III–VI] Missing baselines and statistical reporting (§IV-B, Tables III–VI): Success rates are reported without confidence intervals, variance across trials, or comparison to non-LLM baselines (manual searchsploit+script, Metasploit modules, or prior agents such as PentestGPT/AutoAttacker on the same images). Because the attack set was chosen to match known IoTGoat vulnerabilities, high success is partly by construction of the testbed. At minimum, report binomial CIs or bootstrap intervals for the 20-trial rates and a simple baseline (e.g., fixed script from searchsploit without LLM planning) so the incremental value of the multi-agent orchestration can be assessed.
  3. [§III-E; Table I; §IV-B] Validation criteria and transferability (§III-E, Table I; §IV-B observed outcomes): Success is defined by immediately observable side-effects (new port, red page background, wiped dmesg, permanent DNS down, etc.). These criteria work for the chosen lab images but do not address partial success, non-binary firmware behavior, or services absent from searchsploit. The paper should either (a) add at least one experiment on a service/version not present in Exploit-DB or with non-binary validation, or (b) explicitly limit the claim and discuss how validation would fail outside these anchors. As written, Table I success does not support generalization beyond the two testbeds.
minor comments (6)
  1. [Throughout] Inconsistent product naming: VEXAIoT / VEXA IoT / VEXAIoT appear throughout (title, abstract, body). Standardize.
  2. [§III-A; §IV-B] Model identifier "ChatGPT 5.1 thinking" is non-standard and not version-pinned; state the exact API model string and date for reproducibility.
  3. [§III-B; Figs. 2–5] Figures 2–5 are described as prompt/output examples but are not fully reproduced as text; include the full prompt templates (or a repository link) so the agent behavior can be audited.
  4. [§II] Related work (§II) cites PentestGPT, AutoAttacker, AutoPentester, and CTF agents but does not quantify how VEXAIoT differs in architecture or metrics on a shared task; a short comparison table would help.
  5. [Tables II–III; §III-E] Typographical issues: "OW ASP" spacing, "V alidation", "dnsmasq2.73", and occasional missing spaces around citations.
  6. [§IV-B Parallel Execution] Parallel-execution results (Figs. 7–8) report wall-clock and tokens but not success rate under concurrency; state whether any race or dependency failures occurred.

Circularity Check

0 steps flagged

No derivation circularity: empirical agent success rates on fixed known-vulnerable testbeds are measured outcomes, not predictions forced by construction or self-citation.

full rationale

VEXAIoT is an engineering/systems paper whose central claims are measured attack success rates (94.5% on IoTGoat over 200 trials, 96.7% on Metasploitable2 over 60 trials, 95% overall across 260 executions), execution times, and token counts. There are no equations, fitted parameters, uniqueness theorems, or first-principles derivations. The vulnerability-detection agent feeds nmap + searchsploit results into an LLM prompt that plans attacks; the attack-execution agent generates and runs commands; success is scored against the explicit, attack-specific criteria of Table I (credential interception, payload effect, service disruption, etc.). Failures (model refusals, syntax errors, one hallucination) are reported and prevent the rates from being definitional. Self-citations ([19], [20]) are background surveys by overlapping authors and are not load-bearing for the empirical numbers. Mapping the ten scenarios to OWASP IoT categories already present in IoTGoat/Metasploitable2 is ordinary testbed selection, not a circular reduction of a claimed prediction to its inputs. The paper is therefore self-contained against its stated external benchmarks (the two VMs) with no circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

The central empirical claim rests on standard offensive-security tooling, the assumption that current LLMs can plan and emit shell commands for known CVEs, and the authors’ own multi-agent orchestration. No free parameters are fitted; the only invented entity is the named framework itself.

axioms (3)
  • domain assumption nmap + searchsploit correctly surface the services and public exploits present on the target firmware
    Invoked in Section III-A steps 1–2 and Algorithm 1; treated as ground truth for planning.
  • domain assumption The LLM will emit syntactically valid, non-refused attack commands given the constrained prompt and tool descriptions
    Core of the attack-execution agent (Section III-D); failures of this axiom account for most of the 5% error rate.
  • ad hoc to paper Attack success can be decided solely by the criteria listed in Table I (credential capture, service down, file change, etc.)
    Defined by the authors for this evaluation; not an external standard.
invented entities (1)
  • VEXAIoT multi-agent framework (detection agent + execution agent with dependency-aware orchestration) no independent evidence
    purpose: Coordinates reconnaissance, planning, tool selection, execution, validation and retry for IoT exploits
    The named system is the paper’s primary artifact; no independent prior existence.

pith-pipeline@v1.1.0-grok45 · 17560 in / 2417 out tokens · 31832 ms · 2026-07-13T01:25:33.917044+00:00 · methodology

0 comments
read the original abstract

Internet of Things (IoT) systems are inherently vulnerable due to constrained hardware, outdated firmware, and insecure default configurations, creating a need for scalable and adaptive security testing approaches. While recent adoptions of Large Language Model (LLM) agents have demonstrated promise in penetration testing and Capture-the-Flag (CTF) environments, their application to IoT specific vulnerabilities remains unexplored. This paper presents an autonomous multi-agent framework, referred to as Vulnerability EXploitation using AI Agents (VEXAIoT), for vulnerability discovery and exploitation in IoT environments using LLM-based reasoning and offensive security tools. The framework combines a vulnerability detection agent and an attack execution agent to perform reconnaissance, plan attack sequences, and execute exploits against vulnerable IoT services. The system is evaluated in IoTGoat and Metasploitable environments across ten attack scenarios mapped to OWASP IoT vulnerabilities. Experimental results show attack success rate of up to 100% with low token overhead and average execution times under two minutes for most attacks. Across 260 attack executions, VEXAIoT achieves a 95.0% overall success rate, including 94.5% success in IoTGoat and 96.7% success in Metasploitable2. These results demonstrate the potential for LLM-driven agents to automate IoT vulnerability assessment and offensive security workflows in controlled environments

Figures

Figures reproduced from arXiv: 2607.09653 by Katherine Swinea, Kshitiz Aryal, Lopamudra Praharaj, Maanak Gupta.

Figure 1
Figure 1. Figure 1: Proposed VEXAIoT Framework for Vulnerability Scanning and Attack Execution agent performs reconnaissance on the target environment, identifies exposed services and protocols, and analyzes po￾tential vulnerabilities using publicly available exploit intel￾ligence in searchsploit’s Exploit Database [24]. The attack execution agent then uses this information to determine appropriate attack methods, select offe… view at source ↗
Figure 2
Figure 2. Figure 2: Example vulnerability detection prompt containing [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example attack execution prompt for the DNS DoS [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example output from the vulnerability detection agent [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example output from the attack execution agent select [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: IoTGoat Testbed device design rather than firmware or network behavior. The “Insecure Ecosystem Interfaces” category contains two distinct exploitable interfaces, resulting in 10 evaluated IoTGoat attack scenarios. Table II summarizes the vulnerability categories, concrete vulnerabilities, and corresponding attack scenarios used in the evaluation. These scenarios cover credential recovery, insecure service… view at source ↗
Figure 7
Figure 7. Figure 7: Time taken to run parallel attacks in IoTGoat [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Tokens used to run parallel attacks in IoTGoat [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 2 linked inside Pith

  1. [1]

    Number of connected iot devices growing 14% to 21.1 billion,

    S. Sinha, “Number of connected iot devices growing 14% to 21.1 billion,”IoT Analytics. https://iot-analytics. com/number-connected-iot- devices, 2025

    2025

  2. [2]

    Available: https://www.fortinet.com/resources/cyberglossary/i ot-security

    [Online]. Available: https://www.fortinet.com/resources/cyberglossary/i ot-security

  3. [3]

    Demystifying iot security: An exhaustive survey on iot vulnerabilities and a first empirical look on internet-scale iot exploitations,

    N. Neshenko, E. Bou-Harb, J. Crichigno, G. Kaddoum, and N. Ghani, “Demystifying iot security: An exhaustive survey on iot vulnerabilities and a first empirical look on internet-scale iot exploitations,”IEEE Communications Surveys & Tutorials, vol. 21, no. 3, pp. 2702–2733, 2019

    2019

  4. [4]

    Hacking iot: A case study on baby monitor exposures and vulnerabilities,

    M. Stanislav and T. Beardsley, “Hacking iot: A case study on baby monitor exposures and vulnerabilities,”Rapid7 Report, vol. 2, 2015

    2015

  5. [5]

    Firmware modification attacks on programmable logic controllers,

    Z. Basnight, J. Butts, J. Lopez, and T. Dube, “Firmware modification attacks on programmable logic controllers,”International Journal of Critical Infrastructure Protection, vol. 6, no. 2, pp. 76–84, 2013

    2013

  6. [6]

    Iot vulnerability assessment for sustainable computing: Threats, current solutions, and open challenges,

    P. Anand, Y . Singh, A. Selwal, M. Alazab, S. Tanwar, and N. Kumar, “Iot vulnerability assessment for sustainable computing: Threats, current solutions, and open challenges,”IEEE access, 2020

    2020

  7. [7]

    A two-layer dimension reduction and two-tier classification model for anomaly-based intrusion detection in iot backbone networks,

    H. Haddadpajouh, R. Javidan, R. Khayami, A. Dehghantanha, and K.- K. R. Choo, “A two-layer dimension reduction and two-tier classification model for anomaly-based intrusion detection in iot backbone networks,” IEEE Trans. on Emerging Topics in Computing, vol. PP, 2016

    2016

  8. [8]

    Framework for mobile internet of things security monitoring based on big data processing and machine learning,

    I. Kotenko, I. Saenko, and A. Branitskiy, “Framework for mobile internet of things security monitoring based on big data processing and machine learning,”IEEE Access, vol. PP, pp. 1–1, 11 2018

    2018

  9. [9]

    Attack and anomaly detection in iot sensors in iot sites using machine learning approaches,

    M. Hasan, M. M. Islam, M. I. I. Zarif, and M. Hashem, “Attack and anomaly detection in iot sensors in iot sites using machine learning approaches,”Internet of Things, vol. 7, p. 100059, 2019

    2019

  10. [10]

    Iot vulnerability assessment for sustainable computing: Threats, current solutions, and open challenges,

    P. Anand, Y . Singh, A. Selwal, M. Alazab, S. Tanwar, and N. Kumar, “Iot vulnerability assessment for sustainable computing: Threats, current solutions, and open challenges,”IEEE Access, vol. 8, 09 2020

    2020

  11. [11]

    Shieldgpt: An llm-based framework for ddos mitigation,

    T. Wang, X. Xie, L. Zhang, C. Wang, L. Zhang, and Y . Cui, “Shieldgpt: An llm-based framework for ddos mitigation,” 08 2024, pp. 108–114

    2024

  12. [12]

    PentestGPT: Evaluating and harnessing large language models for automated penetration testing,

    G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “PentestGPT: Evaluating and harnessing large language models for automated penetration testing,” in33rd USENIX Security Symposium, Philadelphia, PA, 2024

    2024

  13. [13]

    Autoattacker: A large language model guided system to implement automatic cyber-attacks,

    J. Xu, J. W. Stokes, G. McDonald, X. Bai, D. Marshall, S. Wang, A. Swaminathan, and Z. Li, “Autoattacker: A large language model guided system to implement automatic cyber-attacks,”ArXiv, vol. abs/2403.01038, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Hacking ctfs with plain agents,

    R. Turtayev, A. Petrov, D. V olkov, and D. V olk, “Hacking ctfs with plain agents,” 12 2024

    2024

  15. [15]

    Enigma: Interactive tools substantially assist lm agents in finding security vulnerabilities,

    T. Abramovich, M. Udeshi, M. Shaoet al., “Enigma: Interactive tools substantially assist lm agents in finding security vulnerabilities,” 2025. [Online]. Available: https://arxiv.org/abs/2409.16165

    Pith/arXiv arXiv 2025

  16. [16]

    Iotgoat,

    “Iotgoat,” [Accessed: 2019-06-05]. [Online]. Available: https://github.c om/OW ASP/IoTGoat/

    2019

  17. [17]

    Quick start guide — metasploit documentation

    Rapid7, “Quick start guide — metasploit documentation.” [Online]. Available: https://docs.rapid7.com/metasploit/

  18. [18]

    Available: https://wiki.owasp.org/index.php/OWASP Interne t of Things Project#tab=IoT Top 10

    [Online]. Available: https://wiki.owasp.org/index.php/OWASP Interne t of Things Project#tab=IoT Top 10

  19. [19]

    From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy,

    M. Gupta, C. Akiri, K. Aryal, E. Parker, and L. Praharaj, “From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy,”IEEE Access, vol. PP, pp. 1–1, 08 2023

    2023

  20. [20]

    A survey of agentic ai and cybersecurity: Challenges, opportunities and use-case prototypes,

    S. J. Lazer, K. Aryal, M. Gupta, and E. Bertino, “A survey of agentic ai and cybersecurity: Challenges, opportunities and use-case prototypes,” arXiv preprint arXiv:2601.05293, 2026

    arXiv 2026

  21. [21]

    Autopentester: An llm agent-based framework for automated pentesting,

    Y . Ginige, A. Niroshan, S. Jain, and S. Seneviratne, “Autopentester: An llm agent-based framework for automated pentesting,” in2025 IEEE 24th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), 2025, pp. 163–174

    2025

  22. [22]

    Development of iot security exercise contents for cyber security exercise system,

    S. Shin and Y . Seto, “Development of iot security exercise contents for cyber security exercise system,” 06 2020, pp. 1–6

    2020

  23. [23]

    Automated penetration testing framework for smart-home-based iot devices,

    R. Akhilesh, O. Bills, N. Chilamkurti, and M. J. M. Chowdhury, “Automated penetration testing framework for smart-home-based iot devices,”Future Internet, vol. 14, no. 10, 2022. [Online]. Available: https://www.mdpi.com/1999-5903/14/10/276

    2022

  24. [24]

    Available: https://www.exploit-db.com/searchsploit

    [Online]. Available: https://www.exploit-db.com/searchsploit

  25. [25]

    The official nmap project guide to network discovery and security scanning

    G. Lyon, “The official nmap project guide to network discovery and security scanning.” [Online]. Available: https://nmap.org/book/toc.html

  26. [26]

    Bettercap overview

    S. Margaritelli, “Bettercap overview.” [Online]. Available: https: //www.bettercap.org/project/introduction/

  27. [27]

    Towards the development of realistic botnet dataset in the internet of things for network forensic analytics: Bot-iot dataset,

    N. Koroniotis, N. Moustafa, E. Sitnikova, and B. Turnbull, “Towards the development of realistic botnet dataset in the internet of things for network forensic analytics: Bot-iot dataset,”Future Generation Computer Systems, vol. 100, pp. 779–796, 2019

    2019

  28. [28]

    Selection of efficient machine learning algorithm on bot-iot dataset for intrusion detection in internet of things networks,

    I. Kerrakchou, A. Abou El Hassan, S. Chadli, M. Emharraf, and M. Saber, “Selection of efficient machine learning algorithm on bot-iot dataset for intrusion detection in internet of things networks,”Indonesian Journal of Electrical Engineering and Computer Science, vol. 31, p. 1784, 09 2023

    2023