Agent-based (BDI) modeling for automation of penetration testing

Alexei Lisitsa; Ge Chu

REVIEW 3 major objections 4 minor 19 references

Agent-based (BDI) modeling for automation of penetration testing

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a belief-desire-intention (BDI) agent, implemented in Jason, automates penetration testing by encoding attacks as plans and using a reasoning cycle to interact with dynamic, uncertain targets.

desk verdict Plausible BDI-to-pentesting mapping, but the simulations never exercise the dynamic interactivity that is the paper's central claim. read the letter →

arxiv 1908.06970 v1 pith:TWASS4QL submitted 2019-08-18 cs.CR cs.MA

Ge Chu , Alexei Lisitsa This is my paper · ORCID

classification cs.CRcs.MA

keywords penetrationtestingBDIagentbelief-desire-intentionautomatedpentestingJasonattackplansmulti-stepattacks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes automating penetration testing with a belief-desire-intention (BDI) agent instead of the static attack-graph planning used in prior tools. Penetration testing actions are encoded as BDI plans, and the agent's reasoning cycle—perceive, update beliefs, select an intention, execute, retry on failure—drives interaction with targets. A prototype implemented in Jason combines external tools such as Nmap, Metasploit, SQLmap, and Hydra, and two simulations demonstrate the agent escalating to root privilege through different attack paths. The stated goal is to handle dynamic, uncertain, and complex target environments that non-interactive automation cannot.

What carries the argument

The central mechanism is the BDI reasoning cycle implemented in Jason (AgentSpeak): the agent perceives target information via tools like Nmap and OpenVAS, updates its belief set, derives candidate plans whose context matches current beliefs, selects one plan as an intention using priorities from a human knowledge database, executes it, and on failure loops back to choose another plan. The plan structure with trigger event, context and body is what carries the automation, making the model extensible by adding new plans.

What would settle it

Run the same BDI agent in a real network or a benchmark with empirically measured exploit success rates, and compare whether the agent selects and times its attacks as well as, or better than, a human tester or a graph-based planner. If the hand-set thresholds are replaced by realistic probabilities and the agent's success drops sharply, the claimed benefit over static planners is not established.

Watch

Extended reading notes

Core claim

The central claim is that the BDI agent model can automate penetration testing by treating attacks as plans selected through a reasoning cycle, giving the automation an interactive, reactive capability that graph-based and automata-based approaches lack. The paper defines the agent as a tuple of beliefs, desires, intentions, plans, actions and perceptions, and specifies how beliefs update from scans, how candidate plans become intentions via a human-knowledge priority database, and how failed attacks trigger retry with an alternative plan. Two simulated runs show the agent gathering target information, failing a password attack and then succeeding via remote buffer overflow in one case, and succeeding at password plus local and remote buffer overflow in the other, reaching root privilege.

Load-bearing premise

The evaluation relies on success probabilities chosen by the first author's experience (0.8 for SSH password attack, 0.5 for remote, and 0.3 for local buffer overflow), so the demonstrated infiltrations only validate the model if those thresholds match real-world attack success rates.

Editorial extensions

If this is right

Automated pentesting can progressively escalate privileges (none to user to root) through multi-step attacks within a single reasoning session.
The model can recover from failed attack attempts by selecting an alternative plan, something static attack-graph planners cannot do.
New attack types can be added as new BDI plans elicited from human experts, without reworking the core agent.
External tools (Nmap, Metasploit, SQLmap, Hydra, Ettercap, SET) can be invoked as part of the action space, so the framework integrates with existing toolchains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the BDI approach generalizes beyond simulation, it suggests that the bottleneck in automated pentesting shifts from path planning to encoding expert decision knowledge as plan priorities and success probabilities.
The same plan-trigger-context-body structure could be applied to other security operations that require interactive decision-making, such as incident response or red-team simulation, where conditions change mid-operation.
A testable extension would be to replace the hand-set success thresholds with data from real exploit outcomes or public vulnerability statistics, then compare agent success rates against those baselines.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is reasonable: model penetration testing as a BDI agent that selects attack plans based on gathered evidence, and implement it in Jason so external tools like Nmap and Metasploit can be invoked. That is a legitimate extension of established agent technology to a new domain, and the paper is honest about being a prototype. The survey of attack-graph approaches and the argument that they are mostly non-interactive is fair and points to a real gap.

The problem is the validation. The two simulations do not test the thing the abstract promises: interactive dealing with dynamic, uncertain and complex environments. The target is a static table of OS, ports, services and vulnerabilities. Nothing changes during a run—no new ports open, no services change, no defenses react. Uncertainty is a single random draw compared against author-set thresholds (0.8 for SSH password attack, 0.5 remote and 0.3 local buffer overflow, said to come from the first author's experience). In Simulation 2 the password attack succeeds because the random number is 0.9, above the 0.8 threshold. A hand-coded if-then-else script with random outcomes would produce the same traces. The BDI reasoning cycle's claimed benefit—revising intentions mid-plan based on environment feedback—is never actually exercised. So the conclusion that the simulations 'validate the modelling' overreaches.

There are softer issues too. Equations (1)-(3) are placeholders with undefined functions. There is no baseline comparison with AEPT or attack-graph techniques, so the claimed 'higher probability of success' is unquantified. No code is released, which makes the prototype hard to assess. The related work section does not survey other agent-based pentesting proposals, so the novelty claim is not fully benchmarked—though the specific BDI plan-based model does appear to be new relative to the cited literature.

None of this makes the model itself unsound. The BDI architecture is a plausible fit for choosing and retrying attack plans. The flaw is in the evidence: the central claim is unsupported by the present simulations, not contradicted by them.

Who is this for? Someone working on automated security testing and curious about whether cognitive agent architectures add anything over planning-based approaches. It is a short proof-of-concept, not a system paper. It deserves a serious referee for a workshop or applied conference, but it would need substantial revision to be accepted: a truly dynamic target model, repeated trials, a baseline comparison, and ideally released code. If I were editor, I would send it to review rather than desk reject, because the direction is worth engaging with, but I would expect the validation section to be redone.

Referee Report

3 major / 4 minor

Summary. The paper proposes a belief-desire-intention (BDI) agent model for automating penetration testing. Penetration testing actions are represented as BDI plans, and the Jason implementation of AgentSpeak is used to implement a prototype. The authors argue that, unlike attack-graph-based and automata-based approaches, the BDI reasoning cycle enables interactive adaptation to dynamic, uncertain, and complex target environments. Two simulations are presented in which a BDI agent gathers information about a simulated target and performs password and buffer-overflow attacks; the paper claims these simulations validate the model. The paper concludes with future work on extending the action set and experimenting in real environments.

Significance. If the central claim were substantiated, an interactive BDI-based approach to pentesting automation could be a novel and useful contribution, since existing automated approaches are largely static. The paper correctly identifies a real limitation of prior work: non-interactive processing cannot recover from failed attack attempts or adapt to new information. However, the manuscript does not provide evidence for this claimed advantage. The simulations in Section V exercise only a static decision tree with random outcomes; they do not demonstrate that the BDI reasoning cycle adds value over a simple script. The formal model in Section III is a tuple and three equations with undefined functions, so it is not a complete specification against which an implementation could be verified. The paper ships a prototype and shows executable traces, which is a useful starting point, but the evidence base is far too thin to support the conclusion that the BDI model 'validates' automated interactive pentesting or that it achieves 'a higher probability of success'.

major comments (3)

[V.C] The validation evidence is circular and unquantified. In Section V.A the success thresholds are set to 0.8 for SSH password attack, 0.5 for remote buffer overflow, and 0.3 for local buffer overflow, 'based on personal penetration testing experience of the first author.' These thresholds fully determine the simulation outcomes: in Simulation 1 the random draw for the password attack is below 0.8, so it fails; in Simulation 2 the draw is 0.9, so it succeeds. The paper presents two runs without error bars, repeated trials, or statistical analysis, and no baseline comparison with AEPT or attack-graph approaches. Consequently, the Section VI claim that the model improves 'efficiency and probability of success' is unsupported. The simulations show only that the agent executes the pre-scripted plans, not that the modeled probabilities correspond to real attack success rates or that the approach is superior to existing methods.
[Section III, Equations (1)-(3)] The formalization of the BDI model is a placeholder, not a specification. The equations B = f1(B × S), D = f2(B × I), and I = f3(B × D × I) use functions f1, f2, and f3 that are never defined, and the domains and semantics of the update operations are not given. The reasoning cycle in Section III is described in prose and in Figure 3, but there is no formal link between the equations and the Jason implementation. This makes it impossible to verify that the implementation is faithful to the model, and it weakens the paper's claim of a model-based approach. The authors should either provide well-defined update functions (e.g., in terms of plan selection and perception) or explicitly state that the equations are schematic and give a precise operational semantics.
[Section V, experimental methodology] The experiments are not reproducible or comparable as reported. There is no description of the random seed, the number of runs, or the distribution from which the random numbers are drawn. The target is an idealized agent whose beliefs contain the exact vulnerability information (CVE-remote, CVE-local) and password ('SSH:456'), so the information-gathering stage is effectively a direct readout of the target's internal state. The paper does not compare against the methods discussed in Section II (AEPT, attack graphs), so the assertion that the BDI approach overcomes their limitations is not tested. At minimum, the authors should run many simulations, report success-rate distributions, and compare against a scripted non-BDI baseline on the same simulated target.

minor comments (4)

[Section I and references] The citation 'Bradman [4]' should be 'Bratman' (Michael Bratman, the originator of the BDI framework). The reference list should be checked carefully for spelling and completeness.
[Section V.C, captions] The caption for the second simulation reads 'Figure 6 BDI agent result in simulation 2', which duplicates the figure number of the first simulation and should presumably be Figure 8. Also, the text refers to 'Future 7' but the figure is likely 'Figure 7'. These cross-reference errors need correction.
[General] The paper frequently states that the BDI model 'enables interaction between dynamic and uncertain targets' but the description of the reasoning cycle (items 1-7 in Section III) is a fixed sequence with no branch for a mid-plan environmental change. Clarify what interactions are actually supported and where the cycle handles a new perception arriving during plan execution.
[Section V.A] The sentence describing the target agent's password threshold is ambiguous: 'we set 0.8 as the threshold to determine if the SSH password attack is successful' and later in Simulation 2 'the rate of the SSH password attack was set to 0.9 which is greater than the 0.8 threshold.' Clarify whether 0.8 is an attack-success probability used in a single draw or a parameter that the user can set per simulation.

Circularity Check

1 steps flagged · score 5.0 of 10

The stated validation is circular: simulation outcomes are defined by author-chosen random thresholds, so the 'validated' successes and failures are generated by the validation rule itself, not by independent evidence.

self definitional [Section V.A (Target agent) and Section VI (Conclusion)]
"To make the scenario uncertain, we use randomization and set 0.8 as the threshold to determine if the SSH password attack is successful by generating a random number and comparing it with the threshold. In terms of the remote or local buffer overflow attack successful rate, we set thresholds as 0.5 and 0.3, respectively (this is based on personal penetration testing experience of the first author) ... Two simulations show the BDI agent behavior and reasoning process to validate the modelling."

The success of each attack is defined, in the simulation, as a random draw compared with the author-set threshold (0.8, 0.5, or 0.3). The reported outcomes (password failure in Simulation 1, password success in Simulation 2, and successful buffer overflows) are direct outputs of that random-number comparison, not independent observations of the BDI model's effectiveness. Calling these traces a validation reduces to checking that the program executes its own stochastic rule; a hand-coded script with the same thresholds would produce identical evidence. Thus the simulations validate the implementation's bookkeeping, but not the model's ability to succeed in real dynamic environments.

full rationale

The paper's core proposal—encoding pentesting actions as BDI plans and implementing them in Jason—is not circular: it is a design artifact with code excerpts and a reasoning-cycle description, and it does not rest on self-citations or imported uniqueness theorems. The only significant circularity is in the validation step. Section V.A sets success by random-number-threshold comparison, and Section VI presents the resulting traces as 'validat[ing] the modelling'; this is self-definitional because the outcome is generated by the same rule that defines success. The arbitrary thresholds are not fitted to external data, so this is not a fitted-parameter-as-prediction case, but they are unvalidated and no baseline comparison is provided. The conclusion's own limitation statement—future work will 'experiment with real (non-simulated) environments'—further confirms that the simulated traces cannot independently support the dynamic-environment claim. Score 5 reflects partial circularity in the validation while the central architectural claim retains independent content.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on free parameters (the three thresholds) that determine simulation outcomes, and on domain assumptions about BDI suitability and the fidelity of the simulated target. No new physical or formal entities are introduced.

free parameters (3)

SSH password attack success threshold = 0.8
Set in Section V.A based on personal experience; used as a gate for password attack success.
Remote buffer overflow attack success threshold = 0.5
Set in Section V.A; determines whether the remote exploit succeeds in the simulations.
Local buffer overflow attack success threshold = 0.3
Set in Section V.A; used as the success probability for local exploitation.

assumptions (3)

domain assumption BDI model is a suitable formalism for modeling interactive penetration testing
The paper assumes the BDI reasoning cycle can represent the pentesting process without comparing to alternative interactive formalisms.
domain assumption External security tools execute successfully with probabilities equal to the chosen thresholds
The simulation treats tools like Metasploit and Hydra as black boxes whose success is governed by the author-set random thresholds.
domain assumption The simulated target agent accurately represents a real target environment
The validation replaces a real network with an agent whose belief set is pre-populated, simplifying away real-world noise and countermeasures.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Agent-based (BDI) modeling for automation of penetration testing." pith.science (2026). https://pith.science/paper/TWASS4QL

@misc{pith2026190806970,
  author       = {Pith},
  title        = {Pith review of: Agent-based (BDI) modeling for automation of penetration testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TWASS4QL}},
  note         = {Machine review of arXiv:1908.06970}
}

read the original abstract

Penetration testing (or pentesting) is one of the widely used and important methodologies to assess the security of computer systems and networks. Traditional pentesting relies on the domain expert knowledge and requires considerable human effort all of which incurs a high cost. The automation can significantly improve the efficiency, availability and lower the cost of penetration testing. Existing approaches to the automation include those which map vulnerability scanner results to the corresponding exploit tools, and those addressing the pentesting as a planning problem expressed in terms of attack graphs. Due to mainly non-interactive processing, such solutions can deal effectively only with static and simple targets. In this paper, we propose an automated penetration testing approach based on the belief-desire-intention (BDI) agent model, which is central in the research on agent-based processing in that it deals interactively with dynamic, uncertain and complex environments. Penetration testing actions are defined as a series of BDI plans and the BDI reasoning cycle is used to represent the penetration testing process. The model is extensible and new plans can be added, once they have been elicited from the human experts. We report on the results of testing of proof of concept BDI-based penetration testing tool in the simulated environment.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

[1]

Initial beliefs and intentions will be set up by the penetration tester and normally represents information regarding the target such as the domain or IP address and the privilege which the penetration testing must achieve, respectively

work page
[2]

For example, Nmap can collect OS type and ports opened at the target

The BDI agen t perceives the target information by performing various information gathering actions. For example, Nmap can collect OS type and ports opened at the target

work page
[3]

At this time, the BD I agent should hold the current information about the target

After perceivin g the feedback, current beliefs will be updated. At this time, the BD I agent should hold the current information about the target

work page
[4]

According to the new current belief, all relevant action plans will be found. For example, if port 80 of the target is opened, then password attack, buffer overflow attack, SQL injection attack are all become candidate options for the human penetration tester

work page
[5]

We pre-define the priority of the chosen actions in the human knowledge database

The BDI agent chooses one plan from the candidate action plans to become the intention and waits to be executed according to the context of the plan and the human knowledge database which chooses the plan based on human penetration testing experience in the real world. We pre-define the priority of the chosen actions in the human knowledge database

work page
[6]

If the plan fails , then the agent chooses another plan

The BDI agent executes the chosen plan. If the plan fails , then the agent chooses another plan

work page
[7]

Figure 3 The BDI agent reasoning cycle for penetration testing IV

The BDI agent che cks whether the initial goal is achieved or not and decides either (1) to output the report which records the process of the whole penetration testing or (2) to return back to the new reasoning cycle. Figure 3 The BDI agent reasoning cycle for penetration testing IV. IMPLEMENTATION OF BDI FOR PENETRATION TESTING Our model is implemented ...

work page
[8]

Hence, the BDI agent cannot perform local the buffer overflow attack as well since we define the prerequisite of it as successful password attack

Simulation 1 Figure 6 BDI agent result in simulation 1 We can see from the output of the processes of the BDI agent in Figure 6, the BDI agent probed all information about the target in the belief set but failed to perform the password attack because the rate of the password attack has not reached the specified 0.8 threshold. Hence, the BDI agent cannot p...

work page

Show all 19 references

[9]

Moreover, the BDI agent performed successfully in both the local and the remote buffer overflow attacks

Simulation 2 In this simulation, the BDI agent probed all the information of the target and successfully broke the SSH password because the rate of the SSH password attack was set to 0.9 which is greater than the 0.8 thres hold. Moreover, the BDI agent performed successfully i...
[10]

An overview of vulnerability assessment and penetration testing techniques,

S. Shah and B. M. Mehtre, “An overview of vulnerability assessment and penetration testing techniques,” J. Comput. Virol. Hacking Tech., vol. 11, no. 1, 2015

2015
[11]

Agent -based modeling and simulation,

C. M. Macal and M. J. North, “Agent -based modeling and simulation,” Proc. 2009 Winter Simul. Conf. , pp. 86–98, 2009

2009
[12]

Agent Architecture: An Overview,

K. On Chin, K. S. Gan, R. Alfred, P. Anthony, and D. Lukose, “Agent Architecture: An Overview,” Trans. Sci. Technol., vol. 1, no. 1, pp. 18–35, 2014

2014
[13]

BDI Agents: From Theory to Practice,

A. S. Rao and M. P. Georgeff, “BDI Agents: From Theory to Practice,” Proc. First Int. Conf. Multiagent Syst., vol. 95, pp. 312–319, 1995

1995
[14]

R. H. Bordini, J. F. Hü bner, and M. Wooldridge, Programming Mu lti-Agent Systems in AgentSpeak using Jason. 2007

2007
[15]

An automated method of penetration testing,

Xue Qiu, Shuguang Wang, Qiong Jia, Chunhe Xia, and Qingxin Xia, “An automated method of penetration testing,” 2014 IEEE Comput. Commun. IT Appl. Conf., pp. 211–216, 2014

2014
[16]

A Graph -based System for Network -vulnerability Analysis,

C. Phillips and L. P. Swil er, “A Graph -based System for Network -vulnerability Analysis,” Proc. 1998 Work. New Secur. Paradig., pp. 71–79, 1998

1998
[17]

Modeling modern network attacks and countermeasures using attack graphs,

K. Ingols, M. Chu, R. Lippmann, S. Webster, and S. Boyer, “Modeling modern network attacks and countermeasures using attack graphs,” Proc. - Annu. Comput. Secur. Appl. Conf. ACSAC , pp. 117 –126, 2009

2009
[18]

Automatic generation algorithm of penetration graph in penetration testing,

X. Qiu, Q. Jia, S. Wang, C. Xia, and L. S. Lv, “Automatic generation algorithm of penetration graph in penetration testing,” Proc. - 2014 9th Int. Conf. P2P, Parallel, Grid, Cloud Intern et Comput. 3PGCIC 2014, pp. 531–537, 2014

2014
[19]

AGENT - BASED MODELING: AN INTRODUCTION AND PRIMER,

C. W. Weimer, J. O. Miller, and R. R. Hill, “AGENT - BASED MODELING: AN INTRODUCTION AND PRIMER,” pp. 65–79, 2016

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Initial beliefs and intentions will be set up by the penetration tester and normally represents information regarding the target such as the domain or IP address and the privilege which the penetration testing must achieve, respectively

work page

[2] [2]

For example, Nmap can collect OS type and ports opened at the target

The BDI agen t perceives the target information by performing various information gathering actions. For example, Nmap can collect OS type and ports opened at the target

work page

[3] [3]

At this time, the BD I agent should hold the current information about the target

After perceivin g the feedback, current beliefs will be updated. At this time, the BD I agent should hold the current information about the target

work page

[4] [4]

According to the new current belief, all relevant action plans will be found. For example, if port 80 of the target is opened, then password attack, buffer overflow attack, SQL injection attack are all become candidate options for the human penetration tester

work page

[5] [5]

We pre-define the priority of the chosen actions in the human knowledge database

The BDI agent chooses one plan from the candidate action plans to become the intention and waits to be executed according to the context of the plan and the human knowledge database which chooses the plan based on human penetration testing experience in the real world. We pre-define the priority of the chosen actions in the human knowledge database

work page

[6] [6]

If the plan fails , then the agent chooses another plan

The BDI agent executes the chosen plan. If the plan fails , then the agent chooses another plan

work page

[7] [7]

Figure 3 The BDI agent reasoning cycle for penetration testing IV

The BDI agent che cks whether the initial goal is achieved or not and decides either (1) to output the report which records the process of the whole penetration testing or (2) to return back to the new reasoning cycle. Figure 3 The BDI agent reasoning cycle for penetration testing IV. IMPLEMENTATION OF BDI FOR PENETRATION TESTING Our model is implemented ...

work page

[8] [8]

Hence, the BDI agent cannot perform local the buffer overflow attack as well since we define the prerequisite of it as successful password attack

Simulation 1 Figure 6 BDI agent result in simulation 1 We can see from the output of the processes of the BDI agent in Figure 6, the BDI agent probed all information about the target in the belief set but failed to perform the password attack because the rate of the password attack has not reached the specified 0.8 threshold. Hence, the BDI agent cannot p...

work page