{"id":"f570e131-ea51-4c74-9a46-a8b286f727a9","arxiv_id":"1908.06970","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A proof-of-concept using BDI agents in Jason to automate penetration testing, demonstrated on two simulated targets with author-set success probabilities.","lead":"The paper applies a standard artificial intelligence agent model, BDI, to automate penetration testing, where an agent gathers information and launches attacks based on plans. It shows a prototype working in two small simulated scenarios, though the scenarios are set up so the chosen attacks succeed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The simulations do not exercise the dynamic interactivity the BDI model claims: the target is a static table and uncertainty is a single random draw, so the central validation claim is unsupported.","rationale":"The paper is a modest proof-of-concept and is honest about its prototype status and simulated environment. The BDI formalization and Jason implementation are coherent, and the simulations do exhibit information gathering and plan selection. However, the abstract's central claim that BDI automation deals interactively with dynamic, uncertain and complex environments is not supported by the experiments. The target in Section V is static: all facts are present in the initial belief set (Table II), and the only uncertainty is a random draw against author-chosen thresholds. No environmental change occurs during execution, so the BDI reasoning cycle's re-planning and intention-revision capabilities are never stressed. This is more fundamental than the reader's threshold concern: even perfectly calibrated thresholds would not make the simulation dynamic. The threshold values are unvalidated and no baseline comparison exists, so the claimed improvement in success probability is unquantified. I therefore keep the CONDITIONAL verdict, but for a slightly different primary reason. The concrete test—introducing genuine environmental change and comparing the Jason trace to a scripted decision tree—would settle whether the central claim lands.","tokens_in":6320,"tokens_out":4254,"duration_ms":47127,"concrete_test":"Run the same BDI prototype against a target that actually changes state during the attack—for example, an SSH service that rate-limits or disables after failed attempts, a web server that patches a vulnerability mid-scan, or a second target that comes online after initial compromise—and instrument the Jason reasoning cycle to record intention revisions triggered by these events. A complementary check is to replace the Jason layer with a hand-coded decision tree using the same thresholds and run the same two simulations: if the traces are identical, the BDI architecture contributes nothing beyond the pre-defined plan priorities.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that the BDI approach 'deals interactively with dynamic, uncertain and complex environments.' The validation in Section V does not test that property. The target agent is initialized with a fixed belief set (Table II: OS, ports, services, vulnerabilities, password), and the interaction between the BDI agent and the target is simulated with Jason internal communication actions (Section V, Figure 5). The only uncertainty is a random-number draw compared with author-chosen thresholds (0.8 for SSH password attack, 0.5 remote and 0.3 local buffer overflow, Section V.A). Nothing in the environment changes during a run: no new ports open, no services change, no defenses react, no new information forces the BDI reasoning cycle to revise an intention mid-plan. Simulations 1 and 2 merely show two branches of a static decision tree (password fails or succeeds, then buffer overflow succeeds). A hand-coded if-then-else script with random outcomes would produce the same traces; the BDI architecture's claimed benefit—interactive adaptation to a dynamic environment—is never exercised. The thresholds are also unvalidated and no baseline comparison with attack-graph or AEPT approaches is provided, so even the claimed 'higher probability of success' is unquantified. Thus the conclusion that the two simulations 'validate the modelling' overreaches: the mechanism is exhibited, but the central environmental claim is not validated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a belief-desire-intention (BDI) agent model for automating penetration testing. Penetration testing actions are represented as BDI plans, and the Jason implementation of AgentSpeak is used to implement a prototype. The authors argue that, unlike attack-graph-based and automata-based approaches, the BDI reasoning cycle enables interactive adaptation to dynamic, uncertain, and complex target environments. Two simulations are presented in which a BDI agent gathers information about a simulated target and performs password and buffer-overflow attacks; the paper claims these simulations validate the model. The paper concludes with future work on extending the action set and experimenting in real environments.","tokens_in":6630,"tokens_out":2230,"duration_ms":24436,"significance":"If the central claim were substantiated, an interactive BDI-based approach to pentesting automation could be a novel and useful contribution, since existing automated approaches are largely static. The paper correctly identifies a real limitation of prior work: non-interactive processing cannot recover from failed attack attempts or adapt to new information. However, the manuscript does not provide evidence for this claimed advantage. The simulations in Section V exercise only a static decision tree with random outcomes; they do not demonstrate that the BDI reasoning cycle adds value over a simple script. The formal model in Section III is a tuple and three equations with undefined functions, so it is not a complete specification against which an implementation could be verified. The paper ships a prototype and shows executable traces, which is a useful starting point, but the evidence base is far too thin to support the conclusion that the BDI model 'validates' automated interactive pentesting or that it achieves 'a higher probability of success'.","major_comments":[{"comment":"The validation evidence is circular and unquantified. In Section V.A the success thresholds are set to 0.8 for SSH password attack, 0.5 for remote buffer overflow, and 0.3 for local buffer overflow, 'based on personal penetration testing experience of the first author.' These thresholds fully determine the simulation outcomes: in Simulation 1 the random draw for the password attack is below 0.8, so it fails; in Simulation 2 the draw is 0.9, so it succeeds. The paper presents two runs without error bars, repeated trials, or statistical analysis, and no baseline comparison with AEPT or attack-graph approaches. Consequently, the Section VI claim that the model improves 'efficiency and probability of success' is unsupported. The simulations show only that the agent executes the pre-scripted plans, not that the modeled probabilities correspond to real attack success rates or that the approach is superior to existing methods.","section":"V.C"},{"comment":"The formalization of the BDI model is a placeholder, not a specification. The equations B = f1(B × S), D = f2(B × I), and I = f3(B × D × I) use functions f1, f2, and f3 that are never defined, and the domains and semantics of the update operations are not given. The reasoning cycle in Section III is described in prose and in Figure 3, but there is no formal link between the equations and the Jason implementation. This makes it impossible to verify that the implementation is faithful to the model, and it weakens the paper's claim of a model-based approach. The authors should either provide well-defined update functions (e.g., in terms of plan selection and perception) or explicitly state that the equations are schematic and give a precise operational semantics.","section":"Section III, Equations (1)-(3)"},{"comment":"The experiments are not reproducible or comparable as reported. There is no description of the random seed, the number of runs, or the distribution from which the random numbers are drawn. The target is an idealized agent whose beliefs contain the exact vulnerability information (CVE-remote, CVE-local) and password ('SSH:456'), so the information-gathering stage is effectively a direct readout of the target's internal state. The paper does not compare against the methods discussed in Section II (AEPT, attack graphs), so the assertion that the BDI approach overcomes their limitations is not tested. At minimum, the authors should run many simulations, report success-rate distributions, and compare against a scripted non-BDI baseline on the same simulated target.","section":"Section V, experimental methodology"}],"minor_comments":[{"comment":"The citation 'Bradman [4]' should be 'Bratman' (Michael Bratman, the originator of the BDI framework). The reference list should be checked carefully for spelling and completeness.","section":"Section I and references"},{"comment":"The caption for the second simulation reads 'Figure 6 BDI agent result in simulation 2', which duplicates the figure number of the first simulation and should presumably be Figure 8. Also, the text refers to 'Future 7' but the figure is likely 'Figure 7'. These cross-reference errors need correction.","section":"Section V.C, captions"},{"comment":"The paper frequently states that the BDI model 'enables interaction between dynamic and uncertain targets' but the description of the reasoning cycle (items 1-7 in Section III) is a fixed sequence with no branch for a mid-plan environmental change. Clarify what interactions are actually supported and where the cycle handles a new perception arriving during plan execution.","section":"General"},{"comment":"The sentence describing the target agent's password threshold is ambiguous: 'we set 0.8 as the threshold to determine if the SSH password attack is successful' and later in Simulation 2 'the rate of the SSH password attack was set to 0.9 which is greater than the 0.8 threshold.' Clarify whether 0.8 is an attack-success probability used in a single draw or a parameter that the user can set per simulation.","section":"Section V.A"}],"recommendation":"reject","confidential_remarks":"The manuscript is a very early-stage position/implementation sketch. The central claim of interactive, adaptive pentesting is not validated: the experiments are illustrative traces, not evaluations, and the formal model is under-specified. The paper may be suited to a workshop or a short demo paper, but it does not meet the bar for a full journal publication in its current form. The main risks are (1) the absence of a dynamic environment in the experiments, (2) the arbitrarily chosen success thresholds that trivially determine the outcomes, and (3) the lack of any baseline comparison. These issues are unlikely to be fixable by a small revision, which is why I recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is reasonable: model penetration testing as a BDI agent that selects attack plans based on gathered evidence, and implement it in Jason so external tools like Nmap and Metasploit can be invoked. That is a legitimate extension of established agent technology to a new domain, and the paper is honest about being a prototype. The survey of attack-graph approaches and the argument that they are mostly non-interactive is fair and points to a real gap.\n\nThe problem is the validation. The two simulations do not test the thing the abstract promises: interactive dealing with dynamic, uncertain and complex environments. The target is a static table of OS, ports, services and vulnerabilities. Nothing changes during a run—no new ports open, no services change, no defenses react. Uncertainty is a single random draw compared against author-set thresholds (0.8 for SSH password attack, 0.5 remote and 0.3 local buffer overflow, said to come from the first author's experience). In Simulation 2 the password attack succeeds because the random number is 0.9, above the 0.8 threshold. A hand-coded if-then-else script with random outcomes would produce the same traces. The BDI reasoning cycle's claimed benefit—revising intentions mid-plan based on environment feedback—is never actually exercised. So the conclusion that the simulations 'validate the modelling' overreaches.\n\nThere are softer issues too. Equations (1)-(3) are placeholders with undefined functions. There is no baseline comparison with AEPT or attack-graph techniques, so the claimed 'higher probability of success' is unquantified. No code is released, which makes the prototype hard to assess. The related work section does not survey other agent-based pentesting proposals, so the novelty claim is not fully benchmarked—though the specific BDI plan-based model does appear to be new relative to the cited literature.\n\nNone of this makes the model itself unsound. The BDI architecture is a plausible fit for choosing and retrying attack plans. The flaw is in the evidence: the central claim is unsupported by the present simulations, not contradicted by them.\n\nWho is this for? Someone working on automated security testing and curious about whether cognitive agent architectures add anything over planning-based approaches. It is a short proof-of-concept, not a system paper. It deserves a serious referee for a workshop or applied conference, but it would need substantial revision to be accepted: a truly dynamic target model, repeated trials, a baseline comparison, and ideally released code. If I were editor, I would send it to review rather than desk reject, because the direction is worth engaging with, but I would expect the validation section to be redone.","headline":"Plausible BDI-to-pentesting mapping, but the simulations never exercise the dynamic interactivity that is the paper's central claim.","tokens_in":7096,"tokens_out":1697,"would_cite":false,"duration_ms":19202,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a belief-desire-intention (BDI) agent, implemented in Jason, automates penetration testing by encoding attacks as plans and using a reasoning cycle to interact with dynamic, uncertain targets.","keywords":["penetration testing","BDI agent","belief-desire-intention","automated pentesting","Jason","attack plans","multi-step attacks"],"falsifier":"Run the same BDI agent in a real network or a benchmark with empirically measured exploit success rates, and compare whether the agent selects and times its attacks as well as, or better than, a human tester or a graph-based planner. If the hand-set thresholds are replaced by realistic probabilities and the agent's success drops sharply, the claimed benefit over static planners is not established.","tokens_in":6156,"feed_emoji":"🤖","tokens_out":3331,"duration_ms":29838,"temperature":0.7,"pith_summary":"The paper proposes automating penetration testing with a belief-desire-intention (BDI) agent instead of the static attack-graph planning used in prior tools. Penetration testing actions are encoded as BDI plans, and the agent's reasoning cycle—perceive, update beliefs, select an intention, execute, retry on failure—drives interaction with targets. A prototype implemented in Jason combines external tools such as Nmap, Metasploit, SQLmap, and Hydra, and two simulations demonstrate the agent escalating to root privilege through different attack paths. The stated goal is to handle dynamic, uncertain, and complex target environments that non-interactive automation cannot.","feed_headline":"BDI agents automate penetration testing through interactive reasoning","feed_subtitle":"A Jason-based agent perceives targets, picks attack plans, and retries on failure—something static attack-graph planners cannot do.","key_machinery":"The central mechanism is the BDI reasoning cycle implemented in Jason (AgentSpeak): the agent perceives target information via tools like Nmap and OpenVAS, updates its belief set, derives candidate plans whose context matches current beliefs, selects one plan as an intention using priorities from a human knowledge database, executes it, and on failure loops back to choose another plan. The plan structure with trigger event, context and body is what carries the automation, making the model extensible by adding new plans.","core_discovery":"The central claim is that the BDI agent model can automate penetration testing by treating attacks as plans selected through a reasoning cycle, giving the automation an interactive, reactive capability that graph-based and automata-based approaches lack. The paper defines the agent as a tuple of beliefs, desires, intentions, plans, actions and perceptions, and specifies how beliefs update from scans, how candidate plans become intentions via a human-knowledge priority database, and how failed attacks trigger retry with an alternative plan. Two simulated runs show the agent gathering target information, failing a password attack and then succeeding via remote buffer overflow in one case, and succeeding at password plus local and remote buffer overflow in the other, reaching root privilege.","pith_inferences":["If the BDI approach generalizes beyond simulation, it suggests that the bottleneck in automated pentesting shifts from path planning to encoding expert decision knowledge as plan priorities and success probabilities.","The same plan-trigger-context-body structure could be applied to other security operations that require interactive decision-making, such as incident response or red-team simulation, where conditions change mid-operation.","A testable extension would be to replace the hand-set success thresholds with data from real exploit outcomes or public vulnerability statistics, then compare agent success rates against those baselines."],"forward_implications":["Automated pentesting can progressively escalate privileges (none to user to root) through multi-step attacks within a single reasoning session.","The model can recover from failed attack attempts by selecting an alternative plan, something static attack-graph planners cannot do.","New attack types can be added as new BDI plans elicited from human experts, without reworking the core agent.","External tools (Nmap, Metasploit, SQLmap, Hydra, Ettercap, SET) can be invoked as part of the action space, so the framework integrates with existing toolchains."],"supporting_citations":[{"why":"Supplies the BDI theory and architecture the model builds on.","marker":"[4]"},{"why":"Provides the Jason language and interpreter used for the implementation.","marker":"[5]"},{"why":"The automata-based AEPT approach that the paper contrasts as non-interactive.","marker":"[6]"},{"why":"Foundational attack-graph method the paper claims is static and non-interactive.","marker":"[7]"},{"why":"NetSPA attack-graph system representing the static planning baseline.","marker":"[8]"}],"fun_headline_variants":["BDI agents automate pentesting with adaptive plans","Interactive BDI agents take on pentesting","Pentesting that adapts: BDI agents in action","Automated pentesting with reactive BDI agents","BDI reasoning drives adaptive penetration testing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation relies on success probabilities chosen by the first author's experience (0.8 for SSH password attack, 0.5 for remote, and 0.3 for local buffer overflow), so the demonstrated infiltrations only validate the model if those thresholds match real-world attack success rates.","fun_headline_variants_meta":{"raw":{"variants":["BDI agents automate pentesting with adaptive plans","Interactive BDI agents take on pentesting","Pentesting that adapts: BDI agents in action","Automated pentesting with reactive BDI agents","BDI reasoning drives adaptive penetration testing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1817,"prompt_tokens":897,"completion_tokens":920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":850}},"tokens_in":513,"tokens_out":920,"duration_ms":7810,"temperature":1.0,"reasoning_tokens":850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:44:07.899496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same BDI agent in a real network or a benchmark with empirically measured exploit success rates, and compare whether the agent selects and times its attacks as well as, or better than, a human tester or a graph-based planner. If the hand-set thresholds are replaced by realistic probabilities and the agent's success drops sharply, the claimed benefit over static planners is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BDI theory and architecture the model builds on."},{"cited_title":"We pre-define the priority of the chosen actions in the human knowledge database","cited_arxiv_id":null,"evidence_quote":"Provides the Jason language and interpreter used for the implementation."},{"cited_title":"If the plan fails , then the agent chooses another plan","cited_arxiv_id":null,"evidence_quote":"The automata-based AEPT approach that the paper contrasts as non-interactive."},{"cited_title":"Figure 3 The BDI agent reasoning cycle for penetration testing IV","cited_arxiv_id":null,"evidence_quote":"Foundational attack-graph method the paper claims is static and non-interactive."},{"cited_title":"Hence, the BDI agent cannot perform local the buffer overflow attack as well since we define the prerequisite of it as successful password attack","cited_arxiv_id":null,"evidence_quote":"NetSPA attack-graph system representing the static planning baseline."}],"review_version":1}