Pith. sign in

REVIEW 3 major objections 5 minor 26 references

AI planner turns network scans into guaranteed hardening strategies

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 21:25 UTC pith:I7RXKDPP

load-bearing objection Demo-track tool integrating SPEAR planning with LLM explanations; the new automated pipeline is unvalidated the 3 major comments →

arxiv 2607.05888 v1 pith:I7RXKDPP submitted 2026-07-07 cs.CR cs.AI

i-EXAM: Instructable and Explainable Attack Connectivity Graph Modeler

Rakesh Podder , Wadia Ganim , Sarath Sreedharan , Indrajit Ray , Indrakshi Ray This is my paper
classification cs.CR cs.AI
keywords network securityautomated planningPDDLattack graphattack connectivity graphhardening strategiesexplainable AILLM explanation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

i-EXAM is an interactive tool that bridges automated AI planning and network security administration. Its central claim is that by compiling network scanner data into formal planning problems, it can automatically identify attack paths, compute security metrics, generate diverse hardening strategies with optimality guarantees, and explain those strategies in natural language. The tool builds on the SPEAR framework's Attack Connectivity Graph (ACG), a hypergraph that simultaneously models attack paths and service connectivity. i-EXAM automates the entire pipeline: it ingests data from standard network scanners, converts it to PDDL planning models, solves them using optimal search, and uses an LLM to translate the results into plain-English explanations for sysadmins who need no expertise in formal planning.

Core claim

The paper demonstrates that the gap between formal AI planning theory and practical network security administration can be closed by an automated pipeline. The key mechanism is the compilation of network state into PDDL, which allows security questions (can an attacker reach this target? what is the cheapest attack? what changes block all attacks?) to be answered with formal soundness and completeness guarantees inherited from the underlying planning framework. The LLM explanation layer converts structured counterfactual plan traces into natural language, making the formal reasoning accessible to administrators without planning expertise.

What carries the argument

Attack Connectivity Graph (ACG): a hypergraph capturing both attack paths and service connectivity in a network, compiled into PDDL planning problems where host-attribute pairs become fluents and attack/connectivity edges become actions.

Load-bearing premise

The formal guarantees hold only over the planning model, not over the actual network. If the automated scanner-to-PDDL pipeline misses vulnerabilities, misconfigurations, or zero-days that the scanners cannot detect, the soundness and completeness guarantees become vacuous with respect to the real system being defended.

What would settle it

If the JSON-to-PDDL translation loses security-relevant state present in the scanner outputs, or if the ACG model cannot represent attack vectors that real adversaries use, then the optimal hardening strategies would be formally correct but practically incomplete.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This demo-track paper presents i-EXAM, an interactive tool built on the SPEAR framework (Podder et al. 2025) that automates the pipeline from network scanner outputs (Nmap, Wazuh, OpenVAS) through JSON representation to PDDL planning models, and uses LLMs to generate natural-language explanations of hardening strategies. The tool supports visualization of attack paths, computation of two security metrics (impenetrability M1 and attack difficulty M2), diverse hardening strategy enumeration, and counterfactual explanations. The formal soundness and completeness guarantees are inherited from SPEAR's Theorem 1 and Proposition 4.

Significance. The integration of formal AI planning with practical network security tooling is a worthwhile contribution, and the tool addresses a genuine practitioner need. The use of diverse solution selection and LLM-based explanation of counterfactual plan failures is a reasonable design choice for accessibility. The demo video and case-study screenshot provide concrete evidence of a working system. However, the paper's new contribution over SPEAR—the automated scanner-to-PDDL pipeline—lacks the specification and validation needed to support the claims made about it.

major comments (3)
  1. Generalization section: The paper claims the automated pipeline 'constructs planning models for any network without manual modeling effort' and that i-EXAM 'inherits SPEAR's soundness and completeness guarantees.' However, Theorem 1 (Podder et al. 2025) establishes a correspondence between plans and ACG paths over a correctly constructed model. The new contribution of i-EXAM is the automated JSON-to-PDDL translation, but the paper provides no specification of the translation rules, no example PDDL output for a known configuration, and no comparison between manually-constructed and automatically-generated models. Without any evidence that the translation preserves ACG semantics (e.g., vulnerability preconditions requiring both network reachability and a specific vulnerable service, or the F1_C connectivity preservation requirement), the claim that formal guarantees transfer to the auto-g
  2. Explanation Generation section: The LLM component (llama-3.1-nemotron-70b-instruct) is presented as converting 'plan failure information into natural language,' but no example input/output pair is shown, and there is no discussion of how the LLM is prompted or what happens when the LLM misrepresents the planner's counterfactual information. For a tool whose name includes 'Explainable,' this component is underspecified. Even a single worked example from the case study in Fig. 1 would substantially strengthen the contribution.
  3. Scalability section: The scalability claims reference empirical results from Podder et al. 2025 on 30-node networks, but it is unclear whether these results were obtained with i-EXAM's automated pipeline or with manually constructed models. If the latter, the overhead of the automated pipeline (scanner data collection, JSON processing, PDDL generation) is not discussed. The claim that 'the top-k planner enables enumeration of diverse attack paths even in larger networks' is stated without any evidence.
minor comments (5)
  1. The impenetrability metric M1 is written as F^I_A(E_A) = 1 iff |E_A| = 0, but the notation F^I_A is not defined beyond this equation; clarifying what the superscript I and subscript A denote would help readers unfamiliar with SPEAR.
  2. The M2 compilation mentions auxiliary fluents (init_change_mode, act_mode, goal_reached) but these are not formally defined; a brief description of their role would improve self-containedness.
  3. The reference to 'Dubey et al. 2024' for the Llama 3 model is cited, but the specific model used (llama-3.1-nemotron-70b-instruct) appears to be an NVIDIA variant; the citation may need updating to the correct source.
  4. The paper mentions 'other scanners' and 'documented data' in the Building Planning Models section without specifying what these are; this vagueness makes the pipeline hard to reproduce.
  5. Fig. 1 is referenced but the screenshot is too small to read in the provided text; if accepted, ensure the figure is legible in the camera-ready.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful and constructive review. The referee correctly identifies that the automated scanner-to-PDDL pipeline, the LLM explanation component, and the scalability claims are underspecified in the current manuscript. We agree with all three major comments and will revise the paper accordingly. Below we address each point and describe the specific revisions we will make.

read point-by-point responses

  1. Referee: Generalization section: The paper claims the automated pipeline 'constructs planning models for any network without manual modeling effort' and that i-EXAM 'inherits SPEAR's soundness and completeness guarantees.' However, Theorem 1 (Podder et al. 2025) establishes a correspondence between plans and ACG paths over a correctly constructed model. The new contribution of i-EXAM is the automated JSON-to-PDDL translation, but the paper provides no specification of the translation rules, no example PDDL output for a known configuration, and no comparison between manually-constructed and automatically-generated models. Without any evidence that the translation preserves ACG semantics (e.g., vulnerability preconditions requiring both network reachability and a specific vulnerable service, or the F1_C connectivity preservation requirement), the claim that formal guarantees transfer to the auto-g

    Authors: The referee is correct. The current manuscript does not specify the translation rules from JSON to PDDL, does not provide example PDDL output, and does not demonstrate semantic preservation between automatically generated and manually constructed models. We agree that the claim about inheriting SPEAR's formal guarantees cannot be made without evidence that the automated translation preserves ACG semantics. In the revision we will: (1) include a specification of the JSON-to-PDDL translation rules, covering how vulnerability preconditions (requiring both network reachability and a specific vulnerable service) and connectivity preservation constraints (F1_C) are encoded; (2) add a worked example showing the PDDL domain and problem files generated from a known small network configuration; and (3) include a comparison between manually constructed and automatically generated models on the same network to demonstrate semantic equivalence. We will also qualify the generalization claim to state that the pipeline constructs models for networks whose scanner outputs cover the attribute types required by the ACG formalism, rather than claiming coverage of 'any network' without qualification. revision: yes

  2. Referee: Explanation Generation section: The LLM component (llama-3.1-nemotron-70b-instruct) is presented as converting 'plan failure information into natural language,' but no example input/output pair is shown, and there is no discussion of how the LLM is prompted or what happens when the LLM misrepresents the planner's counterfactual information. For a tool whose name includes 'Explainable,' this component is underspecified. Even a single worked example from the case study in Fig. 1 would substantially strengthen the contribution.

    Authors: We agree. The LLM explanation component is underspecified for a tool whose name includes 'Explainable.' In the revision we will add: (1) the prompt template used to query the LLM, showing how structured counterfactual plan traces from the planner are formatted as input; (2) at least one concrete input/output pair from the case study shown in Fig. 1, including the planner's counterfactual trace and the LLM's generated natural-language explanation; and (3) a discussion of failure modes when the LLM misrepresents the planner's output, including the mitigation strategy of presenting the raw planner trace alongside the LLM-generated explanation so the sysadmin can verify the explanation against the formal output. We acknowledge that without these details the reader cannot evaluate the reliability or usefulness of the explanation component. revision: yes

  3. Referee: Scalability section: The scalability claims reference empirical results from Podder et al. 2025 on 30-node networks, but it is unclear whether these results were obtained with i-EXAM's automated pipeline or with manually constructed models. If the latter, the overhead of the automated pipeline (scanner data collection, JSON processing, PDDL generation) is not discussed. The claim that 'the top-k planner enables enumeration of diverse attack paths even in larger networks' is stated without any evidence.

    Authors: The referee is correct on both points. The empirical results cited from Podder et al. 2025 were obtained with manually constructed models, not with i-EXAM's automated pipeline. The current manuscript does not discuss the overhead of the automated pipeline (scanner data collection, JSON processing, PDDL generation), and the claim about the top-k planner enabling enumeration in larger networks is stated without supporting evidence. In the revision we will: (1) clarify that the cited 30-node results used manually constructed models; (2) report the overhead introduced by the automated pipeline stages (scanner data collection, JSON processing, PDDL generation) measured on our case-study network; and (3) either provide empirical evidence for the top-k planner claim on larger networks or remove the unsupported claim and note it as future work. We agree that the scalability section as written overstates what has been demonstrated. revision: yes

Circularity Check

0 steps flagged

No significant circularity; self-citation to SPEAR is load-bearing for formal guarantees but independently published, and the paper's new contributions (automated pipeline, LLM explanation) are not claimed to be formally guaranteed.

full rationale

The paper is a tool/demo paper that builds on the SPEAR framework (Podder et al. 2025). The formal guarantees — Theorem 1 (soundness/completeness of ACG-to-PDDL compilation), Proposition 4 (optimal plan cost equals attack difficulty metric M2), and the definitions of metrics M1/M2 — are all explicitly attributed to the prior work and stated as 'inherited,' not re-derived or claimed as novel results of this paper. The author overlap (Podder, Sreedharan, Ray, Ray on both papers) constitutes self-citation, but SPEAR is independently published at ACM SACMAT 2025, so the citation points to an external, peer-reviewed result rather than an unverified claim. The paper's actual new contributions — the automated scanner-to-PDDL pipeline, the visualization UI, the diverse solution selection, and the LLM-based explanation generation — are not presented as formally guaranteed; the paper is careful to scope the guarantees to the ACG/PDDL model ('any valid plan in the model corresponds to a valid attack or connectivity path in the ACG, and vice versa'). The reader's concern that the JSON-to-PDDL translation may not preserve ACG semantics is a legitimate correctness risk (the translation rules are not formally specified), but it is not a circularity: the paper does not define the translation in terms of the guarantees it claims to produce, nor does it fit a parameter and call the fit a prediction. No step in the derivation chain reduces to its own inputs by construction. Score 2 reflects the minor self-citation that is load-bearing for the 'formal guarantees' framing but does not undermine the paper's independent contributions.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The axiom ledger reveals that i-EXAM's formal guarantees are entirely inherited from the SPEAR framework (Podder et al. 2025), and the tool's real-world correctness depends on the unstated assumption that scanner outputs faithfully capture network state. The LLM explanation accuracy is assumed but not evaluated. Two free parameters (hardening costs, top-k) are used but not specified.

free parameters (2)
  • Hardening action costs = Not specified
    The paper states that 'the cost of a modification is quite hard to capture exactly' and provides diverse solutions instead. No cost function is specified, but A* with LMCut requires costs for optimal hardening. The paper mentions 'learned cost functions' as future work, implying current costs are unspecified or default.
  • Top-k parameter for attack path enumeration = Not specified
    The top-k planner (Katz et al. 2018) is used to visualize multiple attack paths, but k is not specified.
axioms (4)
  • domain assumption The ACG-to-PDDL compilation is sound and complete (Theorem 1, Podder et al. 2025): a plan exists iff an attack path exists in the ACG.
    Invoked in 'Network-Level Analysis' and 'Discussion' sections. This is the foundational guarantee the tool claims to inherit. It is a theorem from the cited prior work, not re-proven here.
  • domain assumption The minimum-cost plan in the compiled M2 model equals the attack difficulty metric F^D_A (Proposition 4, Podder et al. 2025).
    Invoked in 'Network-Level Analysis' section. Another inherited guarantee from prior work.
  • domain assumption Scanner outputs (Nmap, Wazuh, OpenVAS) faithfully capture the security-relevant state of the network.
    Implicit in the 'Building Planning Models' section. The formal guarantees hold over the PDDL model, but the model's fidelity to the real network depends on this unstated assumption.
  • ad hoc to paper LLM-generated explanations accurately reflect the planner's counterfactual plan failure information.
    The 'Explanation Generation' section describes using llama-3.1-nemotron-70b-instruct to convert plan failure information into natural language, but no evaluation of explanation accuracy is provided.
invented entities (1)
  • i-EXAM tool (software system) no independent evidence
    purpose: Interactive tool for network security analysis, visualization, hardening, and explanation
    The tool is described and a demo video is linked, but no source code, no empirical evaluation, and no independent deployment data are provided. The case study (Fig. 1) is a screenshot, not a falsifiable test.

pith-pipeline@v1.1.0-glm · 7510 in / 3110 out tokens · 395771 ms · 2026-07-08T21:25:03.509643+00:00 · methodology

0 comments
read the original abstract

i-EXAM is a planning-powered tool that helps system administrators to create security profiles of complex networks and perform what-if analyses to identify network hardening strategies. It leverages planning compilation that provides soundness and completeness guarantees to identify attack paths, evaluate security metrics, generate diverse hardening strategies, and explain these strategies in natural language using Large Language Models.

Figures

Figures reproduced from arXiv: 2607.05888 by Indrajit Ray, Indrakshi Ray, Rakesh Podder, Sarath Sreedharan, Wadia Ganim.

Figure 1
Figure 1. Figure 1: is a screenshot from a case-study where i-EXAM was instantiated with a test network given the system configu￾rations, network connectivity, CVEs, and a list of possible targets/goals. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    Foundations of

    Sreedharan, Sarath and Chakraborti, Tathagata and Kambhampati, Subbarao , keywords =. Foundations of. Artificial Intelligence , volume=. 2021 , publisher=

    work page 2021

  2. [2]

    arXiv e-prints , pages=

    The Llama 3 Herd of Models , author=. arXiv e-prints , pages=

    work page

  3. [3]

    , author=

    Domain Independent Approaches for Finding Diverse Plans. , author=. IJCAI , pages=

    work page

  4. [4]

    A Novel Iterative Approach to Top-k Planning

    Michael Katz and Shirin Sohrabi and Octavian Udrea and Dominik Winterer. A Novel Iterative Approach to Top-k Planning. Proceedings of the Twenty-Eighth International Conference on Automated Planning and Scheduling (ICAPS 2018). 2018

    work page 2018

  5. [5]

    Proceedings of the 30th ACM Symposium on Access Control Models and Technologies , pages=

    SPEAR: Security Posture Evaluation using AI Planner-Reasoning on Attack-Connectivity Hypergraphs , author=. Proceedings of the 30th ACM Symposium on Access Control Models and Technologies , pages=

    work page

  6. [6]

    Explainable

    Ray, Indrajit and Sreedharan, Sarath and Podder, Rakesh and Bashir, Shadaab Kawnain and Ray, Indrakshi , booktitle=. Explainable. 2023 , volume=

    work page 2023

  7. [7]

    2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA) , pages=

    Resiliency Graphs: Modelling the Interplay between Cyber Attacks and System Failures through AI Planning , author=. 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA) , pages=. 2024 , organization=

    work page 2024

  8. [8]

    How the Mind Explains Behavior: Folk Explanations, Meaning, and Social Interaction

    Malle, Bertram F , year=. How the Mind Explains Behavior: Folk Explanations, Meaning, and Social Interaction

    work page

  9. [9]

    , title = "

    Ou, Xinming and Govindavajhala, Sudhakar and Appel, Andrew W. , title = ". Proceedings of the 14th Conference on USENIX Security Symposium , volume=. 2005 , publisher =

    work page 2005

  10. [10]

    A Planner-Based Approach to Generate and Analyze Minimal attack Graph

    Ghosh, Nirnay and Ghosh, Soumya K , journal=. A Planner-Based Approach to Generate and Analyze Minimal attack Graph. 2012 , publisher=

    work page 2012

  11. [11]

    P D D L A ssistant: A Tool for Assisting Construction and Maintenance of Attack Graphs Using P D D L

    Tiwary, Kushagra and Weerawardhana, Sachini and Ray, Indrajit and Howe, Adele , booktitle=. P D D L A ssistant: A Tool for Assisting Construction and Maintenance of Attack Graphs Using P D D L

    work page

  12. [12]

    Bezawada, Bruhadeshwar and Ray, Indrajit and Tiwary, Kushagra , booktitle=. A. 2019 , organization=

    work page 2019

  13. [13]

    Journal of Artificial Intelligence Research , volume=

    The Fast Downward Planning System , author=. Journal of Artificial Intelligence Research , volume=

    work page

  14. [14]

    Proceedings of the International Conference on Automated Planning and Scheduling , volume=

    Landmarks, Critical Paths and Abstractions: What's the Difference Anyway? , author=. Proceedings of the International Conference on Automated Planning and Scheduling , volume=

    work page

  15. [15]

    Journal of Artificial Intelligence Research , volume=

    Contrastive Explanations of Plans through Model Restrictions , author=. Journal of Artificial Intelligence Research , volume=

    work page

  16. [16]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

    work page

  17. [17]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

    work page

  18. [18]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    work page 1980

  19. [19]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

    work page

  20. [20]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    work page doi:10.1016/s0020-7373(84)80003-6 1984

  21. [21]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

    work page

  22. [22]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

    work page

  23. [23]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

    work page

  24. [24]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

    work page

  25. [25]

    2017 , eprint=

    Attention Is All You Need , author=. 2017 , eprint=

    work page 2017

  26. [26]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

    work page