REVIEW 5 major objections 5 minor 18 references
This paper claims that reducing attack-chain predicate categories from nine to five leaves plan validity and cost unchanged on a sixteen-technique corpus, while acknowledging that this particular comparison is guaranteed by the construction
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 01:10 UTC pith:5SPYIU43
load-bearing objection The headline invariance claim is an artifact of the paper's own relabeling construction, but the honest failure-mode data and the A/B test on remote UAC token filtering are real contributions worth engaging. the 5 major comments →
Symbolic Attack Chain Generation from Atomic Red Team Techniques: An Empirical Study of Predicate Representation Granularity
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the author's own terms, the central discovery is that a five-category predicate scheme—built bottom-up from observed technique executions rather than adopted from an existing nine-category model—reproduces every planning outcome of the finer model on a sixteen-technique corpus. The scheme folds nine categories into five: Environment, Executor, Process, Privilege/User, and Information. Extra granularity changes the internal structural 'why' of a plan for exactly one technique (an archival technique whose environment-level predicates split into distinct categories), and for no others. The paper is explicit that plan validity and cost cannot vary between schemes because the nine-category dom
What carries the argument
The load-bearing object is the five-category predicate scheme (an Attack Action Linking Model reduction) built bottom-up from observed executions: Environment, Executor, Process, Privilege/User, and Information. It carries the argument because the comparison with the nine-category model is a relabeling: only category labels change, never predicate names or their roles, and since the planner reasons solely over predicate structure, plan validity and cost are identical by construction. The metric that actually tests granularity is Predicate Category Resolution—the difference in how many distinct categories each technique uses under the two schemes—which shows a genuine gain for one technique o
Load-bearing premise
The paper's headline invariance is guaranteed by its construction: it compares a nine-category model that is only a relabeling of the five-category one, so equal outcomes were fixed in advance; if that relabeling is not a faithful representation of the finer model, the comparison says nothing about granularity.
What would settle it
A concrete test: build a five-category encoding that genuinely merges two causally distinct predicates—for example, treating 'has admin privilege' and 'remote token filtering disabled' as a single fact—and run the planner on the lateral-movement scenario. If the plan still solves, granularity truly does not matter for this corpus; if the plan fails or changes cost, the paper's relabeling-based comparison has missed a real granularity effect.
If this is right
- For this corpus, the reduced five-category scheme can stand in for the nine-category model without losing any plan or changing its cost; the finer model is not justified as a general default on this evidence.
- Predicate category load is driven by the operating-system mechanism a technique uses, not by its tactic label, so scheme design should be informed by mechanism rather than taxonomy.
- A controlled distinction such as holding administrator privilege versus being able to exercise it over a network logon can be load-bearing even when plan validity is unaffected, so collapsing categories carelessly can produce operationally wrong plans.
- Encoding unsupported techniques as absent actions turns the planner's 'no solution' into a truthful statement about the corpus, making failure modes explicit data.
- A pipeline that confines the language model to translation and delegates all reasoning to a deterministic planner removes hallucination risk from the planning step, at the cost of an 11% human correction rate on translated predicates.
Where Pith is reading between the lines
- Editorial inference: Because the plan-validity comparison is fixed by the relabeling construction, the paper's real claim about granularity is much narrower than the headline suggests; a genuine test would merge or split actual predicates, not just labels.
- Editorial inference: The one genuine resolution gain suggests that the fold collapsing file-like and environment-like state may be the weakest point of the five-category scheme; a targeted test with more collection and archival techniques could show whether that fold is generally load-bearing.
- Editorial inference: The 11% correction rate implies that the pipeline's reliability depends on the independent rule-based validator and manual review; scaling to hundreds of techniques would likely require automating such validation rather than relying on human inspection.
- Editorial inference: If category load tracks OS mechanism rather than tactic, a testable extension is to choose a predicate scheme per technique class at translation time, rather than using a single global scheme.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether predicate representation granularity affects automatically generated attack chains. It formalizes sixteen Atomic Red Team techniques under a reduced five-category predicate scheme, constructs a nine-category version by relabeling the same predicates, and runs Fast Downward over both. The manuscript reports identical plan validity (13/16), identical mean plan cost (1.0), and an exact fidelity match, concluding that plan validity and cost are largely insensitive to predicate granularity. It also contributes failure-mode classifications and a controlled A/B finding on the remote-admin-token-unfiltered precondition. The paper itself acknowledges in §4.4 and §4.6 that the invariance and the fidelity match are guaranteed by construction, yet the abstract and conclusion still present them as empirical findings.
Significance. The research question is important: AURORA's nine-category AALM is a design choice, and testing whether a reduced representation preserves planning capability would be valuable. The paper has several honest and useful components: the failure-mode-as-evidence framing is constructive, the remote-admin-token-unfiltered A/B test is a genuine empirical contribution, and the explicit discussion of the relabeling construction is unusually transparent. However, the central claim—that plan validity and cost are insensitive to granularity—is not supported by the experiment as designed. Because Fast Downward reasons only over predicate structure and the nine-category domain is a byte-identical relabeling of the five-category domain, the reported invariance is a logical consequence of the setup, not a measured property. The fidelity check is similarly pre-arranged. These problems are load-bearing and cannot be repaired by local revision; the experimental design itself would need to change.
major comments (5)
- [§4.4, Table 3, Abstract, Conclusion] The central empirical claim is fixed by construction. §4.4 states that domain_nine_category.pddl is a byte-identical relabeling of domain.pddl, changing only category labels. Fast Downward plans over predicate names, action preconditions, and effects; it has no notion of category. Therefore the identical 81.3% success rate and identical mean cost of 1.0 are guaranteed consequences of the relabeling, not empirical observations. The abstract and conclusion nevertheless present these numbers as evidence that granularity has little effect on plan validity and cost. This is the manuscript's headline result, and it is unsupported. A genuine granularity ablation would need to change the predicate decomposition itself (e.g., splitting or merging predicates), not merely relabel them.
- [§3.7.3, §4.6] The attack-chain fidelity check is circular. The reference chain G_ref was constructed from the Execution→Persistence→Credential Access→Lateral Movement tactic-ordering, and §4.6 admits that the four-technique corpus was selected to instantiate that same narrative. Hence J=1.0, edit distance=0, and tactic overlap=1.0 are pre-arranged by the selection, not discovered. The conclusion lists the exact match as a result, while the paper's own narrow interpretation (pipeline correctness) is the only defensible one. Demonstrating fidelity requires a ground-truth chain derived from an independent incident report or from a corpus not constructed around the target narrative.
- [§3.2, §4.1, Table 2] The five-category scheme is derived from the same data on which it is evaluated. The Privilege/User category was introduced because the starter-chain techniques failed with access-denied errors, and the corpus then includes those exact techniques. The reported 13/16 success rate and the failure-mode classifications are therefore in-sample. They can confirm internal consistency but cannot validate that the scheme generalizes to techniques not used during scheme construction. A holdout set or a pre-registered scheme is needed before the claim that the scheme is 'empirically derived' can be treated as evidence of broad viability.
- [Table 3 vs. §4.3, §4.5, §4.7, Table 4] There is an internal inconsistency over T1003.001. Table 3 counts T1003.001 as solved (✓, chain cost 4) and includes it in the 13/16 success rate. However, Table 2 and Table 4 report 0/11 successful sub-tests and classify it as a total technique failure; §4.5 says its resolution gain is an artifact of an 'lsass-memory-accessible placeholder precondition'; and §5.4 says T1003.001 was encoded as action-less. These statements cannot all be true. If T1003.001 is a total empirical failure, then counting it as solved violates the §3.7.1 requirement that a solved technique be 'empirically evidenced' and inflates the headline success rate. If it is solved only through a placeholder precondition, that placeholder is an invented entity and the chain-level ablation on it cannot support any granularity claim. The manuscript must reconcile these statements and recompute all aggregate numbers.
- [§3.7.5, Eq. (3), Figure 2] Predicate Category Resolution does not measure what the conclusion attributes to it. ΔP_i counts distinct category labels before and after a pure relabeling of the same predicates. Since predicate names and action roles are unchanged, a nonzero ΔP_i only reflects the manual assignment of existing predicates to AURORA's nine labels. The single genuine gain, T1560.001, is that one Environment predicate was assigned to the File category; this is a label-taxonomy distinction, not a planner-visible difference. The conclusion that 'granularity has almost no effect on structural resolution' overstates the evidence. The metric cannot detect whether an important causal distinction was lost unless the predicate decomposition itself is changed.
minor comments (5)
- [§1.4] Typographical corruption: 'theF ormal Methods' should read 'the Formal Methods'.
- [§5.6] The phrase 'Several limitations are to be addressed in this work' should be 'Several limitations apply to this work' or 'are acknowledged here'.
- [Table 3] The entries '✓(chain, cost 4)' in the rows for T1059.001, T1547.001, T1003.001, and T1021.002 are ambiguous: cost 4 is the chain cost, not a per-technique cost. This should be stated explicitly in the table caption.
- [General] No artifact or data-availability statement is provided. Given the dependence of the results on the exact PDDL files, the predicate_table.py cross-check, and the corrections manifest, a public artifact would be necessary for reproducibility.
- [Abstract] The phrase '81.3% identical outcomes across both schemes' is misleading because the identity is guaranteed by construction. Even with a revised experiment, the abstract should distinguish 'identical by design' from 'identical by measurement'.
Circularity Check
The headline invariance is guaranteed by the relabeling construction; the fidelity check is pre-arranged by corpus selection.
specific steps
-
self definitional
[Section 4.4 (Representation Granularity Analysis), Table 3; echoed in Abstract and Section 6 Conclusion]
"domain nine category.pddl was constructed as a byte-identical relabeling of domain.pddl under AURORA's full nine-category AALM, following the fold given in Table 1: only the category label attached to each predicate changes, not the predicate names or their role in any action's preconditions or effects. Because Fast Downward reasons exclusively over predicate structure and has no notion of category, this construction method guarantees identical plan validity and cost between the two domains by design, independent of any empirical property of the corpus."
The nine-category domain is defined as a byte-identical relabeling of the five-category domain, so the two configurations present the same predicate structure to the planner. Plan validity and cost are therefore equal by construction, not measured. The abstract's 'identical 81.3% success rate' and conclusion's 'same mean plan cost of 1.0' are logical consequences of the experimental design. Section 4.4 admits this, but the central claim in the abstract/conclusion still treats the identity as an empirical finding.
-
fitted input called prediction
[Section 4.6 (Attack-Chain Consistency Check); also used in Section 6 Conclusion]
"Because the four-technique corpus was selected to instantiate the same Execution→Persistence→Credential Access→Lateral Movement narrative used to construct the reference chain, the result primarily confirms pipeline correctness: the plan-to-technique-ID mapping preserves the intended ordering and does not introduce substitution errors. It does not demonstrate external fidelity to an independently observed real-world incident chain."
The reference 'ground truth' was constructed from the same narrative that guided corpus selection, so J=1.0 and edit distance=0 are pre-arranged up to correct mapping. The exact-match fidelity result is therefore not independent evidence for the chain; it is a check on pipeline bookkeeping. The conclusion nevertheless reports the exact match as part of the results, while the caveat appears only later.
-
other
[Section 5.6 (Limitations), plan-cost paragraph; echoed in Abstract and Conclusion]
"the plan cost metric (Section 3.7) has limited discriminatory power within this corpus: because all nine newly-solved techniques (Section 4.3) were formalised as single-action plans, their cost is necessarily 1 in every case, and only the four-step starter chain varies at all. The identical mean cost of 1.0 reported across both configurations (Section 4.4) should therefore be read as evidence that increased predicate granularity did not introduce additional planning steps or intermediate states for this corpus, rather than as a sensitive comparative measure of plan efficiency between the two s"
The 'identical mean plan cost of 1.0' is fixed before the planner runs: every independently solved technique is encoded as a single-action plan, so cost cannot vary by granularity. This is not a fitted parameter, but it is a headline quantity that is degenerate by construction. The paper acknowledges the limited discriminatory power yet still lists identical cost as evidence of granularity insensitivity in the abstract and conclusion.
full rationale
The paper's own text supplies the specific reductions. Section 4.4 states that the nine-category domain is a byte-identical relabeling of the five-category domain, so Fast Downward—which sees only predicate structure—necessarily returns identical validity and cost. The central 81.3%/cost-1.0 invariance is therefore a theorem of the experimental design, and presenting it as the headline empirical result is circular. Section 4.6 provides the second reduction: the reference chain was built from the same Execution-Persistence-Credential Access-Lateral Movement narrative used to select the corpus, so the exact fidelity match is a selection artifact rather than validation against independent ground truth. Section 5.6 adds a third: the plan-cost comparison is degenerate because every solved technique is a single-action plan, forcing cost 1.0 for all solved cases. Notably, the paper is unusually transparent: it explicitly flags each of these limitations, which is why the appropriate finding is that the headline claim is forced by construction rather than concealed. The non-headline contributions—the empirically-derived five-category scheme, the remote-admin-token-unfiltered A/B finding, and the failure-mode data—are independent and not circular, but they do not rescue the central granularity-insensitivity claim. Score 8 reflects that the central result is forced by definition while significant honest, independent content remains.
Axiom & Free-Parameter Ledger
free parameters (4)
- Five-category scheme composition =
Executor, Process, Privilege/User, Information, Environment
- arbitrary-code-executed precondition on set-registry-run-key =
predicate (boolean)
- lsass-memory-accessible placeholder precondition =
predicate (boolean)
- Ground-truth reference chain G_ref =
<T1059.001, T1547.001, T1003.001, T1021.002>
axioms (4)
- domain assumption MITRE ATT&CK and Atomic Red Team are authoritative ground truth for adversary behavior
- domain assumption Fast Downward with astar(blind()) is a correct and complete planner for these PDDL domains
- ad hoc to paper The Table 1 fold mapping is semantically lossless except where explicitly noted
- domain assumption VM execution evidence is a reliable indicator of real system state
invented entities (3)
-
remote-admin-token-unfiltered predicate
independent evidence
-
lsass-memory-accessible placeholder predicate
no independent evidence
-
arbitrary-code-executed predicate
no independent evidence
read the original abstract
Automated attack chain generation is critical for modern cybersecurity, yet manual construction fails to scale as adversary behaviors expand. While classical AI planning using PDDL offers a formal method to automate this process, it relies on the accurate translation of techniques into symbolic predicates. Current state-of-the-art systems like AURORA employ a nine-category Attack Action Linking Model (AALM), but the necessity of this specific granularity remains unvalidated. This work investigates the impact of predicate representation granularity on plan validity, cost, and fidelity. Utilizing a pipeline where a Large Language Model (LLM) performs translation and the Fast Downward engine performs deterministic reasoning, the study compares the full nine-category AALM against a reduced five-category scheme derived empirically from Atomic Red Team (ART) execution evidence. Results from a sixteen-technique corpus demonstrate that plan validity and cost are largely insensitive to granularity, with 81.3% identical outcomes across both schemes. The findings suggest that higher granularity primarily enhances the internal structural resolution of a plan's justification rather than the viability of the generated attack chain itself.
Figures
Reference graph
Works this paper leans on
-
[1]
B. Al-Sada, A. Sadighian, and G. Oligeri. Mitre att&ck: State of the art and way forward.ACM Computing Surveys, 57(1):1–37, 2024. doi: 10.1145/3687300
doi:10.1145/3687300 2024
-
[2]
Pddl—the planning domain definition language
Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. Pddl—the planning domain definition language. Technical Report CVC TR-98-003, Yale Center for Computational Vision and Control, 1998
1998
-
[3]
The fast downward planning system.Journal of Artificial Intelligence Research, 26: 191–246, 2006
Malte Helmert. The fast downward planning system.Journal of Artificial Intelligence Research, 26: 191–246, 2006. doi: 10.1613/jair.1705
-
[4]
Joseph Spracklen, Raveen Wijewickrama, A. H. M. Nazmus Sakib, Anindya Maiti, and Bimal Viswanath. We have a package for you! a comprehensive analysis of package hallucinations by code generating LLMs. In34th USENIX Security Symposium (USENIX Security 25), pages 3687–3706, Seattle, W A, USA, 2025. USENIX Association
2025
-
[5]
Lei Wang, Zhe Li, Yuxuan Jiang, Zixuan Wang, Zhi Guo, Jia Wang, Yichen Wei, Xuan Shen, Wen Ruan, and Yuxin Chen. From sands to mansions: Towards automated cyberattack emulation with classical planning and large language models.arXiv preprint arXiv:2407.16928, 2024. doi: 10.48550/ arXiv.2407.16928
-
[6]
Cynthia Phillips and Laura P. Swiler. A graph-based system for network-vulnerability analysis. In Proceedings of the 1998 Workshop on New Security Paradigms, pages 71–79. ACM, 1998
1998
-
[7]
Attack planning in the real world.arXiv preprint arXiv:1306.4044, 2013
Jorge Luis Obes, Carlos Sarraute, and Gerardo Richarte. Attack planning in the real world.arXiv preprint arXiv:1306.4044, 2013
Pith/arXiv arXiv 2013
-
[8]
Dynamic topologies for robust scale-free networks
Shishir Nagaraja and Ross Anderson. Dynamic topologies for robust scale-free networks. InBIOWIRE 2007, volume 5151 ofLNCS, pages 411–426. Springer-Verlag Berlin Heidelberg, 2008. 13
2007
-
[9]
Botyacc: Unified p2p botnet detection using behavioural analysis and graph analysis
Shishir Nagaraja. Botyacc: Unified p2p botnet detection using behavioural analysis and graph analysis. InESORICS 2014, Part II, volume 8713 ofLNCS, pages 439–456. Springer International Publishing, 2014
2014
-
[10]
Boddy, John Gohde, Thomas Haigh, and Stephen A
Mark S. Boddy, John Gohde, Thomas Haigh, and Stephen A. Harp. Course of action generation for cyber security using classical planning. InProceedings of the International Conference on Automated Planning and Scheduling (ICAPS), pages 12–21, 2005
2005
-
[11]
Concise finite-domain representations for pddl planning tasks.Artificial Intelligence, 173 (5–6):503–535, 2009
Malte Helmert. Concise finite-domain representations for pddl planning tasks.Artificial Intelligence, 173 (5–6):503–535, 2009
2009
-
[12]
Simulated penetration testing: From ”dijkstra” to ”turing test++”
J¨ org Hoffmann. Simulated penetration testing: From ”dijkstra” to ”turing test++”. InProceedings of the International Conference on Automated Planning and Scheduling (ICAPS), volume 25, pages 364–372, 2015
2015
-
[13]
´Alvaro L. R. Ferraz, Stefano Barbieri, Matheus E. de Souza, and Luiz A. P. J´ unior. The procedural semantics gap in structured cti: A measurement-driven stix analysis for apt emulation.arXiv preprint arXiv:2512.12078, 2025
Pith/arXiv arXiv 2025
-
[14]
Bo Liu, Yu Jiang, Xiaohan Zhang, Qiang Liu, Shiyu Zhang, Joydeep Biswas, and Peter Stone. Llm+p: Empowering large language models with optimal planning proficiency.arXiv preprint arXiv:2304.11477, 2023
Pith/arXiv arXiv 2023
-
[15]
Leveraging pre-trained large language models to construct and utilize world models for model-based task planning
Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. InAdvances in Neural Information Processing Systems, volume 36, pages 79081–79094, 2023
2023
-
[16]
Benchmarking considerations for agentic ai systems
Ajay Dholakia, Sachin Gopal Wani, David Ellison, Miro Hodak, Debojyoti Dutta, Shishir Nagaraja, and Raj Ranjan. Benchmarking considerations for agentic ai systems. InTPCTC 2025, volume 16261 of LNCS, pages 89–98. Springer Nature Switzerland AG, 2026
2025
-
[17]
Chained attacks on drone-based federated learning: From network disrup- tion to device impersonation
Suleiman Muhammad Sabo, Hamed Alkharsh, Peilin Li, Chuadhry Mujeeb Ahmed, Aydin Abadi, Shishir Nagaraja, and Rajiv Ranjan. Chained attacks on drone-based federated learning: From network disrup- tion to device impersonation. arXiv preprint arXiv:2607.20280v1, 2026
Pith/arXiv arXiv 2026
-
[18]
Defense Evasion
Kaspar Rosager Ludvigsen and Shishir Nagaraja. Dissecting liabilities in adversarial surgical robot failures: A national (danish) and eu law perspective.Computer Law & Security Review, 44:105656, 2022. A Atomic Red Team Technique Reference Table 5 lists the sixteen MITRE ATT&CK techniques and sub-techniques comprising the corpus, with their official names...
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.