Pith. sign in

REVIEW 3 major objections 5 minor 11 references

On Automating Security Policies with Contemporary LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that giving an LLM a few retrieved Windows API descriptions improves automated translation of attack-mitigation policies into API calls by about 22 percentage points in average F1-score across three models.

desk verdict An honest, incremental RAG-for-security-policy paper whose 22-point F1 gain is an oracle-retrieval upper bound, not a deployable system's expected gain; still worth a refereed short paper once the K selection is fixed or reframed. read the letter →

arxiv 2506.04838 v1 pith:TG4GIIBL submitted 2025-06-05 cs.CR cs.AI

Pablo Fernández Saura , K. R. Jayaram , Vatche Isahagian , Jorge Bernal Bernabé , Antonio Skarmeta This is my paper
classification cs.CRcs.AI
keywords retrieval-augmentedgenerationlargelanguagemodelsattackmitigationpoliciesWindowsAPIcallspolicyautomationcybersecurityin-contextlearningtoolcalling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that attack-mitigation policies written in natural language can be turned into executable Windows API calls automatically by an LLM pipeline, and that the translation becomes substantially more accurate when the second LLM is given a small set of retrieved API descriptions as context. The central empirical claim is that this retrieval-augmented setup beats a no-context baseline on every evaluated task, with an average F1 improvement of about 22 percentage points across Llama-3-70b, Llama-3-8b, and Mixtral-8x7b. A sympathetic reader would care because the approach points to a way of enforcing security policies faster and more consistently than manual expert configuration, and because it lets a system adopt new tools or new policies without retraining the models.

What carries the argument

The load-bearing mechanism is retrieval-augmented generation over a vector store of API documentation. Each Windows API function is embedded into a dense vector; a task from the policy is used as the query, and the system retrieves the $K$ most similar API descriptions and includes them in the prompt to the second LLM. The paper sets $K$ per task as the smallest number of retrieved documents that contains all ground-truth API calls, so the measured effect is what the second LLM can do when it is guaranteed the relevant documentation is in context. The dual-LLM split matters too: separating policy decomposition from API generation lets the first model work without retrieval while the second model focuses on mapping a concrete task to concrete calls.

What would settle it

Re-run the 14-task evaluation with a second ground truth independently annotated by different security experts and with a fixed retrieval budget instead of per-task optimal $K$; if the two ground truths disagree or the RAG advantage largely disappears under a fixed $K$, then the 22-point improvement is an artifact of the annotation or of oracle retrieval rather than a stable property of the method.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a dual-LLM architecture can close the gap between high-level mitigation policy and low-level enforcement: an unretrieved first model decomposes a policy into discrete tasks, a second model translates each task into a list of Windows API calls, and a vector database of 2,637 API specifications supplies the second model with the $K$ most similar function descriptions for each task. In experiments over six STIXv2 policies containing 14 tasks, the retrieval-augmented version outperformed the non-RAG baseline in precision, recall, and F1 for every model tested, by an average of about 22 percentage points in F1; the largest model gained 37 points, and smaller models with RAG matched or surpassed the largest model without it. The paper also reports that the non-RAG baseline still produced passable calls for some tasks, indicating that part of the gain comes from grounding the model in current documentation rather than from fixing a total inability to generate API calls.

Load-bearing premise

The manually created ground truth that maps each of the 14 tasks to its correct Windows API calls is accurate and complete, and the idea of a single correct set of calls is unambiguous; every metric and every per-task retrieval size depends on that mapping.

Editorial extensions

If this is right

  • A smaller, cheaper LLM with retrieval-augmented context can match the translation accuracy of a much larger model that is given no context, so the technique lowers the cost of deploying automated policy enforcement.
  • New mitigation tools can be added to the vector database without retraining the LLMs, and new policy documents can be fed straight into the pipeline; both the tool set and the policy set can change independently.
  • Because retrieved context is drawn from live API documentation, generated calls reflect the target environment's actual capabilities rather than the model's training-time knowledge of Windows APIs.
  • Automating the policy-to-API step shrinks the time from threat disclosure to defense, since no human expert has to manually interpret the policy and configure tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The measured 22-point gain assumes per-task optimal retrieval size; turning this into a deployable system requires a way to choose $K$ without the ground truth, so the reported number is likely an upper bound for production use rather than a guaranteed operating point.
  • Since the no-context baseline still works reasonably for some Windows tasks, the RAG advantage is probably larger for custom or proprietary APIs that are underrepresented in LLM training data; this could be tested by repeating the experiment with a private API set.
  • The same architecture should transfer to other API ecosystems, such as Linux utilities, cloud SDKs, and network appliances, because nothing in the mechanism is Windows-specific except the corpus of API documentation used for retrieval.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a framework for automating the translation of attack mitigation policies into Windows API calls, using a dual-LLM pipeline: LLM1 decomposes a policy into discrete tasks, and LLM2 translates each task into API calls with retrieval-augmented generation (RAG) providing relevant API documentation as context. The authors evaluate the framework on 14 tasks extracted from 6 MITRE STIX policies, comparing three LLMs (Llama-3-70b, Llama-3-8b, Mixtral-8x7b) in RAG and non-RAG modes, and report an average improvement of about 22 percentage points in F1-score when RAG is used. The system design is modular, with a Chroma vector store populated from 2,637 Windows API specifications.

Significance. If the quantitative claim were supported by a sound protocol, the paper would demonstrate a practical method for automating security policy enforcement, a timely and relevant problem. The architecture is sensible, and the use of publicly available data (MITRE CTI, Windows API documentation) is a strength, as is the explicit comparison of multiple LLMs. However, the experimental evaluation as designed does not support the headline claim: the RAG condition is given access to the ground-truth answers via the oracle choice of K, the dataset is extremely small, and the ground truth is unvalidated. The paper's contribution is therefore primarily a proof-of-concept with an upper-bound measurement, not a validated system.

major comments (3)
  1. [Section 4.2] The definition of K as 'the smallest number of documents recovered that includes all ground-truth API call documents' (Section 4.2) makes the RAG condition an oracle retrieval: the number of retrieved documents is chosen per task using the ground-truth answers, ensuring 100% retrieval recall by construction. The non-RAG baseline receives no such candidate set, so the reported 22-point average F1 improvement (Section 4.4) is an upper bound on the benefit of feeding the model a context that already contains the correct API descriptions, not the performance of a deployable retrieval system that must select K without knowing the target calls. This concern is independent of the accuracy of the manual ground truth; the protocol should either fix K a priori (e.g., a constant or tuned on a validation split) and report retrieval recall, or explicitly reframe the headline result as an oracle-retrieval bound.
  2. [Sections 3.1 and 3.2] The evaluation is based on 14 tasks drawn from 6 policies (Section 3.1), and the ground truth is a manual mapping created by the authors without independent verification, inter-annotator agreement, or a definition of what makes a set of API calls 'correct' (Section 3.2). Since all precision, recall, and F1 calculations (Section 3.5) depend entirely on this mapping, an incomplete or incorrect ground truth would invalidate the quantitative results. The paper should report the number of tasks per model, provide per-task results with variance, and validate the ground truth (e.g., with a second annotator or a documented procedure for resolving ambiguous API choices).
  3. [Sections 3.5 and 4.3] The methodology does not specify whether the tasks used to compute the metrics are those generated by LLM1 or the manually defined tasks from Section 3.2, nor how generated tasks are matched to the ground-truth API calls. Without an explicit alignment procedure, precision and recall are ill-defined; for example, if LLM1 splits a policy into different task granularities than the manual decomposition, comparing the resulting API calls to the manual ground truth requires a matching rule that is not described. Please clarify this matching and, if the generated tasks differ, evaluate the decomposition stage as well.
minor comments (5)
  1. [Section 2.2] The chunking parameters for CharacterTextSplitter are not given; retrieval results are sensitive to chunk size and overlap, so these should be reported.
  2. [Tables 1 and 2] The header 'Max K' is ambiguous; it should be clarified whether this is the maximum K across tasks or the specific K used for the example, and the caption should define the notation.
  3. [Section 4.3] Figure 2 is referenced but the underlying numerical results are not reported in the text; include the actual F1 values for each model and condition, along with error bars or per-task ranges.
  4. [Section 4.2] The exclusion of Llama-3-8b and Mixtral-8x7b from the decomposition LLM1 is a post-hoc model selection based on observed accuracy; this should be disclosed in the methodology, and ideally the excluded results should be reported in an appendix for transparency.
  5. [References and Section 6] Typographical errors: reference [6] contains 'F ourteenth' and Section 6 contains 'allign'; these should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

The 22-point RAG gain is measured under per-task oracle retrieval: K is chosen as the smallest value whose top-K includes all ground-truth API documents, so the comparison demonstrates an upper bound, not a deployable retrieval system.

  1. fitted input called prediction [Section 4.2, Baselines and Comparisons; Section 4.4, Analysis]
    "For each task, we computed the optimal value of K for the similarity search, defined as the smallest number of documents recovered that includes all ground-truth API call documents. This ensures that, in the RAG scenario, the second LLM has access to all the necessary API descriptions for each task. [...] Moreover, the data show that using RAG confers an average improvement of about 22 % point in F1-scores across all models."

    K is chosen per task from the ground-truth answer set: it is defined as the smallest retrieval size whose top-K documents include every correct API document. Thus the RAG prompt is guaranteed by construction to contain the correct API descriptions before LLM2 generates any output, while the non-RAG baseline is given no candidate set. The Section 4.4 result ('RAG confers an average improvement of about 22 % point') is therefore an oracle-retrieval upper bound, not a measured property of a deployable retrieval system that must choose K without knowing the gold calls. The retrieval hyperparameter is fitted to the answer key, and the F1 difference is then reported as if it validated RAG.

full rationale

The manuscript is self-contained and does not rely on load-bearing self-citations: the empirical setup uses public MITRE CTI policies, public Windows API documentation, and standard precision/recall/F1 metrics. The central circular issue is the per-task oracle choice of K in Section 4.2. The paper defines K as the smallest top-K that contains all ground-truth API call documents, so RAG's context is guaranteed to include the correct answers before generation, while the non-RAG baseline receives no such context. The 22-point F1 improvement reported in Section 4.4 is therefore an upper-bound result of a retrieval condition built from the answer key, not a fair comparison of a deployable RAG pipeline with an untuned or fixed retrieval hyperparameter. This is a partial circularity, warranting a score of 6: the F1 scores are not entirely forced, because LLM2 must still produce the correct calls from the provided documents, and the claim that RAG can help remains plausible; but the headline magnitude is constructed by the definition of K. The manual ground-truth mapping is a separate correctness and quality concern, not itself circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the author-created ground truth and on per-task K values that are fitted to that ground truth. No external benchmark is used, so the evaluation is self-contained but not independently grounded. No new entities are introduced; the framework relies on existing LLMs, embedding models, and vector stores.

free parameters (1)
  • K (number of retrieved API documents per task) = varies per task, e.g., 17 and 54 shown in Tables 1 and 2
    Chosen per task as the smallest K that includes all ground-truth API call documents (Section 4.2), so the retrieval hyperparameter is fitted to the labels it is evaluated against.
assumptions (5)
  • domain assumption The manually authored ground-truth mapping from policy tasks to Windows API calls is correct and exhaustive.
    The entire evaluation (Section 3.2) depends on this mapping; no inter-annotator agreement or external validation is provided.
  • domain assumption LLM1 (Llama-3-70B) decomposes mitigation policies into tasks with sufficient accuracy.
    Smaller LLMs were excluded due to lower decomposition accuracy (Section 4.2), but no quantitative threshold or evaluation of LLM1's own accuracy is reported.
  • domain assumption The scraped Microsoft Windows API documentation is sufficient to specify the correct API calls for the evaluated tasks.
    Section 3.3 treats the 2,637 scraped API descriptions as the complete knowledge base; the evaluation only verifies that ground-truth calls are present, not that retrieval ranks them correctly.
  • domain assumption The six selected MITRE CTI policies are representative of attack mitigation policies for Windows systems.
    The authors randomly selected 10 and excluded 4 that involved human participation (Section 3.1), a small and filtered sample.
  • standard math Precision, recall, and F1 as defined in Equations (1)-(3) are appropriate for evaluating API call generation.
    Standard definitions; the issue is not the math but the data used to compute it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Automating Security Policies with Contemporary LLMs." pith.science (2026). https://pith.science/paper/TG4GIIBL

@misc{pith2026250604838,
  author       = {Pith},
  title        = {Pith review of: On Automating Security Policies with Contemporary LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TG4GIIBL}},
  note         = {Machine review of arXiv:2506.04838}
}
read the original abstract

The complexity of modern computing environments and the growing sophistication of cyber threats necessitate a more robust, adaptive, and automated approach to security enforcement. In this paper, we present a framework leveraging large language models (LLMs) for automating attack mitigation policy compliance through an innovative combination of in-context learning and retrieval-augmented generation (RAG). We begin by describing how our system collects and manages both tool and API specifications, storing them in a vector database to enable efficient retrieval of relevant information. We then detail the architectural pipeline that first decomposes high-level mitigation policies into discrete tasks and subsequently translates each task into a set of actionable API calls. Our empirical evaluation, conducted using publicly available CTI policies in STIXv2 format and Windows API documentation, demonstrates significant improvements in precision, recall, and F1-score when employing RAG compared to a non-RAG baseline.

Figures

Figures reproduced from arXiv: 2506.04838 by the authors.

Figure 1
Figure 1. Proposed system architecture API generation process for each task. For each task, the system queries the vector database to retrieve relevant API specifications, then provides both the task description and retrieved documentation to LLM2 for API call generation. This architecture provides several key advantages. The separation of policy decomposition and API generation al￾lows each LLM to specialize in its respectiv… view at source ↗
Figure 2
Figure 2. Comparison of average F1-scores for each model with and without [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

11 extracted references · 8 canonical work pages

  1. [1]

    Cyber threat intelligence repository expressed in stix 2.0,

    MITRE, “Cyber threat intelligence repository expressed in stix 2.0,” https://github.com/mitre/cti, 2024

    work page 2024

  2. [2]

    Stix: Structured threat information expression,

    OASIS, “Stix: Structured threat information expression,” https:// oasis-open.github.io/cti-documentation/stix/intro.html, accessed: 10 March 2025

    work page 2025

  3. [3]

    (2024) Windows api index - win32 apps

    Microsoft. (2024) Windows api index - win32 apps. [On- line]. Available: https://learn.microsoft.com/en-us/windows/win32/ apiindex/windows-api-list

    work page 2024

  4. [4]

    Large language models for cyber security: A systematic literature review,

    H. Xu, S. Wang, N. Li, K. Wang, Y . Zhao, K. Chen, T. Yu, Y . Liu, and H. Wang, “Large language models for cyber security: A systematic literature review,” 2024. [Online]. Available: https://arxiv.org/abs/2405.04760

    arXiv 2024

  5. [5]

    When llms meet cybersecurity: a systematic literature review,

    J. Zhang, H. Bu, H. Wen, Y . Liu, H. Fei, R. Xi, L. Li, Y . Yang, H. Zhu, and D. Meng, “When llms meet cybersecurity: a systematic literature review,” Cybersecurity, vol. 8, 02 2025

    work page 2025

  6. [6]

    Security policy generation and verification through large language models: A proposal,

    F. Martinelli, F. Mercaldo, L. Petrillo, and A. Santone, “Security policy generation and verification through large language models: A proposal,” in Proceedings of the F ourteenth ACM Conference on Data and Application Security and Privacy , ser. CODASPY ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 143–145. [Online]. Available: h...

    work page doi:10.1145/3626232 2024

  7. [7]

    Position paper: Leverag- ing large language models for cybersecurity compliance,

    A. Salman, S. Creese, and M. Goldsmith, “Position paper: Leverag- ing large language models for cybersecurity compliance,” in 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), 2024, pp. 496–503

    work page 2024

  8. [8]

    Securebert: A domain-specific language model for cybersecurity,

    E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer, “Securebert: A domain-specific language model for cybersecurity,” in Security and Privacy in Communication Networks , F. Li, K. Liang, Z. Lin, and S. K. Katsikas, Eds. Cham: Springer Nature Switzerland, 2023, pp. 39–56

    work page 2023

Show all 11 references
  1. [9]

    Automating threat intelligence analysis with retrieval augmented generation (rag) for enhanced cybersecu- rity posture,

    J. Singh and S. Agrawal, “Automating threat intelligence analysis with retrieval augmented generation (rag) for enhanced cybersecu- rity posture,” International Journal of Science and Research (IJSR) , vol. 13, pp. 251–255, 05 2024

    2024

  2. [10]

    Threatmodeling-llm: Automating threat modeling using large language models for banking system,

    S. Yang, T. Wu, S. Liu, D. Nguyen, S. Jang, and A. Abuadbba, “Threatmodeling-llm: Automating threat modeling using large language models for banking system,” 2024. [Online]. Available: https://arxiv.org/abs/2411.17058

    2024 arXiv

  3. [11]

    Crimson: Empowering strategic reasoning in cybersecurity through large language models,

    J. Jin, B. Tang, M. Ma, X. Liu, Y . Wang, Q. Lai, J. Yang, and C. Zhou, “Crimson: Empowering strategic reasoning in cybersecurity through large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2403.00878

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.