{"id":"a5de4d76-559f-4026-a075-82b9f4beccdc","arxiv_id":"2506.04838","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using a vector database to retrieve relevant API documentation before code generation improves LLM translation of attack mitigation policies into Windows API calls by an average of 22 F1 points.","lead":"Researchers built a two-stage LLM pipeline that turns written attack-mitigation policies into concrete Windows API calls, using retrieval-augmented generation to supply the model with relevant API documentation. In tests on six real policies, the RAG-augmented version beat a no-context baseline by about 22 percentage points in F1 score across three LLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 22-point RAG gain is measured under oracle retrieval: K is chosen per task as the smallest value whose top-K retrieves all ground-truth API documents, so the comparison tests an upper bound, not a deployable retrieval system.","rationale":"Of the reader's concerns, the ground-truth accuracy issue is real but secondary; even a perfect ground truth would not fix the oracle-K selection. The K definition in Section 4.2 is a direct leakage of the label into the retrieval setup, so it undermines the central RAG-vs-baseline comparison more fundamentally than any metric ambiguity. The paper's contribution is otherwise reasonable: the pipeline is clearly described, the use of public STIX policies and Windows API docs is appropriate, and the authors do report non-RAG baselines rather than only RAG results. My proposed rerun with fixed K would settle whether the 22-point improvement is a property of RAG or of the experimental protocol. Because the current evidence is insufficient to support the headline claim as stated, I would keep the reader's CONDITIONAL verdict, making a fixed-K evaluation and explicit reframing conditions of acceptance.","tokens_in":7571,"tokens_out":4459,"duration_ms":55279,"concrete_test":"Re-run the entire evaluation with K fixed to 5 and to 10 for all tasks, selected before inspecting ground truth, and report retrieval recall and generation F1 for Llama-3-70b, Llama-3-8b, and Mixtral-8x7b. If the average RAG-vs-baseline F1 gap drops below 22 points or reverses, the headline claim is an artifact of ground-truth-based K selection. An additional check is to choose K via a similarity threshold on the task embedding without using ground-truth coverage and compare the resulting F1 scores.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.2 defines K as 'the smallest number of documents recovered that includes all ground-truth API call documents.' This is oracle selection: the retrieval hyperparameter is tuned per task using the answer key, and the RAG condition is therefore guaranteed to place every correct API description inside the LLM's context. The baseline receives no such candidate set. Consequently the 22-point F1 gap measures the benefit of giving the model a context that contains the ground-truth answers, not the benefit of a deployable retrieval system that must pick K without knowing the target calls. This concern is independent of whether the manual ground truth itself is accurate; even a perfect ground truth cannot justify oracle K for a claim about RAG's practical value. The authors frame Section 4.4 as validating RAG, but the protocol only demonstrates an upper bound. The paper should either fix K a priori (e.g., fixed or tuned on a validation split) and report retrieval recall, or explicitly reframe the headline as an oracle-retrieval upper bound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for automating the translation of attack mitigation policies into Windows API calls, using a dual-LLM pipeline: LLM1 decomposes a policy into discrete tasks, and LLM2 translates each task into API calls with retrieval-augmented generation (RAG) providing relevant API documentation as context. The authors evaluate the framework on 14 tasks extracted from 6 MITRE STIX policies, comparing three LLMs (Llama-3-70b, Llama-3-8b, Mixtral-8x7b) in RAG and non-RAG modes, and report an average improvement of about 22 percentage points in F1-score when RAG is used. The system design is modular, with a Chroma vector store populated from 2,637 Windows API specifications.","tokens_in":7750,"tokens_out":5240,"duration_ms":54538,"significance":"If the quantitative claim were supported by a sound protocol, the paper would demonstrate a practical method for automating security policy enforcement, a timely and relevant problem. The architecture is sensible, and the use of publicly available data (MITRE CTI, Windows API documentation) is a strength, as is the explicit comparison of multiple LLMs. However, the experimental evaluation as designed does not support the headline claim: the RAG condition is given access to the ground-truth answers via the oracle choice of K, the dataset is extremely small, and the ground truth is unvalidated. The paper's contribution is therefore primarily a proof-of-concept with an upper-bound measurement, not a validated system.","major_comments":[{"comment":"The definition of K as 'the smallest number of documents recovered that includes all ground-truth API call documents' (Section 4.2) makes the RAG condition an oracle retrieval: the number of retrieved documents is chosen per task using the ground-truth answers, ensuring 100% retrieval recall by construction. The non-RAG baseline receives no such candidate set, so the reported 22-point average F1 improvement (Section 4.4) is an upper bound on the benefit of feeding the model a context that already contains the correct API descriptions, not the performance of a deployable retrieval system that must select K without knowing the target calls. This concern is independent of the accuracy of the manual ground truth; the protocol should either fix K a priori (e.g., a constant or tuned on a validation split) and report retrieval recall, or explicitly reframe the headline result as an oracle-retrieval bound.","section":"Section 4.2"},{"comment":"The evaluation is based on 14 tasks drawn from 6 policies (Section 3.1), and the ground truth is a manual mapping created by the authors without independent verification, inter-annotator agreement, or a definition of what makes a set of API calls 'correct' (Section 3.2). Since all precision, recall, and F1 calculations (Section 3.5) depend entirely on this mapping, an incomplete or incorrect ground truth would invalidate the quantitative results. The paper should report the number of tasks per model, provide per-task results with variance, and validate the ground truth (e.g., with a second annotator or a documented procedure for resolving ambiguous API choices).","section":"Sections 3.1 and 3.2"},{"comment":"The methodology does not specify whether the tasks used to compute the metrics are those generated by LLM1 or the manually defined tasks from Section 3.2, nor how generated tasks are matched to the ground-truth API calls. Without an explicit alignment procedure, precision and recall are ill-defined; for example, if LLM1 splits a policy into different task granularities than the manual decomposition, comparing the resulting API calls to the manual ground truth requires a matching rule that is not described. Please clarify this matching and, if the generated tasks differ, evaluate the decomposition stage as well.","section":"Sections 3.5 and 4.3"}],"minor_comments":[{"comment":"The chunking parameters for CharacterTextSplitter are not given; retrieval results are sensitive to chunk size and overlap, so these should be reported.","section":"Section 2.2"},{"comment":"The header 'Max K' is ambiguous; it should be clarified whether this is the maximum K across tasks or the specific K used for the example, and the caption should define the notation.","section":"Tables 1 and 2"},{"comment":"Figure 2 is referenced but the underlying numerical results are not reported in the text; include the actual F1 values for each model and condition, along with error bars or per-task ranges.","section":"Section 4.3"},{"comment":"The exclusion of Llama-3-8b and Mixtral-8x7b from the decomposition LLM1 is a post-hoc model selection based on observed accuracy; this should be disclosed in the methodology, and ideally the excluded results should be reported in an appendix for transparency.","section":"Section 4.2"},{"comment":"Typographical errors: reference [6] contains 'F ourteenth' and Section 6 contains 'allign'; these should be corrected.","section":"References and Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem with a sensible modular architecture, but the experimental design has a fundamental flaw: the oracle selection of K in Section 4.2 makes the headline result an upper bound rather than a measure of a deployable system. The small, unaudited ground truth is another serious weakness. These issues are fixable within the manuscript's scope (e.g., by re-running with fixed K and reporting retrieval recall, expanding the dataset, and documenting the task-alignment procedure), so major revision is more appropriate than rejection. The contribution is incremental relative to existing work, but the specific end-to-end policy-to-API translation focus is a useful niche."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a transparent, useful demonstration of a dual-LLM pipeline that translates STIXv2 mitigation policies into Windows API calls, with RAG retrieval of API documentation. The new thing is the end-to-end application of known components (task decomposition, vector retrieval, tool calling) to this specific problem, with three LLMs compared. That is a legitimate, if modest, contribution.\n\nWhat it does well: the architecture is clearly described; the evaluation uses public data (MITRE CTI STIX, Microsoft's API index); the metrics are defined explicitly; the examples in Tables 1 and 2 are helpful; and the limitations section is honest about the Windows-only scope. The authors also state their K-selection rule openly, which makes the weakness easy to spot.\n\nThe soft spot is the one the stress-test note puts its finger on, and I think it lands. Section 4.2 defines K per task as the smallest number of retrieved documents that includes all ground-truth API calls. That is oracle selection: the retrieval hyperparameter is chosen using the answer key, so the RAG condition is guaranteed to place the correct API descriptions in the model's context. The baseline gets no such guarantee. The reported 22-point F1 gain therefore measures an upper bound on what RAG could do in a deployable setting, not the expected benefit of the actual retrieval system. Even a perfect ground truth would not fix this; K must be fixed a priori or tuned on a validation split, and retrieval recall reported separately. This is a load-bearing flaw for the headline claim, though not for the weaker claim that putting the right API docs in context helps.\n\nOther issues are secondary: 14 tasks is small, the ground truth is author-created with no inter-annotator check, and there is no variance or significance testing. None of these are fatal for a short paper; they reinforce the 'proof of concept' framing.\n\nMy take: the core idea is probably right, and the paper is honest enough that a referee can work with it. I would send it to peer review (a solid short-paper track), but require either a non-oracle K selection or a clear rewrite of the claim as an oracle-retrieval upper bound. I would not cite the 22-point number in my own work until that is fixed.","headline":"An honest, incremental RAG-for-security-policy paper whose 22-point F1 gain is an oracle-retrieval upper bound, not a deployable system's expected gain; still worth a refereed short paper once the K selection is fixed or reframed.","tokens_in":8301,"tokens_out":1153,"would_cite":false,"duration_ms":15459,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that giving an LLM a few retrieved Windows API descriptions improves automated translation of attack-mitigation policies into API calls by about 22 percentage points in average F1-score across three models.","keywords":["retrieval-augmented generation","large language models","attack mitigation policies","Windows API calls","policy automation","cybersecurity","in-context learning","tool calling"],"falsifier":"Re-run the 14-task evaluation with a second ground truth independently annotated by different security experts and with a fixed retrieval budget instead of per-task optimal $K$; if the two ground truths disagree or the RAG advantage largely disappears under a fixed $K$, then the 22-point improvement is an artifact of the annotation or of oracle retrieval rather than a stable property of the method.","tokens_in":7384,"feed_emoji":"🛡️","tokens_out":8417,"duration_ms":91037,"temperature":0.7,"pith_summary":"This paper tries to establish that attack-mitigation policies written in natural language can be turned into executable Windows API calls automatically by an LLM pipeline, and that the translation becomes substantially more accurate when the second LLM is given a small set of retrieved API descriptions as context. The central empirical claim is that this retrieval-augmented setup beats a no-context baseline on every evaluated task, with an average F1 improvement of about 22 percentage points across Llama-3-70b, Llama-3-8b, and Mixtral-8x7b. A sympathetic reader would care because the approach points to a way of enforcing security policies faster and more consistently than manual expert configuration, and because it lets a system adopt new tools or new policies without retraining the models.","feed_headline":"Retrieval-boosted LLMs lift policy-to-API accuracy by 22 points","feed_subtitle":"Feeding LLMs a few retrieved Windows API descriptions turns attack-mitigation policies into correct calls.","key_machinery":"The load-bearing mechanism is retrieval-augmented generation over a vector store of API documentation. Each Windows API function is embedded into a dense vector; a task from the policy is used as the query, and the system retrieves the $K$ most similar API descriptions and includes them in the prompt to the second LLM. The paper sets $K$ per task as the smallest number of retrieved documents that contains all ground-truth API calls, so the measured effect is what the second LLM can do when it is guaranteed the relevant documentation is in context. The dual-LLM split matters too: separating policy decomposition from API generation lets the first model work without retrieval while the second model focuses on mapping a concrete task to concrete calls.","core_discovery":"On its own terms, the paper claims that a dual-LLM architecture can close the gap between high-level mitigation policy and low-level enforcement: an unretrieved first model decomposes a policy into discrete tasks, a second model translates each task into a list of Windows API calls, and a vector database of 2,637 API specifications supplies the second model with the $K$ most similar function descriptions for each task. In experiments over six STIXv2 policies containing 14 tasks, the retrieval-augmented version outperformed the non-RAG baseline in precision, recall, and F1 for every model tested, by an average of about 22 percentage points in F1; the largest model gained 37 points, and smaller models with RAG matched or surpassed the largest model without it. The paper also reports that the non-RAG baseline still produced passable calls for some tasks, indicating that part of the gain comes from grounding the model in current documentation rather than from fixing a total inability to generate API calls.","pith_inferences":["The measured 22-point gain assumes per-task optimal retrieval size; turning this into a deployable system requires a way to choose $K$ without the ground truth, so the reported number is likely an upper bound for production use rather than a guaranteed operating point.","Since the no-context baseline still works reasonably for some Windows tasks, the RAG advantage is probably larger for custom or proprietary APIs that are underrepresented in LLM training data; this could be tested by repeating the experiment with a private API set.","The same architecture should transfer to other API ecosystems, such as Linux utilities, cloud SDKs, and network appliances, because nothing in the mechanism is Windows-specific except the corpus of API documentation used for retrieval."],"forward_implications":["A smaller, cheaper LLM with retrieval-augmented context can match the translation accuracy of a much larger model that is given no context, so the technique lowers the cost of deploying automated policy enforcement.","New mitigation tools can be added to the vector database without retraining the LLMs, and new policy documents can be fed straight into the pipeline; both the tool set and the policy set can change independently.","Because retrieved context is drawn from live API documentation, generated calls reflect the target environment's actual capabilities rather than the model's training-time knowledge of Windows APIs.","Automating the policy-to-API step shrinks the time from threat disclosure to defense, since no human expert has to manually interpret the policy and configure tools."],"supporting_citations":[{"why":"Supplies the public STIXv2 cyber-threat-intelligence repository from which the evaluation policies were sampled and filtered.","marker":"[1]"},{"why":"Defines the STIX language and serialization format that the input attack-mitigation policies are expressed in, so the pipeline's inputs have a fixed structure.","marker":"[2]"},{"why":"Provides the Windows API documentation used both to scrape the 2,637-specification knowledge base and to manually construct the ground-truth API calls for each task.","marker":"[3]"}],"fun_headline_variants":["RAG boosts LLM policy-to-API accuracy by 22 points","Retrieval-augmented LLMs beat non-RAG in policy-to-API precision","22-point F1 gain: LLMs with RAG map policies to API calls","With RAG, LLMs translate STIXv2 policies to Windows API calls better","LLMs with retrieval surpass baselines in security policy translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The manually created ground truth that maps each of the 14 tasks to its correct Windows API calls is accurate and complete, and the idea of a single correct set of calls is unambiguous; every metric and every per-task retrieval size depends on that mapping.","fun_headline_variants_meta":{"raw":{"variants":["RAG boosts LLM policy-to-API accuracy by 22 points","Retrieval-augmented LLMs beat non-RAG in policy-to-API precision","22-point F1 gain: LLMs with RAG map policies to API calls","With RAG, LLMs translate STIXv2 policies to Windows API calls better","LLMs with retrieval surpass baselines in security policy translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000368,"raw_usage":{"total_tokens":1947,"prompt_tokens":891,"completion_tokens":1056,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":954}},"tokens_in":507,"tokens_out":1056,"duration_ms":11291,"temperature":1.0,"reasoning_tokens":954,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:32:22.753039+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 14-task evaluation with a second ground truth independently annotated by different security experts and with a fixed retrieval budget instead of per-task optimal $K$; if the two ground truths disagree or the RAG advantage largely disappears under a fixed $K$, then the 22-point improvement is an artifact of the annotation or of oracle retrieval rather than a stable property of the method.","supporting_citations":[{"cited_title":"Cyber threat intelligence repository expressed in stix 2.0,","cited_arxiv_id":null,"evidence_quote":"Supplies the public STIXv2 cyber-threat-intelligence repository from which the evaluation policies were sampled and filtered."},{"cited_title":"Stix: Structured threat information expression,","cited_arxiv_id":null,"evidence_quote":"Defines the STIX language and serialization format that the input attack-mitigation policies are expressed in, so the pipeline's inputs have a fixed structure."},{"cited_title":"(2024) Windows api index - win32 apps","cited_arxiv_id":null,"evidence_quote":"Provides the Windows API documentation used both to scrape the 2,637-specification knowledge base and to manually construct the ground-truth API calls for each task."}],"review_version":1}