{"id":"1e4b52c2-f5f7-4a7a-aab7-912f44039062","arxiv_id":"2505.11565","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ACSE-Eval benchmarks LLMs on threat-modeling 100 AWS architectures and finds that GPT-4.1 and Gemini 2.5 Pro lead threat identification, while all models score below 50% on exact CWE and ATT&CK classification.","lead":"This paper introduces ACSE-Eval, a dataset of 100 AWS deployment scenarios with expert-written threat models, and uses it to test six large language models on cloud threat identification. A generalist reader might care because it is an open benchmark for whether AI can spot misconfigurations and design flaws in real cloud architectures.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TFC is recall-only and the matching rule for generated threats to reference threats is unspecified; the reported 96-99% STRIDE recall may be achieved by verbose superset outputs, so the headline capability claim is not yet established.","rationale":"The reader's weakest assumption concerns the validity of the expert reference labels; my concern is one step downstream. Even if every reference threat model were correct, the recall-only TFC metric and the unspecified matching procedure leave the reported 96-99% STRIDE recall uninterpretable without a false-positive analysis. This is more specific than 'no inter-annotator agreement,' and it directly threatens the strongest claim because the TFC tables are the principal evidence for 'excel at threat identification.' I still see the paper as a useful benchmark contribution, and the concern is addressable by augmenting the released harness with precision/F1 and an over-generation analysis; hence the reader's CONDITIONAL verdict remains appropriate. The paper deserves credit for open-sourcing the evaluation code and for excluding Claude 3.5 Sonnet from evaluation, which makes this additional check straightforward to run.","tokens_in":10487,"tokens_out":5332,"duration_ms":55416,"concrete_test":"Instrument the released ACSE-Eval harness on all 100 scenarios to record per-scenario candidate-threat counts and per-candidate match status using the same TFC matcher, then recompute Tables 1-3 with precision and F1 for GPT-4.1 and Gemini 2.5 Pro in their best settings, and compare candidate counts to reference threat counts. Also run a union-baseline that emits the full ACSE threat taxonomy for every scenario. If precision at the reported recall levels is below about 60%, if candidate counts exceed 2x the reference counts on average, or if the union-baseline reaches at least 95% TFC, then the central claim that these models 'excel at threat identification' should be downgraded to 'high recall at unmeasured precision.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2, Threat Framework Coverage (TFC) is defined exactly as recall: TP/(TP+FN), and Tables 1-3 are the paper's primary evidence for 'threat identification.' No precision, F1, false-positive rate, or constraint on the number of generated threat entries is reported, and the matching procedure for deciding whether a model-generated threat is a 'true positive' is not specified. Because the task is open-ended generation, a model can raise recall by emitting many candidate threats, including duplicates and near-duplicates, and the 95-99% STRIDE recall for GPT-4.1 and Gemini 2.5 Pro is therefore not evidence of precision. The same recall-style construction underlies the CWE and AWS service coverage metrics (Tables 6-7). The paper's limitation section (4.1) asserts that subjectivity was 'minimized through structured rubrics,' but no rubric or matching algorithm is provided. Even if the expert reference threat models were perfectly valid, the reported leaderboard cannot be interpreted without measuring how many generated threats are actually spurious.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces ACSE-Eval, a benchmark of 100 AWS deployment scenarios with Diagrams-as-Code, AWS CDK implementations, and expert-authored threat models mapped to STRIDE, ATT&CK, and OWASP Top 10. The authors evaluate six LLMs under 0-shot and 3-shot prompting, with and without component relationship context, using Threat Framework Coverage (TFC), ROUGE-L, semantic similarity, CWE coverage, and AWS Service Coverage. They report that GPT-4.1 and Gemini 2.5 Pro reach 96-99% STRIDE recall in their best settings, while CWE coverage stays below 40% and ATT&CK coverage below 50%, and they conclude that frontier models are strong at threat identification but weaker at threat categorization and generalization.","tokens_in":10681,"tokens_out":5097,"duration_ms":50326,"significance":"The dataset is a potentially valuable community resource: it is externally labeled, spans 100 realistic AWS scenarios, and the evaluation excludes Claude 3.5 Sonnet, the model used in the dataset-generation agents, which mitigates generation-to-evaluation leakage. The open-source release of data, code, and standard errors is a concrete strength. However, the headline capability claim is not yet established because the primary metric, TFC, is a recall-only measure and the matching rule between generated and reference threats is not specified. Without precision information, false-positive counts, or a human-expert baseline, the reported leaderboard scores are difficult to interpret as evidence of threat-identification ability.","major_comments":[{"comment":"TFC is defined exactly as TP/(TP+FN), i.e., recall, and the task is open-ended generation. Nothing in the paper reports precision, F1, output-length constraints, duplicate counts, or false-positive rates. A model can therefore raise TFC by emitting many candidate threats, including duplicates and near-duplicates. The abstract's claim that GPT-4.1 and Gemini 2.5 Pro \"excel at threat identification\" on the basis of 96-99% STRIDE recall is not supported without this information.","section":"Section 3.2, Framework Coverage paragraph and Tables 1-3"},{"comment":"The paper never specifies how a model-generated threat is judged to be a true positive. Is it an exact STRIDE category match, a semantic similarity threshold, keyword overlap, or expert review? This matters because TFC is the foundation of Tables 1-3 and all cross-framework comparisons. Section 4.1 says subjectivity was \"minimized through structured rubrics,\" but no rubric, matching algorithm, or manual verification procedure is provided. The exact matching function and any human adjudication steps should be released.","section":"Section 3.2, TFC computation; Section 4.1, Limitations"},{"comment":"The expert-generated threat models are treated as ground truth for recall computations, but the paper reports no inter-annotator agreement for the two Security Engineers, no independent validation of the reference threat lists, and no human-expert baseline on the benchmark. If the reference labels are incomplete, biased, or non-unique, every TFC, CWE Coverage, and AWS Service Coverage score shifts and the model rankings may not generalize. Adding agreement statistics, a validation protocol, and at least one human-expert run would anchor the absolute scores.","section":"Section 2.2, Threat Models; Section 4.1, Limitations"},{"comment":"CWE Coverage and AWS Service Coverage are also presented as \"coverage\" percentages without a precise denominator definition and without false-positive counts. Table 6 shows best CWE coverage below 40%, which the paper interprets as evidence that LLMs are in early stages for vulnerability detection, but the recall-only construction makes even this qualitative conclusion fragile. Please specify the denominators (for example, the number of CWE entries in the reference threat model) and report precision for these metrics as well.","section":"Section 3.2, Tables 6-7"}],"minor_comments":[{"comment":"The sentence that Gemini 2.5 Pro demonstrates \"consistent superiority across all three security frameworks\" is not supported by the 3-shot columns of Tables 1-3, where GPT-4.1 is higher in several settings; please clarify that this claim refers to 0-shot performance only.","section":"Section 3.2, text after Table 3"},{"comment":"The table headers are inconsistent: Table 1 uses \"CDK IaC + CRC\" while Table 2 uses \"IaC IaC + CRC\" and the rest of the paper uses \"IaC + CRC\"; please unify the notation.","section":"Tables 1-2, headers"},{"comment":"The code snippets in Figure 1 appear truncated or syntactically invalid (for example, \"subnetConfiguration: [,\" is not valid TypeScript); if these are meant to illustrate real IaC, they should compile or be explicitly marked as simplified excerpts.","section":"Figure 1"},{"comment":"The random selection of 3-shot examples is described but the seeds or the exact example sets are not reported; please provide them to make the few-shot results reproducible.","section":"Section 3.1, Implementation and Tooling"},{"comment":"Minor typos and metadata issues: \"systemic assessment\" should be \"systematic assessment,\" \"corporac\" in Section 4.1 should be \"corpora,\" and reference [31] lists the malformed arXiv identifier \"22408.01605,\" which should likely be \"2408.01605.\"","section":"Abstract and References"}],"recommendation":"major_revision","confidential_remarks":"The dataset and open-source release are valuable, and the exclusion of the dataset-generation model from evaluation is a thoughtful design choice. My recommendation is driven by the missing precision and matching specification for the primary metric and by the absence of reference-label validation; both are load-bearing and both are fixable within the scope of this manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best to read this as a dataset and benchmark paper, not as a precise capability measurement. What's genuinely new: 100 AWS scenarios with paired Diagrams-as-Code, AWS CDK IaC, and expert-written STRIDE/ATT&CK/OWASP threat models, released openly, plus an evaluation harness that cost about $500 to run. That fills a real gap; prior work was CTF-style or banking-specific. The decision to exclude Claude 3.5 Sonnet (the model used by their generation agents) from evaluation is a thoughtful check on circularity.\n\nThe soft spot is the headline metric. TFC is defined as recall, TP/(TP+FN), for each framework. Tables 1-3 are recall with no precision, no F1, no constraint on number of generated threats, and no explicit matching rule for when a generated threat counts as a true positive. In an open-ended generation task, a verbose model that emits many threat candidates, including duplicates and near-misses, can rack up high recall. The reported 96-99% STRIDE recall for GPT-4.1 and Gemini 2.5 Pro is therefore not yet evidence that their threat identification is precise. The same recall structure underlies CWE and AWS service coverage. The paper says subjectivity was 'minimized through structured rubrics,' but those rubrics and the matching algorithm are not in the text.\n\nAlso missing: inter-annotator agreement on the reference threat models and a human-expert baseline. Without either, we cannot tell whether the reference labels are stable or how a good security engineer would score. These are addressable, not fatal. The qualitative findings—smaller models degrade with few-shot and CRC, Claude 3.7 has semantic sophistication but weak classification—are plausible and coherent with the tables.\n\nOne smaller quibble: 'production-grade' overstates the case; the architectures are realistic and generated, not drawn from live production systems. That is a minor wording issue.\n\nWho is this for? Researchers building or evaluating LLM security assistants. The dataset is reusable and the paper gives them a starting point, but they should treat the leaderboard as provisional. I would give it a serious referee. The right outcome is conditional acceptance with the metric and matching procedure fixed, plus a human baseline.\n\nRecommendation: engage as a venue paper; it needs revision before the numbers can be trusted, but the artifact deserves to enter the literature.","headline":"The 100-scenario cloud threat-modeling benchmark is a real contribution, but the paper's leaderboard claims rest on a recall-only metric and an underspecified matching step, so the capability numbers should not be taken at face value.","tokens_in":11222,"tokens_out":2077,"would_cite":true,"duration_ms":20708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces ACSE-Eval, a 100-scenario AWS threat-modeling dataset, and claims that GPT-4.1 and Gemini 2.5 Pro recall most STRIDE threats yet cover under half of ATT&CK and CWE categories.","keywords":["LLM evaluation","cloud security","threat modeling","STRIDE","MITRE ATT&CK","Infrastructure as Code","AWS","benchmark dataset"],"falsifier":"Take a random sample of the 100 scenarios, ask a different panel of cloud security experts to independently produce threat models without seeing the ACSE-Eval reference labels, then recompute the TFC, CWE, and ATT&CK scores against the second panel's labels. If inter-panel agreement is low, or if model coverage drops sharply when measured against the independent labels, the paper's recall claims are an artifact of one labeling perspective.","tokens_in":10323,"feed_emoji":"🛡️","tokens_out":7531,"duration_ms":66944,"temperature":0.7,"pith_summary":"This paper is trying to establish whether large language models can perform security threat modeling on realistic cloud infrastructure, and it argues they partly can. The authors built a dataset of 100 production-grade AWS deployment scenarios, each with architecture and Infrastructure-as-Code files plus expert-written threat models aligned to STRIDE, ATT&CK, and OWASP Top 10. Evaluating six LLMs, they find that GPT-4.1 and Gemini 2.5 Pro recover 96-99% of reference STRIDE threats in their best settings, while all tested models stay below 40% on CWE coverage and below 50% on ATT&CK coverage. The upshot, if the results hold, is that LLMs are useful drafting partners for cloud threat enumeration but still unreliable when precise security taxonomies are required.","feed_headline":"LLMs recall 98% of STRIDE threats but miss most CWE codes","feed_subtitle":"On a new 100-scenario AWS dataset, the best models top threat recall but stay under half on ATT&CK and CWE coverage.","key_machinery":"The load-bearing object is ACSE-Eval itself: a dataset of 100 AWS deployment scenarios, each packaged with a text architecture description, an AWS CDK Infrastructure-as-Code implementation, a Diagrams-as-Code Python specification, and an expert-generated reference threat model tagged with STRIDE, ATT&CK, and OWASP labels. The accompanying evaluation pipeline compares model outputs against those reference labels using Threat Framework Coverage (TFC), a recall percentage of reference threats captured per framework, plus ROUGE-L and semantic (cosine) similarity for linguistic closeness, and CWE and AWS Service Coverage for precision of technical classification. The reference threat model is the ground truth that every metric is computed against, so the entire argument rests on those expert labels being complete and fair.","core_discovery":"On the paper's own terms, the central discovery is a measured capability gap: current frontier LLMs can enumerate architectural threats at the level of STRIDE categories almost as completely as the expert-written reference models, but their coverage collapses when the task requires mapping to finer-grained taxonomies. In the best configuration (IaC plus Component Relationship Context), GPT-4.1 reaches 98.7% STRIDE recall with three-shot prompting and Gemini 2.5 Pro reaches 98.4% zero-shot; the same models score at most 39.1% on CWE detection and 49.9% on ATT&CK technique coverage. The paper also finds that adding relationship context helps the strongest models, that few-shot prompting helps GPT-4.1 but mildly hurts Gemini 2.5 Pro, and that Claude 3.7 Sonnet produces the most semantically similar threat descriptions even though its taxonomic accuracy lags.","pith_inferences":["My inference: because the reference labels come from a single expert workflow with no inter-annotator agreement, the reported recall numbers are best read as upper bounds; a second label set would probably shift scores and possibly model rankings.","My inference: the wide gap between STRIDE recall and CWE/ATT&CK coverage suggests the models are generating category-named threats fluently without a reliable semantic mapping to taxonomy IDs; a direct test would be to ask the model to quote the specific IaC line or resource that justifies each CWE assignment.","My inference: if the gap persists across future model generations, the next bottleneck for automated threat modeling is not threat enumeration but taxonomy grounding, and fine-tuning on CWE and ATT&CK corpora may yield larger gains than larger prompts or more context.","My inference: because the IaC files deliberately omit deployment metadata, the benchmark tests code-reading and architecture reasoning rather than live infrastructure behavior; executing the CDK in a sandbox and comparing discovered misconfigurations to the reference threats would probe whether the 96-99% figure survives contact with a real deployment."],"forward_implications":["If the results generalize, security teams can use GPT-4.1 or Gemini 2.5 Pro to draft initial STRIDE threat lists for AWS architectures, then spend human review time on validation rather than enumeration.","CWE and ATT&CK mappings produced by current LLMs are not yet trustworthy enough to automate; human analysts must verify every taxonomy assignment.","Adding Component Relationship Context (Diagrams-as-Code) raises STRIDE recall for large models but can degrade smaller models like Claude 3.5 Haiku, so context enrichment should be tailored to model size.","Prompting strategy matters: three-shot examples improve GPT-4.1 but reduce Gemini 2.5 Pro's coverage, meaning leaderboards that fix a single prompt template may be misleading.","The AWS-only scope means the benchmark measures one cloud's service vocabulary; similar evaluations on Azure or GCP would be needed before generalizing to multi-cloud."],"supporting_citations":[{"why":"Defines the STRIDE threat categories used as a coverage target in the evaluation.","marker":"[27]"},{"why":"Provides the ATT&CK technique taxonomy used to measure recall in the TFC metric.","marker":"[21]"},{"why":"Supplies the OWASP Top 10 risk list used as the third coverage framework.","marker":"[23]"},{"why":"Defines the diagram-as-code format used as Component Relationship Context alongside the IaC inputs.","marker":"[16]"},{"why":"Provides the evaluation harness used to run the LLM prompts and collect outputs.","marker":"[30]"},{"why":"Defines the longest-common-subsequence metric used for lexical similarity scoring.","marker":"[18]"},{"why":"Supplies the embedding model used to compute semantic (cosine) similarity between threat descriptions.","marker":"[25]"},{"why":"Represents the CTF-style cybersecurity benchmarks the paper distinguishes its architectural threat-modeling dataset from.","marker":"[5]"},{"why":"Prior LLM-assisted threat modeling study that motivates the need for a comprehensive evaluation like ACSE-Eval.","marker":"[9]"},{"why":"Industry-specific threat-modeling LLM work whose narrower scope the paper contrasts with its multi-domain AWS scenarios.","marker":"[32]"}],"fun_headline_variants":["LLMs nail STRIDE threat recall, flub CWE detection","Cloud threat eval: 98% STRIDE recall, <50% CWE and ATT&CK","ACSE-Eval: LLMs see architecture threats, miss taxonomies","GPT-4.1 edges Gemini on few-shot, but both miss CWE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation assumes the expert-generated threat models in ACSE-Eval are complete and correct ground truth, yet the paper reports no inter-annotator agreement, no independent second labeling, and no human-baseline comparison to show that a single expert's 'reference' is the right answer for recall.","fun_headline_variants_meta":{"raw":{"variants":["LLMs nail STRIDE threat recall, flub CWE detection","Cloud threat eval: 98% STRIDE recall, <50% CWE and ATT&CK","ACSE-Eval: LLMs see architecture threats, miss taxonomies","GPT-4.1 edges Gemini on few-shot, but both miss CWE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000897,"raw_usage":{"total_tokens":3859,"prompt_tokens":931,"completion_tokens":2928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2841}},"tokens_in":547,"tokens_out":2928,"duration_ms":22395,"temperature":1.0,"reasoning_tokens":2841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:59:02.753447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 100 scenarios, ask a different panel of cloud security experts to independently produce threat models without seeing the ACSE-Eval reference labels, then recompute the TFC, CWE, and ATT&CK scores against the second panel's labels. If inter-panel agreement is low, or if model coverage drops sharply when measured against the independent labels, the paper's recall claims are an artifact of one labeling perspective.","supporting_citations":[{"cited_title":"John Wiley & Sons (2014)","cited_arxiv_id":null,"evidence_quote":"Defines the STRIDE threat categories used as a coverage target in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ATT&CK technique taxonomy used to measure recall in the TFC metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the OWASP Top 10 risk list used as the third coverage framework."},{"cited_title":"https://diagrams.mingrammer.com/ (2020)","cited_arxiv_id":null,"evidence_quote":"Defines the diagram-as-code format used as Component Relationship Context alongside the IaC inputs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the evaluation harness used to run the LLM prompts and collect outputs."},{"cited_title":"In: Text Summarization Branches Out, pp","cited_arxiv_id":null,"evidence_quote":"Defines the longest-common-subsequence metric used for lexical similarity scoring."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior LLM-assisted threat modeling study that motivates the need for a comprehensive evaluation like ACSE-Eval."}],"review_version":1}