Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces ACSE-Eval, a 100-scenario AWS threat-modeling dataset, and claims that GPT-4.1 and Gemini 2.5 Pro recall most STRIDE threats yet cover under half of ATT&CK and CWE categories.

desk verdict The 100-scenario cloud threat-modeling benchmark is a real contribution, but the paper's leaderboard claims rest on a recall-only metric and an underspecified matching step, so the capability numbers should not be taken at face value. read the letter →

arxiv 2505.11565 v2 pith:WIBIU2MR submitted 2025-05-16 cs.CR cs.AI

Sarthak Munshi , Swapnil Pathak , Sonam Ghatode , Thenuga Priyadarshini , Dhivya Chandramouleeswaran , Ashutosh Rana This is my paper · ORCID
classification cs.CRcs.AI
keywords LLMevaluationcloudsecuritythreatmodelingSTRIDEMITREATT&CKInfrastructureasCodeAWSbenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish whether large language models can perform security threat modeling on realistic cloud infrastructure, and it argues they partly can. The authors built a dataset of 100 production-grade AWS deployment scenarios, each with architecture and Infrastructure-as-Code files plus expert-written threat models aligned to STRIDE, ATT&CK, and OWASP Top 10. Evaluating six LLMs, they find that GPT-4.1 and Gemini 2.5 Pro recover 96-99% of reference STRIDE threats in their best settings, while all tested models stay below 40% on CWE coverage and below 50% on ATT&CK coverage. The upshot, if the results hold, is that LLMs are useful drafting partners for cloud threat enumeration but still unreliable when precise security taxonomies are required.

What carries the argument

The load-bearing object is ACSE-Eval itself: a dataset of 100 AWS deployment scenarios, each packaged with a text architecture description, an AWS CDK Infrastructure-as-Code implementation, a Diagrams-as-Code Python specification, and an expert-generated reference threat model tagged with STRIDE, ATT&CK, and OWASP labels. The accompanying evaluation pipeline compares model outputs against those reference labels using Threat Framework Coverage (TFC), a recall percentage of reference threats captured per framework, plus ROUGE-L and semantic (cosine) similarity for linguistic closeness, and CWE and AWS Service Coverage for precision of technical classification. The reference threat model is the ground truth that every metric is computed against, so the entire argument rests on those expert labels being complete and fair.

What would settle it

Take a random sample of the 100 scenarios, ask a different panel of cloud security experts to independently produce threat models without seeing the ACSE-Eval reference labels, then recompute the TFC, CWE, and ATT&CK scores against the second panel's labels. If inter-panel agreement is low, or if model coverage drops sharply when measured against the independent labels, the paper's recall claims are an artifact of one labeling perspective.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a measured capability gap: current frontier LLMs can enumerate architectural threats at the level of STRIDE categories almost as completely as the expert-written reference models, but their coverage collapses when the task requires mapping to finer-grained taxonomies. In the best configuration (IaC plus Component Relationship Context), GPT-4.1 reaches 98.7% STRIDE recall with three-shot prompting and Gemini 2.5 Pro reaches 98.4% zero-shot; the same models score at most 39.1% on CWE detection and 49.9% on ATT&CK technique coverage. The paper also finds that adding relationship context helps the strongest models, that few-shot prompting helps GPT-4.1 but mildly hurts Gemini 2.5 Pro, and that Claude 3.7 Sonnet produces the most semantically similar threat descriptions even though its taxonomic accuracy lags.

Load-bearing premise

The whole evaluation assumes the expert-generated threat models in ACSE-Eval are complete and correct ground truth, yet the paper reports no inter-annotator agreement, no independent second labeling, and no human-baseline comparison to show that a single expert's 'reference' is the right answer for recall.

Editorial extensions

If this is right

  • If the results generalize, security teams can use GPT-4.1 or Gemini 2.5 Pro to draft initial STRIDE threat lists for AWS architectures, then spend human review time on validation rather than enumeration.
  • CWE and ATT&CK mappings produced by current LLMs are not yet trustworthy enough to automate; human analysts must verify every taxonomy assignment.
  • Adding Component Relationship Context (Diagrams-as-Code) raises STRIDE recall for large models but can degrade smaller models like Claude 3.5 Haiku, so context enrichment should be tailored to model size.
  • Prompting strategy matters: three-shot examples improve GPT-4.1 but reduce Gemini 2.5 Pro's coverage, meaning leaderboards that fix a single prompt template may be misleading.
  • The AWS-only scope means the benchmark measures one cloud's service vocabulary; similar evaluations on Azure or GCP would be needed before generalizing to multi-cloud.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because the reference labels come from a single expert workflow with no inter-annotator agreement, the reported recall numbers are best read as upper bounds; a second label set would probably shift scores and possibly model rankings.
  • My inference: the wide gap between STRIDE recall and CWE/ATT&CK coverage suggests the models are generating category-named threats fluently without a reliable semantic mapping to taxonomy IDs; a direct test would be to ask the model to quote the specific IaC line or resource that justifies each CWE assignment.
  • My inference: if the gap persists across future model generations, the next bottleneck for automated threat modeling is not threat enumeration but taxonomy grounding, and fine-tuning on CWE and ATT&CK corpora may yield larger gains than larger prompts or more context.
  • My inference: because the IaC files deliberately omit deployment metadata, the benchmark tests code-reading and architecture reasoning rather than live infrastructure behavior; executing the CDK in a sandbox and comparing discovered misconfigurations to the reference threats would probe whether the 96-99% figure survives contact with a real deployment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces ACSE-Eval, a benchmark of 100 AWS deployment scenarios with Diagrams-as-Code, AWS CDK implementations, and expert-authored threat models mapped to STRIDE, ATT&CK, and OWASP Top 10. The authors evaluate six LLMs under 0-shot and 3-shot prompting, with and without component relationship context, using Threat Framework Coverage (TFC), ROUGE-L, semantic similarity, CWE coverage, and AWS Service Coverage. They report that GPT-4.1 and Gemini 2.5 Pro reach 96-99% STRIDE recall in their best settings, while CWE coverage stays below 40% and ATT&CK coverage below 50%, and they conclude that frontier models are strong at threat identification but weaker at threat categorization and generalization.

Significance. The dataset is a potentially valuable community resource: it is externally labeled, spans 100 realistic AWS scenarios, and the evaluation excludes Claude 3.5 Sonnet, the model used in the dataset-generation agents, which mitigates generation-to-evaluation leakage. The open-source release of data, code, and standard errors is a concrete strength. However, the headline capability claim is not yet established because the primary metric, TFC, is a recall-only measure and the matching rule between generated and reference threats is not specified. Without precision information, false-positive counts, or a human-expert baseline, the reported leaderboard scores are difficult to interpret as evidence of threat-identification ability.

major comments (4)
  1. [Section 3.2, Framework Coverage paragraph and Tables 1-3] TFC is defined exactly as TP/(TP+FN), i.e., recall, and the task is open-ended generation. Nothing in the paper reports precision, F1, output-length constraints, duplicate counts, or false-positive rates. A model can therefore raise TFC by emitting many candidate threats, including duplicates and near-duplicates. The abstract's claim that GPT-4.1 and Gemini 2.5 Pro "excel at threat identification" on the basis of 96-99% STRIDE recall is not supported without this information.
  2. [Section 3.2, TFC computation; Section 4.1, Limitations] The paper never specifies how a model-generated threat is judged to be a true positive. Is it an exact STRIDE category match, a semantic similarity threshold, keyword overlap, or expert review? This matters because TFC is the foundation of Tables 1-3 and all cross-framework comparisons. Section 4.1 says subjectivity was "minimized through structured rubrics," but no rubric, matching algorithm, or manual verification procedure is provided. The exact matching function and any human adjudication steps should be released.
  3. [Section 2.2, Threat Models; Section 4.1, Limitations] The expert-generated threat models are treated as ground truth for recall computations, but the paper reports no inter-annotator agreement for the two Security Engineers, no independent validation of the reference threat lists, and no human-expert baseline on the benchmark. If the reference labels are incomplete, biased, or non-unique, every TFC, CWE Coverage, and AWS Service Coverage score shifts and the model rankings may not generalize. Adding agreement statistics, a validation protocol, and at least one human-expert run would anchor the absolute scores.
  4. [Section 3.2, Tables 6-7] CWE Coverage and AWS Service Coverage are also presented as "coverage" percentages without a precise denominator definition and without false-positive counts. Table 6 shows best CWE coverage below 40%, which the paper interprets as evidence that LLMs are in early stages for vulnerability detection, but the recall-only construction makes even this qualitative conclusion fragile. Please specify the denominators (for example, the number of CWE entries in the reference threat model) and report precision for these metrics as well.
minor comments (5)
  1. [Section 3.2, text after Table 3] The sentence that Gemini 2.5 Pro demonstrates "consistent superiority across all three security frameworks" is not supported by the 3-shot columns of Tables 1-3, where GPT-4.1 is higher in several settings; please clarify that this claim refers to 0-shot performance only.
  2. [Tables 1-2, headers] The table headers are inconsistent: Table 1 uses "CDK IaC + CRC" while Table 2 uses "IaC IaC + CRC" and the rest of the paper uses "IaC + CRC"; please unify the notation.
  3. [Figure 1] The code snippets in Figure 1 appear truncated or syntactically invalid (for example, "subnetConfiguration: [," is not valid TypeScript); if these are meant to illustrate real IaC, they should compile or be explicitly marked as simplified excerpts.
  4. [Section 3.1, Implementation and Tooling] The random selection of 3-shot examples is described but the seeds or the exact example sets are not reported; please provide them to make the few-shot results reproducible.
  5. [Abstract and References] Minor typos and metadata issues: "systemic assessment" should be "systematic assessment," "corporac" in Section 4.1 should be "corpora," and reference [31] lists the malformed arXiv identifier "22408.01605," which should likely be "2408.01605."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark and metrics are external to the evaluated models, and the generator model is excluded from evaluation.

full rationale

ACSE-Eval's evaluation chain is self-contained relative to the claims being scored. The reference threat models come from a human-expert workflow involving two Security Engineers, and although the generation agents use Claude 3.5 Sonnet, the paper explicitly excludes that model from evaluation: 'We exclude Claude 3.5 Sonnet ... from our evaluations to prevent implicit bias due to its use with the agents.' The scores in Tables 1-7 are computed by comparing model outputs against fixed expert labels using TFC = TP/(TP+FN), ROUGE-L, semantic similarity, CWE coverage, and AWS service coverage. No parameter is fitted on the evaluated model outputs, no predicted quantity is defined in terms of the labels by construction, and no uniqueness theorem or load-bearing self-citation is invoked to force a conclusion. The recall-only TFC metric and the unspecified true-positive matching rule are genuine evaluation-validity limitations—they leave open the possibility that verbose outputs inflate recall, and the unreported inter-annotator agreement is a label-quality risk—but these concerns do not make the measured model performance equivalent to the input data by construction. Section 4.1's statement that subjectivity was 'minimized through structured rubrics' is asserted without providing the rubrics, but that is an annotation-transparency issue rather than a circular-reasoning reduction. I therefore find no circular step.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted. The central output is a dataset and evaluation methodology; the key unverified inputs are the expert labels and metric validity. No new physical or conceptual entities are postulated; ACSE-Eval is a dataset, not a theoretical entity.

assumptions (3)
  • domain assumption The expert-generated threat models in ACSE-Eval are correct and complete enough to serve as ground truth.
    Section 2.1 describes two Security Engineers producing threat models, but no inter-annotator agreement or independent validation is reported; all TFC and similarity scores inherit this assumption.
  • domain assumption Text similarity metrics, ROUGE-L and semantic cosine similarity, are valid proxies for threat-model quality.
    Section 3.2 uses these metrics to credit partial credit; no validation against expert judgment is provided.
  • domain assumption Threats can be uniquely and consistently mapped to STRIDE, ATT&CK, and OWASP categories for recall computation.
    The TFC formula assumes a deterministic reference match, but the matching procedure is not specified in Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?." pith.science (2026). https://pith.science/paper/WIBIU2MR

@misc{pith2026250511565,
  author       = {Pith},
  title        = {Pith review of: ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WIBIU2MR}},
  note         = {Machine review of arXiv:2505.11565}
}
read the original abstract

While Large Language Models have shown promise in cybersecurity applications, their effectiveness in identifying security threats within cloud deployments remains unexplored. This paper introduces AWS Cloud Security Engineering Eval, a novel dataset for evaluating LLMs cloud security threat modeling capabilities. ACSE-Eval contains 100 production grade AWS deployment scenarios, each featuring detailed architectural specifications, Infrastructure as Code implementations, documented security vulnerabilities, and associated threat modeling parameters. Our dataset enables systemic assessment of LLMs abilities to identify security risks, analyze attack vectors, and propose mitigation strategies in cloud environments. Our evaluations on ACSE-Eval demonstrate that GPT 4.1 and Gemini 2.5 Pro excel at threat identification, with Gemini 2.5 Pro performing optimally in 0-shot scenarios and GPT 4.1 showing superior results in few-shot settings. While GPT 4.1 maintains a slight overall performance advantage, Claude 3.7 Sonnet generates the most semantically sophisticated threat models but struggles with threat categorization and generalization. To promote reproducibility and advance research in automated cybersecurity threat analysis, we open-source our dataset, evaluation metrics, and methodologies.

Figures

Figures reproduced from arXiv: 2505.11565 by the authors.

Figure 1
Figure 1. Architecture and implementation of a sample cloud infrastructure sce [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Methodology used for generating the ACSE-Eval dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Category-wise breakdown of the AWS services that are part of ACSE [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of OWASP Top 10 threats in the ACSE-Eval dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Heatmaps showing the relationship between threat likelihood and CIA [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ThreatForest: Multi-Agent Attack Tree Generation with Pluggable TTP Framework Mapping

    cs.CR 2026-07 conditional novelty 6.0 of 10

    A multi-agent system turns code repositories into TTP-mapped attack trees, and an ablation shows the embedding encoder, not the multi-agent design, is the dominant accuracy bottleneck.

Reference graph

Works this paper leans on

32 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2406.07599 (2024)

    Alam, M.T., Bhusal, D., Nguyen, L., Rastogi, N.: Ctibench: A benchmark for evaluating llms in cyber threat intelligence. arXiv preprint arXiv:2406.07599 (2024)

    arXiv 2024

  2. [2]

    https://aws.amazon.com/bedrock/ (2023), accessed: 2024-10-11

    Amazon: Amazon bedrock documentation. https://aws.amazon.com/bedrock/ (2023), accessed: 2024-10-11

    work page 2023

  3. [3]

    AWS Documentation (2024)

    Amazon Web Services: Aws cloud development kit (aws cdk). AWS Documentation (2024)

    work page 2024

  4. [4]

    https://paperswithcode.com/paper/the-claude-3-model-family-opus- sonnet-haiku (2024), accessed: 2024-06-24

    Anthropic: The claude 3 model family: Opus, sonnet, haiku. https://paperswithcode.com/paper/the-claude-3-model-family-opus- sonnet-haiku (2024), accessed: 2024-06-24

    work page 2024

  5. [5]

    arXiv preprint arXiv:2404.13161 (2024)

    Bhatt, M., Chennabasappa, S., Li, Y., Nikolaidis, C., Song, D., Wan, S., Ahmad, F., Aschermann, C., Chen, Y., Kapil, D., Molnar, D., Whitman, S., Saxe, J.: Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161 (2024)

    arXiv 2024

  6. [6]

    In: Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, pp

    Chen, J., Wang, T., Li, Y., Zhang, X.: Seceval: A comprehensive security evaluation framework for large language models. In: Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, pp. 1823–1840, ACM (2024)

    work page 2024

  7. [7]

    Chen, Y., Ding, Z., Alowain, L., Chen, X., Wagner, D.: Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection (2023), URL https://arxiv.org/abs/2304.00409 ACSE-Eval: Can LLMs threat model real-world cloud infrastructure? 15

    arXiv 2023

  8. [8]

    arXiv preprint arXiv:2312.11805 (2023), URL https://arxiv.org/abs/2312.11805

    DeepMind, G.: Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023), URL https://arxiv.org/abs/2312.11805

    arXiv 2023

Show all 32 references
  1. [9]

    Elsharef, I., Zeng, Z., Gu, Z.: Facilitating threat modeling by leveraging large language models (2024), URL https://www.ndss-symposium.org/wp- content/uploads/aiscc2024-16-paper.pdf, [Accessed 16-05-2025]

    2024

  2. [10]

    Ferrag, M.A., Battah, A., Tihanyi, N., Jain, R., Maimut, D., Alwahedi, F., Lestable, T., Thandi, N.S., Mechri, A., Debbah, M., Cordeiro, L.C.: Securefalcon: Are we there yet in automated software vulnerability detection with llms? (2025), URL https://arxiv.org/abs/2307.06616

    2025 arXiv

  3. [11]

    arXiv preprint arXiv:2312.05275 (2023)

    Gupta, A., Mehta, V., Patel, N., Chandrasekaran, B.: Auspex: Automated security posture examination using large language models. arXiv preprint arXiv:2312.05275 (2023)

    2023 arXiv

  4. [12]

    In: Proceedings of the Workshop on Autonomous Cybersecurity, p

    Huang, J., Zhu, Q.: Penheal: A two-stage llm framework for automated pentesting and optimal remediation. In: Proceedings of the Workshop on Autonomous Cybersecurity, p. 11–22, AutonomousCyber ’24, Association for Computing Machinery, New York, NY, USA (2024), ISBN 9798400712296

    2024

  5. [13]

    Journal of Cybersecurity Research12(2), 78–93 (2025)

    Johnson, E., Patel, R.: Cloud security statistics 2025: The state of cloud security. Journal of Cybersecurity Research12(2), 78–93 (2025)

    2025

  6. [14]

    In: The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2024)

    Kon, P.T.J., Liu, J., Qiu, Y., Fan, W., He, T., Lin, L., Zhang, H., Park, O.M., Elengikal, G.S., Kang, Y., Chen, A., Chowdhury, M., Lee, M., Wang, X.: Iac-eval: A code generation benchmark for cloud infrastructure-as-code programs. In: The Thirty-eight Conference on Neural Inf...

    2024

  7. [15]

    arXiv preprint arXiv:2502.15797 (2025)

    Kouremetis, M., Dotter, M., Byrne, A., Martin, D., Michalak, E., Russo, G., Threet, M., Zarrella, G.: Occult: Evaluating large language models for offensive cyber operation capabilities. arXiv preprint arXiv:2502.15797 (2025)

    2025 arXiv

  8. [16]

    https://diagrams.mingrammer.com/ (2020)

    Kwon, M.: Diagrams: Diagram as code. https://diagrams.mingrammer.com/ (2020)

    2020

  9. [17]

    In: Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, p

    Li, Z., Shin, D.: Mutation-based consistency testing for evaluating the code understanding capability of llms. In: Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, p. 150–159, CAIN ’24, Association for Computing Machiner...

    2024

  10. [18]

    In: Text Summarization Branches Out, pp

    Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81, Association for Computational Linguistics, Barcelona, Spain (Jul 2004), URL https://aclanthology.org/W04-1013/

    2004

  11. [19]

    IEEE Transactions on Dependable and Secure Computing21(3), 1558–1571 (2024) 16 S

    Liu, J., Zhang, Y., Wang, H., Gao, J.: Secqa: A comprehensive question answering benchmark for cybersecurity knowledge evaluation. IEEE Transactions on Dependable and Secure Computing21(3), 1558–1571 (2024) 16 S. Munshi et al

    2024

  12. [20]

    Microsoft Security Developer Center (2009)

    Microsoft: The stride threat model. Microsoft Security Developer Center (2009)

    2009

  13. [21]

    MITRE: Att&ck matrix for enterprise v17.1 (2025), URL https://attack.mitre.org/

    2025

  14. [22]

    arXiv preprint arXiv:2303.08774 (2023), URL https://arxiv.org/abs/2303.08774

    OpenAI: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023), URL https://arxiv.org/abs/2303.08774

    2023 arXiv

  15. [23]

    OWASP: Owasp top ten (2025), URL https://owasp.org/www-project-top-ten/

    2025

  16. [24]

    arXiv preprint arXiv:2403.13793 (2024)

    Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna, V., Lindner, D., Rahtz, M., Assael, Y., Hodkinson, S., Howard, H., Lieberum, T., Kumar, R., Raad, M.A., Webson, A., Ho, L., Lin, S., Farquhar, S., Hutter, M., Deletang, G., Ruoss, A., El-Sayed, S., Brown,...

    2024 arXiv

  17. [25]

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics (11 2019), URL https://arxiv.org/abs/1908.10084

    2019 arXiv

  18. [26]

    SentinelOne Research Team: The state of cloud security 2024. Tech. rep., SentinelOne (2024)

    2024

  19. [27]

    John Wiley & Sons (2014)

    Shostack, A.: Threat modeling: Designing for security. John Wiley & Sons (2014)

    2014

  20. [28]

    https://techcrunch.com/2024/02/02/ai-pushes-quarterly-cloud- infrastructure-revenue-to-74b-globally/ (2024)

    TechCrunch: Cloud infrastructure saw its biggest revenue growth ever in q4. https://techcrunch.com/2024/02/02/ai-pushes-quarterly-cloud- infrastructure-revenue-to-74b-globally/ (2024)

    2024

  21. [29]

    arXiv preprint arXiv:2402.07688 (2024)

    Tihanyi, N., Ferrag, M.A., Jain, R., Bisztray, T., Debbah, M.: Cybermetric: A benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge. arXiv preprint arXiv:2402.07688 (2024)

    2024 arXiv

  22. [30]

    UK AI Security Institute: Inspect ai: Framework for large language model evaluations (2024), URL https://inspect.aisi.org.uk/

    2024

  23. [31]

    arXiv preprint arXiv:22408.01605 (2024)

    Wan, S., Nikolaidis, C., Song, D., Molnar, D., Crnkovich, J., Grace, J., Bhatt, M., Chennabasappa, S., Whitman, S., Ding, S., Ionescu, V., Li, Y., Saxe, J.: Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv preprin...

    2024 arXiv

  24. [32]

    arXiv preprint arXiv:2411.17058 (2024)

    Yang, S., Wu, T., Liu, S., Nguyen, D., Jang, S., Abuadbba, A.: Threatmodeling-llm: Automating threat modeling using large language models for banking system. arXiv preprint arXiv:2411.17058 (2024)

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.