REVIEW 4 major objections 4 minor 32 references
ThreMoLIA: Threat Modeling of Large Language Model-Integrated Applications
T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read ThreMoLIA proposes an LLM-powered pipeline that generates and continuously updates threat models for LLM-integrated applications, grounding them in project-specific context and validating output through quality checks.
desk verdict A plausible vision paper for LLM-based threat modeling whose central quality claim is not yet supported by evidence, but which is honest about its open problems. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is retrieval-augmented generation (RAG): a pattern in which the LLM does not answer from memory alone but receives a set of relevant documents retrieved by vector similarity from a database filled by a data-aggregation component. RAG is what grounds the generated threat model in the actual architecture and in accumulated organizational knowledge such as earlier threat models. The second load-bearing component is the quality-assurance layer, which is supposed to make the nondeterministic output trustworthy: it validates the requested output format, extracts aspects of the model and compares them through metamorphic relationships—test oracles derived from expected relations between model versions—and assigns a health score that lets stakeholders decide whether to refine their prompt. Together these two pieces turn an off-the-shelf generative model into a context-aware, continuously updatable threat-modeling service.
What would settle it
A concrete falsifier would be an industrial case study where ThreMoLIA-generated threat models for a real LIA are checked against a vulnerability list obtained from an independent red-team exercise, showing the tool reliably omits a threat category such as indirect prompt-injection chains that a human analyst would have caught.
Extended reading notes
Core claim
The paper sets out the claim that LLMs can support the threat modeling of LIAs well enough to be useful in industrial practice if the LLM is grounded in project-specific context and its output is checked. Concretely, it proposes a composable tool: a data-aggregation layer collects system descriptions, requirement specs, architectural diagrams, prior threat models, and post-release monitoring data; a retrieval-augmented generation (RAG) component vectorizes these sources and injects the relevant ones into the prompt; a prompting layer provides a system prompt, user prompt, and reasoning strategy such as chain of thought; and a quality-assurance layer checks syntactic correctness, runs metamorphic tests on the extracted content, and summarizes the result in a health score. The paper reports a preliminary evaluation in which a zero-shot chatbot prompted with a simple LIA architecture consulted two established LLM threat taxonomies and produced a threat model; the authors describe the results as encouraging and frame the work as the first LLM-supported threat modeling approach for LIAs developed with an industry partner.
Load-bearing premise
The approach depends on being able to measure whether a generated threat model is complete and correct, but the paper states both that established quality metrics are lacking and that no agreed-upon benchmark exists to evaluate a threat model.
Editorial extensions
If this is right
- Threat modeling for AI-backed applications can begin at the architecture and design phase and be regenerated automatically when the architecture changes, rather than being a one-time manual exercise.
- Organizations can reuse their accumulated threat models as a knowledge base, since existing models of similar applications become retrieved context for new work.
- LLM-specific threats such as prompt injection and excessive agency would be covered systematically because the retrieval pipeline draws on LLM-specific threat taxonomies rather than only classical software threat categories.
- The effort of security experts can shift from producing threat models from scratch to reviewing and refining machine-drafted ones, reducing required participation while keeping human oversight.
- The planned industrial evaluation would produce the first benchmark for threat-model quality that is grounded in a systematic literature study and validated with practitioners.
Reading between the lines
- The paper leaves implicit but directly testable: two teams model the same LIA, one using ThreMoLIA and one using manual methods, and the resulting threat lists are compared against a ground truth assembled through red-team attack simulation. If the automated approach misses a class of attacks that the manual team consistently finds, the central claim fails.
- The paper's own admission that threat-model quality metrics do not yet exist suggests the health score may initially measure internal consistency rather than security correctness; validating the health score against real attacks is the first step that the evaluation plan should prioritize.
- The proposed RAG-plus-quality-assurance architecture could generalize beyond security, for example to keep compliance documents or safety cases in sync with a changing system, since the underlying problem of grounding a generative model in evolving project artifacts and checking its output is general.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents ThreMoLIA, a vision for an LLM-based threat modeling approach for LLM-integrated applications (LIAs). The proposed architecture combines retrieval-augmented generation (RAG) with data aggregation from design documents, architecture repositories, and existing threat models, and includes a prompting component and a quality assurance component that checks syntactic correctness, applies metamorphic-relationship-based tests, and reports a health score. The authors map MITRE ATLAS techniques to the OWASP Top 10 for LLMs, outline an evaluation plan involving an industry partner, and report a preliminary pilot in which ChatGPT-3.5 Turbo was asked to perform zero-shot threat modeling on a simple LIA. The paper states that this pilot 'encouraged us to proceed.' The central claims are that automating threat modeling with an LLM accelerates development while keeping threat models up to date, and that quality is ensured through comprehensive data checks.
Significance. If realized as described, ThreMoLIA would address a timely and practically important problem: adapting threat modeling to the dynamic, nondeterministic components introduced by LLM integration. The proposed integration of RAG, continuous threat model updates, and an explicit quality assurance component is a reasonable design direction, and the collaboration with an industry partner gives the work potential practical relevance. Table 1's mapping of MITRE ATLAS techniques to OWASP Top 10 for LLM categories is a useful synthesis. The authors are also transparent about the absence of established threat-model quality metrics. However, the manuscript is currently a vision/position paper rather than a validated tool paper; the quality assurance and evaluation components are described as future work, and the only reported evidence is a single anecdotal pilot. The claimed ability to 'ensure the accuracy and reliability' of generated threat models is not supported by the evidence presented.
major comments (4)
- [§3.1, Quality Assurance Challenges] The paper's central quality claim is undercut by its own statement that 'Currently, established metrics to systematically quantify and evaluate the quality of a threat model are lacking.' The proposed health score and the metamorphic-relationship oracles both presuppose a definition of correctness for a generated threat model. Without such metrics, the 'comprehensive data checks' described in the abstract and Section 3.1 are ungrounded. The authors should either present a concrete, validated operationalization of threat-model quality or explicitly reframe the quality-assurance claims as an open design problem rather than a delivered capability.
- [§3.2, Evaluation Plan] The evaluation plan is circular with respect to metrics. Section 3.2 proposes to create a benchmark by extracting metrics from the literature, validating them in focus groups, and then using those same metrics in the industrial comparison of ThreMoLIA against existing tools. This conflates the validation of the measurement instrument with the validation of the tool under test. The authors need an independent reference point—for example, expert judgment on a held-out set, a known planted-vulnerability corpus, or inter-rater agreement against manually produced threat models—so that the metrics themselves are not the only basis for judging whether ThreMoLIA produces high-quality outputs.
- [§4.1, Initial Prototype] The reported pilot does not exercise any of the proposed ThreMoLIA components. It is a single zero-shot ChatGPT-3.5 session on a simple LIA, with a deliberately vague prompt, no RAG, no data checks, no health score, no baseline, and no quantitative or qualitative error analysis. Consequently, it cannot support the abstract's claim that the approach 'prioritizes the quality of threat modeling reports by implementing comprehensive data checks to ensure the accuracy and reliability of the output.' The pilot may be reported as anecdotal motivational evidence, but the paper should clearly state that it provides no evidence about the effectiveness or quality assurance of ThreMoLIA itself.
- [§4.1, evaluation quality] Even as a preliminary feasibility check, the pilot lacks the measurement needed to justify the authors' conclusion that the results 'encouraged us to proceed.' There is no description of how the generated threat model was assessed, no comparison to a manually produced threat model for the same system, and no discussion of omissions or errors in the ChatGPT output. Without such an assessment, the pilot offers little more than the observation that an LLM can produce threat-model-like text. The authors should specify what would count as success in the planned evaluation and report at least a minimal check (e.g., expert rating, coverage of expected threat categories) for the pilot.
minor comments (4)
- [§2.1, STRIDE description] The text states that STRIDE 'organizes possible threats across five categories,' but the list immediately following contains six categories (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege). The count should be corrected.
- [§3.2 vs. §4.2] There is an inconsistency in the reported number of specialists: Section 3.2 mentions 'two security specialists,' while Section 4.2 says 'a focus group consisting of three security researchers.' Please align these statements.
- [§4.2, Table 2] Table 2 lists many metrics but does not indicate which are objective, measurable quantities versus subjective ratings. For example, 'Accuracy' as defined (true positives and true negatives divided by total classification outcomes) requires a ground truth, while 'Engineer Friendly' is a subjective satisfaction measure. The authors should clarify which metrics are expected to be computable without human judgment.
- [Throughout] The paper would benefit from a explicit statement distinguishing 'proposed design' from 'implemented functionality.' The abstract and introduction use language that implies existing capability (e.g., 'ensures accuracy and reliability'), but the body repeatedly describes planned components and future evaluations. A consistent tense and framing would set accurate expectations.
Circularity Check
No significant circularity: the paper is a vision paper with no derivation chain; minor self-citations are not load-bearing, and the missing-metrics issue is an acknowledged gap, not a circular reduction.
full rationale
ThreMoLIA is a vision/position paper: it describes a proposed pipeline (RAG, data aggregation, prompting, quality assurance) and an evaluation plan, and reports an early zero-shot ChatGPT pilot. There is no mathematical derivation, fitted parameter, or quantitative prediction whose output is equivalent to its input. The two self-citations ([10] MLSMM and [11] the research proposal) are used as background or as a starting point for future work; the central claim that an LLM with RAG can support threat modeling and that data checks can improve reliability is argued from the proposed architecture and the pilot, not from those citations. Section 3.1 explicitly states that established metrics for threat-model quality are lacking, and Section 3.2 plans to extract and validate metrics from the literature and focus groups before using them in the industrial comparison. That is an acknowledged methodological dependency, not a circular reduction in which an output is defined in terms of its input. The early ChatGPT pilot is a preliminary observation rather than a claimed prediction, and it is not used to validate the QA component. At most there is minor self-citation that does not carry the argument, so the circularity score is low.
Assumptions & free parameters
assumptions (5)
- domain assumption Data-flow diagrams (DFDs) are sufficient to represent LLM-integrated applications for threat modeling.
- domain assumption OWASP Top 10 for LLM and MITRE ATLAS provide sufficient coverage of LLM-specific threats for threat modeling.
- domain assumption Existing threat models, architecture repositories, and sensor data can be aggregated via RAG to provide useful context.
- ad hoc to paper LLM outputs can be made reliable enough for security-critical threat modeling through the proposed quality assurance component.
- ad hoc to paper The early ChatGPT zero-shot pilot is representative of the approach's promise.
Cite this review
Pith. "Pith review of ThreMoLIA: Threat Modeling of Large Language Model-Integrated Applications." pith.science (2026). https://pith.science/paper/URW5TZQW
@misc{pith2026250418369,
author = {Pith},
title = {Pith review of: ThreMoLIA: Threat Modeling of Large Language Model-Integrated Applications},
year = {2026},
howpublished = {\url{https://pith.science/paper/URW5TZQW}},
note = {Machine review of arXiv:2504.18369}
}
read the original abstract
Large Language Models (LLMs) are currently being integrated into industrial software applications to help users perform more complex tasks in less time. However, these LLM-Integrated Applications (LIA) expand the attack surface and introduce new kinds of threats. Threat modeling is commonly used to identify these threats and suggest mitigations. However, it is a time-consuming practice that requires the involvement of a security practitioner. Our goals are to 1) provide a method for performing threat modeling for LIAs early in their lifecycle, (2) develop a threat modeling tool that integrates existing threat models, and (3) ensure high-quality threat modeling. To achieve the goals, we work in collaboration with our industry partner. Our proposed way of performing threat modeling will benefit industry by requiring fewer security experts' participation and reducing the time spent on this activity. Our proposed tool combines LLMs and Retrieval Augmented Generation (RAG) and uses sources such as existing threat models and application architecture repositories to continuously create and update threat models. We propose to evaluate the tool offline -- i.e., using benchmarking -- and online with practitioners in the field. We conducted an early evaluation using ChatGPT on a simple LIA and obtained results that encouraged us to proceed with our research efforts.
Figures
Reference graph
Works this paper leans on
-
[1]
Ricardo Britto, Timothy Murphy, Massimo Iovene, Leif Jonsson, Melike Erol-Kantarci, and Benedek Kovács. 2023. Telecom AI Native Systems in the Age of Generative AI–An Engineering Perspective. arXiv preprint arXiv:2310.11770 (2023)
work page Pith review arXiv 2023
-
[2]
Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, Gang Wang, and Jun Xu. 2024. Neural retrievers are biased towards llm-generated content. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 526–537
work page 2024
-
[3]
Leon Derczynski, Hannah Rose Kirk, Vidhisha Balachandran, Sachin Kumar, Yulia Tsvetkov, Mark R Leiser, and Saif Mohammad. 2023. Assessing language model deployment with risk cards. arXiv preprint arXiv:2303.18190 (2023)
arXiv 2023
-
[4]
Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xiaowei Huang. 2024. Building guardrails for large language models. arXiv preprint arXiv:2402.01822 (2024)
arXiv 2024
-
[5]
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2 (2023)
arXiv 2023
-
[6]
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 (2020). 7https://chat.openai.com/share/7a624c6e-9ec2-4625-9ca3-0f4df8d222cf 7 Felix Viktor Jedrzejewski, Davide Fucci, and Oleksandr Adamov
arXiv 2020
-
[7]
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security. 79–90
work page 2023
-
[8]
Kathrin Grosse, Lukas Bieringer, Tarek R Besold, and Alexandre M Alahi. 2024. Towards more Practical Threat Models in Artificial Intelligence Security. In 33rd USENIX Security Symposium (USENIX Security 24) . 4891–4908
work page 2024
Show all 32 references
-
[9]
Daniela Haluza and David Jungwirth. 2023. Artificial intelligence and ten societal megatrends: an exploratory study using GPT-3. Systems 11, 3 (2023), 120
2023
-
[10]
Felix Jedrzejewski, Davide Fucci, and Oleksandr Adamov. 2023. MLSMM: Machine Learning Security Maturity Model.arXiv preprint arXiv:2306.16127 (2023)
2023 arXiv
-
[11]
Felix Viktor Jedrzejewski. 2024. Threat Modeling of ML-intensive Systems: Research Proposal. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI . 264–266
2024
-
[12]
Fengqing Jiang, Zhangchen Xu, Luyao Niu, Boxin Wang, Jinyuan Jia, Bo Li, and Radha Poovendran. 2023. Identifying and mitigating vulnerabilities in llm-integrated applications. arXiv preprint arXiv:2311.16153 (2023)
2023 arXiv
-
[13]
Anton Konev, Alexander Shelupanov, Mikhail Kataev, Valeriya Ageeva, and Alina Nabieva. 2022. A survey on threat-modeling techniques: protected objects and classification of threats. Symmetry 14, 3 (2022), 549
2022
-
[14]
Jiangnan Li, Yingyuan Yang, and Jinyuan Sun. 2024. Risks of Practicing Large Language Models in Smart Grid: Threat Modeling and Validation. arXiv preprint arXiv:2405.06237 (2024)
2024 arXiv
-
[15]
Manqing Mao, Paishun Ting, Yijian Xiang, Mingyang Xu, Julia Chen, and Jianzhe Lin. 2024. Multi-user chat assistant (MUCA): a framework using LLMS to facilitate group conversations. arXiv preprint arXiv:2401.04883 (2024)
2024 arXiv
-
[16]
Lara Mauri and Ernesto Damiani. 2022. Modeling threats to AI-ML systems using STRIDE. Sensors 22, 17 (2022), 6662
2022
-
[17]
Suvda Myagmar, Adam J Lee, and William Yurcik. 2005. Threat modeling as a basis for security requirements. (2005)
2005
-
[18]
Rahul Pankajakshan, Sumitra Biswal, Yuvaraj Govindarajulu, and Gilad Gressel. 2024. Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal. arXiv preprint arXiv:2403.13309 (2024)
2024 arXiv
-
[19]
Simon Schneider, Nicolas E Diaz Ferreyra, Pierre-Jean Queval, Georg Simhandl, Uwe Zdun, and Riccardo Scandariato. 2024. How Dataflow Diagrams Impact Software Security Analysis: an Empirical Experiment. arXiv preprint arXiv:2401.04446 (2024)
2024 arXiv
-
[20]
Nataliya Shevchenko, Timothy A Chick, Paige O’Riordan, Thomas Patrick Scanlon, and Carol Woody. 2018. Threat modeling: a summary of available methods. Software Engineering Institute| Carnegie Mellon University (2018)
2018
-
[21]
Adam Shostack. 2014. Threat modeling: Designing for security . John Wiley & Sons
2014
-
[22]
Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. 2021. Sponge examples: Energy-latency attacks on neural networks. In 2021 IEEE European symposium on security and privacy (EuroS&P) . IEEE, 212–231
2021
-
[23]
Stephen Burabari Tete. 2024. Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications. arXiv preprint arXiv:2406.11007 (2024)
2024 arXiv
-
[24]
Sven Türpe. 2017. The trouble with security requirements. In 2017 IEEE 25th International Requirements Engineering Conference (RE) . IEEE, 122–133
2017
-
[25]
Apurv Verma, Satyapriya Krishna, Sebastian Gehrmann, Madhavan Seshadri, Anu Pradhan, Tom Ault, Leslie Barrett, David Rabinowitz, John Doucette, and NhatHai Phan. 2024. Operationalizing a threat model for red-teaming large language models (llms). arXiv preprint arXiv:2407.14937 (2024)
2024
-
[26]
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In International Conference on Machine Learning. PMLR, 35413–35425
2023
-
[27]
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. Challenges in detoxifying language models. arXiv preprint arXiv:2109.07445 (2021)
2021 arXiv
-
[28]
Xiaoyuan Xie, Zhiyi Zhang, Tsong Yueh Chen, Yang Liu, Pak-Lok Poon, and Baowen Xu. 2020. METTLE: A metamorphic testing approach to assessing and validating unsupervised machine learning systems. IEEE Transactions on Reliability 69, 4 (2020), 1293–1322
2020
-
[29]
Wenjun Xiong and Robert Lagerström. 2019. Threat modeling–A systematic literature review. Computers & security 84 (2019), 53–69
2019
-
[30]
Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing (2024), 100211
2024
-
[31]
Koen Yskout, Thomas Heyman, Dimitri Van Landuyt, Laurens Sion, Kim Wuyts, and Wouter Joosen. 2020. Threat modeling: from infancy to maturity. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results . 9–12
2020
-
[32]
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Yan Xia, Man Lan, and Furu Wei. 2024. K-Level Reasoning with Large Language Models. arXiv preprint arXiv:2402.01521 (2024). 8 ThreMoLIA: Threat Modeling of Large Language Model-Integrated Applications Metric Name Description Adver...
2024 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.