Pith. sign in

REVIEW 4 major objections 5 minor 114 references

LLM vs. SAST: A Technical Analysis on Detecting Coding Bugs of GPT4-Advanced Data Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports a controlled comparison in which GPT-4-Advanced Data Analysis correctly detected 30 of 32 curated coding vulnerabilities, while two standard SAST tools combined detected 11, a statistically significant difference by…

desk verdict The paper's central claim is probably true but the reported statistics are internally inconsistent, so it needs major revision before the numbers can be trusted. read the letter →

arxiv 2506.15212 v2 pith:JDZVID4S submitted 2025-06-18 cs.CR

Madjid G. Tehrani , Eldar Sultanow , William J. Buchanan , Mahkame Houmani , Christel H. Djaha Fodja This is my paper · ORCID
classification cs.CR
keywords largelanguagemodelsvulnerabilitydetectionGPT-4AdvancedDataAnalysisstaticapplicationsecuritytestingMcNemar'stestsecuresoftwaredevelopmentzero-dayvulnerabilitiescode
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a controlled comparison asking whether a large language model can detect software vulnerabilities as well as or better than traditional static analyzers. The authors selected 32 known coding pitfalls with sample code, ran each through GPT-4 (Advanced Data Analysis) and through two SAST tools (SonarQube and Cloud Defence), aggregated the SAST results with a logical OR, and scored each detection as binary. GPT-4 correctly flagged 30 of the 32 scenarios, while the aggregated SAST baseline flagged 11. McNemar's paired test yielded a chi-square of 20.046 with p = 0.000007562, which the authors read as a statistically significant advantage for GPT-4. The result matters because it suggests LLM-enhanced scanning could complement or replace parts of traditional SAST pipelines, with cost and speed implications for secure development.

What carries the argument

The central machinery is the paired comparison protocol: 32 curated vulnerable code snippets drawn from public examples, a binary 'correct detection' label for each tool, an OR-aggregation of the two SAST tools into a single baseline, and McNemar's test applied to the resulting 2x2 table of concordant and discordant outcomes. McNemar's test is the identity that carries the statistical claim, since it evaluates whether the proportion of GPT-4-only detections significantly exceeds the proportion of SAST-only detections.

What would settle it

Re-run the comparison on 32 newly written, non-public vulnerable snippets, asking GPT-4 to detect bugs from the code alone and scoring with a pre-registered rubric; if GPT-4 no longer outperforms the same SAST baseline by a wide margin, the reported advantage is an artifact of memorization or prompt leakage.

Watch

Extended reading notes

Core claim

The paper's central claim is that GPT-4-Advanced Data Analysis detects coding vulnerabilities more effectively than the combined output of two established SAST tools on a shared benchmark of 32 security scenarios. In the paper's own accounting, GPT-4 achieved 30 correct detections (93.75%) against 11 for the SAST baseline, and the paired McNemar test on the discordant pairs (22 GPT-4-only, 0 SAST-only) gives a chi-square of 20.046 with a p-value of 0.000007562, rejecting the null hypothesis that GPT-4 is no better than SAST.

Load-bearing premise

The entire comparison assumes that the 32 public code samples and the binary 'correct detection' labels measure vulnerability-detection skill without leakage: the snippets may already be in GPT-4's training memory, the prompts may hint at the bug class, and no rubric for labeling is documented.

Editorial extensions

If this is right

  • If the result holds, LLM-based scanning could serve as a complement to SAST in CI/CD pipelines, catching vulnerability classes that static analyzers miss.
  • Security teams may be able to reduce tooling costs if a general-purpose LLM matches or beats specialized SAST licenses on common bug classes.
  • The performance gap suggests investing in security-tuned LLMs and cleaner vulnerability datasets could yield further gains.
  • Because the model's knowledge has a training cutoff, the same detection edge may not extend to vulnerabilities discovered after that cutoff.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Given the public provenance of the snippets (GitHub and Snyk lessons), GPT-4's training data may contain these exact examples, so the 93.75% figure could overstate performance on genuinely novel code.
  • The paper does not report the exact prompts sent to GPT-4 or the labeling rubric, so a replication with masked prompts and pre-registered labels could narrow or widen the gap.
  • A natural extension would be to measure false positives and scoring confidence, since binary correct/incorrect hides how often the LLM flags benign code.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript reports a controlled comparison of GPT-4 (Advanced Data Analysis) with two SAST tools, SonarQube/SonarCloud and Cloud Defence, on 32 curated code security scenarios. Each scenario is scored binomially, the two SAST outputs are OR-aggregated into a single baseline, and McNemar's test is applied to the paired outcomes. The paper claims GPT-4 detected 30/32 scenarios (93.75%) versus 11/32 for SAST with a statistically significant difference, and discusses LLM-specific security risks and integration considerations for DevSecOps.

Significance. The question of whether an LLM can outperform traditional SAST on realistic vulnerability snippets is timely and practically relevant. The authors chose an appropriate paired statistical test, use an OR-combined SAST baseline that is favorable to the classical tools, and make their raw records available in a public repository; these are genuine strengths. The significance of the reported effect depends on the credibility of the binary labels and on the absence of prompt leakage and training-data memorization; as written, both are unresolved, so the result should be treated as a preliminary empirical claim rather than a settled comparative benchmark.

major comments (4)
  1. [Section 4, Tables 2 and 3] The headline statistics are internally inconsistent. Table 3 has a=11, b=0, c=22, d=2, which sums to 35 scenarios and implies GPT-4 was correct on 33; the abstract and Table 2 state 30 correct out of 32. OR-aggregating the SonarQube and Cloud Defence columns in Table 2 gives 9 SAST-correct scenarios (IDs 1, 2, 9, 11, 19, 25, 27, 28, 29), not 11. The reported chi-square 20.046 is exactly the continuity-corrected value for b=0, c=22, so the test was run on the inconsistent Table 3 counts. The authors must rebuild the contingency table from Table 2 and update the abstract, Section 4, and the McNemar result accordingly; the qualitative conclusion would survive correction, but the precise numbers in the abstract are currently unsupported.
  2. [Sections 3.1 and 4] The detection protocol is not reproducible. The paper does not give the exact prompts sent to GPT-4, the temperature or number of runs, the ordering of scenarios, or whether the code snippets were accompanied by their CWE names. It also does not describe how the binary 'correct detection' labels were assigned or independently verified. Since every downstream statistic depends on these labels, the authors must disclose the full protocol and an annotation rubric.
  3. [Section 3.2 and repository [22]] The 32 samples are taken from public GitHub repositories and Snyk lessons, so some snippets may be present in GPT-4's training data. The paper does not check for memorization or test variants. This limits external validity; at minimum the authors should state whether CWE identifiers or vulnerability categories were withheld from the prompts and should repeat the comparison on modified or newly generated unseen examples.
  4. [Appendix] The provided Python script cannot reproduce the result. It contains syntax errors in the statsmodels import and in the function call ('McNEmar'/'McNamara'), reads an unspecified Excel file, and initializes cells a and d to 0. A corrected, runnable script with the actual contingency data should be supplied so that the chi-square and p-value can be verified.
minor comments (5)
  1. [Section 4] The sentence 'adopting a significance level of 0.05%' should read 0.05 (5%), and the text 'using the in the appendix' is incomplete.
  2. [Section 3.1] The experimental design is described as a 'Chi-Squared Test for Independence', but the correct description is McNemar's paired test; the later text is correct.
  3. [Table 2] Several CWE mappings are questionable (e.g., Log4J mapped to CWE-502 and ReDoS to CWE-185); please verify against current CWE entries or use the canonical CWE for each vulnerability.
  4. [Section 5] Future research item 5, 'LLMs-trained using Fault-Tolerant Quantum Computers', is an incomplete dangling bullet; it should either be removed or written out as a full item.
  5. [Abstract] The phrase 'by an accuracy of 94%' is imprecise; report the raw 30/32 figure and sample size in the same sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the comparison is an empirical measurement, not a derivation; internal table inconsistencies are reproducibility errors, not circular reasoning.

full rationale

The paper's central claim (GPT-4 detects more of the 32 curated vulnerabilities than the OR-aggregated SAST baseline, with McNemar p < 0.05) is an empirical measurement. There is no fitted parameter, no self-definitional construction, and no 'prediction' that is equivalent to an input by construction. The binary labels and 32 scenarios are author-defined, which raises validity and leakage concerns (e.g., public GitHub/Snyk snippets that GPT-4 may have memorized; no prompt or rubric disclosure), but these are correctness and reproducibility risks, not circularity. The statistical inconsistency between Table 2 (30 GPT-correct, 9 SAST-correct under OR-aggregation) and Table 3 (11 SAST-correct, 22 GPT-only, 35 total scenarios) and the reported chi-square of 20.046 derived from Table 3 means the exact numbers are not reproducible from the paper's own tables; however, recomputation from Table 2 still yields a statistically significant McNemar result (b = 0, c = 21; continuity-corrected chi-square = 19.05), so the qualitative conclusion is not an artifact of the inconsistency. The only self-referential element is the GitHub data repository [22], which is evidence storage, not load-bearing. No circular step can be quoted.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper rests on domain assumptions about the benchmark's representativeness and labeling fairness; no free parameters or invented entities are involved. The main unstated premise is that the curated scenarios and binary labels yield an unbiased comparison.

assumptions (3)
  • domain assumption The 32 selected security scenarios are representative of common coding pitfalls and are appropriate for evaluating SAST tools.
    The scenarios are handpicked from GitHub and Snyk lessons; no sampling strategy or coverage criteria are defined (Section 3.2).
  • domain assumption The OR-aggregation of the two SAST tools is a fair baseline for comparison.
    Aggregating outputs with OR gives SAST the benefit of the doubt, but the two tools are not configured or versioned in the paper (Section 3.1).
  • domain assumption The binary 'correct detection' labels assigned by the authors are accurate and unbiased.
    No independent labeling, inter-rater reliability, or detailed rubric is provided (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM vs. SAST: A Technical Analysis on Detecting Coding Bugs of GPT4-Advanced Data Analysis." pith.science (2026). https://pith.science/paper/JDZVID4S

@misc{pith2026250615212,
  author       = {Pith},
  title        = {Pith review of: LLM vs. SAST: A Technical Analysis on Detecting Coding Bugs of GPT4-Advanced Data Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JDZVID4S}},
  note         = {Machine review of arXiv:2506.15212}
}
read the original abstract

Large language models (LLMs) are increasingly used for code understanding, yet their practical effectiveness for vulnerability detection relative to Static Application Security Testing (SAST) remains insufficiently quantified. We present a controlled comparative study between GPT-4 (Advanced Data Analysis) and two SAST tools (SonarQube and Cloud Defence) on 32 curated security scenarios representing common coding pitfalls. Each scenario is scored with a binary detection rule, the two SAST outputs are aggregated using a logical OR baseline, and paired outcomes are evaluated using McNemar's test for statistical significance. In our dataset, GPT-4 correctly detected 30 of 32 scenarios (93.75\%), while the aggregated SAST baseline detected 11 of 32. The paired comparison shows a statistically significant difference in detection performance in favour of GPT-4. We also discuss security considerations and operational constraints for integrating LLM-enhanced vulnerability scanning into secure software development workflows.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

114 extracted references · 63 canonical work pages

  1. [1]

    Analysis of openssl heartbleed vulnerability for embedded systems

    Imran Ghafoor, Imran Jattala, Shakeel Durrani, and Ch Muhammad Tahir. Analysis of openssl heartbleed vulnerability for embedded systems. In 17th IEEE International Multi Topic Conference 2014, pages 314–319. IEEE, 2014

    2014

  2. [2]

    Challenges to cybersecurity: Current state of affairs

    Ravi Sen. Challenges to cybersecurity: Current state of affairs. Communications of the Association for Information Systems , 43(1):2, 2018

    2018

  3. [3]

    A survey on machine learning techniques for source code analysis

    Tushar Sharma, Maria Kechagia, Stefanos Georgiou, Rohit Tiwari, Indira Vats, Hadi Moazen, and Federica Sarro. A survey on machine learning techniques for source code analysis. arXiv preprint arXiv:2110.09610 , 2021

    arXiv 2021

  4. [4]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774

    arXiv 2023

  5. [5]

    Firepatch: Secure and time-critical dissemination of software patches

    H˚ avard D Johansen and Robbert van Renesse. Firepatch: Secure and time-critical dissemination of software patches. IFIP, pages 373–384, 01 2007. https://doi.org/ 10.1007/978-0-387-72367-9_32 . URL https://link.springer.com/chapter/10 .1007/978-0-387-72367-9_32

    work page doi:10.1007/978-0-387-72367-9_32 2007

  6. [6]

    URL https://www.gov.cn/gongbao/content/2021/content_5641351.htm

    Regulations on the management of network product security vulnerabilities, 2021. URL https://www.gov.cn/gongbao/content/2021/content_5641351.htm

    2021

  7. [7]

    URL https://www.tianfucu p.com/2022/en/

    Tianfu cup international cybersecurity contest, 2022. URL https://www.tianfucu p.com/2022/en/

    2022

  8. [8]

    Artificial intelligence cyber challenge (aixcc), 2023

    DARPA. Artificial intelligence cyber challenge (aixcc), 2023. URL https://www.do dsbirsttr.mil/topics-app/?baa=DOD_SBIR_2023_P1_C4

    2023

Show all 114 references
  1. [9]

    Ai security concerns in a nutshell, 03 2023

    BSI. Ai security concerns in a nutshell, 03 2023. URL https://www.bsi.bund.de/ SharedDocs/Downloads/EN/BSI/KI/Practical_Al-Security_Guide_2023.pdf?_ _blob=publicationFile&v=5

    2023

  2. [10]

    Machine learning in the context of static application security testing - ml-sast, 02 2023

    BSI. Machine learning in the context of static application security testing - ml-sast, 02 2023. URL https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Public ations/Studies/ML-SAST/ML-SAST-Studie-final.pdf?__blob=publicationFil e&v=5

    2023

  3. [11]

    An analysis of the automatic bug fixing performance of chatgpt, 01 2023

    Dominik Sobania, Carol Hanna, Martin Briesch, and Justyna Petke. An analysis of the automatic bug fixing performance of chatgpt, 01 2023. URL https://arxiv.or g/pdf/2301.08653.pdf

    2023 arXiv

  4. [12]

    The scope of chatgpt in software engineering: A thorough investigation, 05 2023

    Wei Ma, Shangqing Liu, Wenhan Wang, Qiang Hu, Ye Liu, Cen Zhang, Liming Nie, and Yang Liu. The scope of chatgpt in software engineering: A thorough investigation, 05 2023. URL https://arxiv.org/pdf/2305.12138.pdf

    2023 arXiv

  5. [13]

    The hitchhiker’s guide to program analysis: A journey with large language models, 08 2023

    Haonan Li, Yu Hao, Yizhuo Zhai, and Zhiyun Qian. The hitchhiker’s guide to program analysis: A journey with large language models, 08 2023. URL https://arxiv.org/ pdf/2308.00245.pdf

    2023 arXiv

  6. [14]

    The formai dataset: Generative ai in software security through the lens of formal verification *, 07 2023

    Norbert Tihanyi, Tamas Bisztray, Ridhi Jain, Mohamed Ferrag, Lucas Cordeiro, and Vasileios Mavroeidis. The formai dataset: Generative ai in software security through the lens of formal verification *, 07 2023. URL https://arxiv.org/pdf/2307.021 92.pdf

    2023

  7. [15]

    Evaluating large language models trained on code, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code, 2021

    2021

  8. [16]

    Technical report: Evalua- tion of chatgpt model for vulnerability detection, 04 2023

    Anton Cheshkov, Pavel Zadorozhny, and Rodion Levichev. Technical report: Evalua- tion of chatgpt model for vulnerability detection, 04 2023. URL https://arxiv.or g/pdf/2304.07232.pdf

    2023 arXiv

  9. [17]

    Not the end of story: An evaluation of chatgpt-driven vulnerability description mappings, 2023

    Xin Liu, Yuan Tan, Zhenghang Xiao, Jianwei Zhuge, and Rui Zhou. Not the end of story: An evaluation of chatgpt-driven vulnerability description mappings, 2023. URL https://aclanthology.org/2023.findings-acl.229.pdf

    2023

  10. [18]

    URL https://owasp.org/www-project-top-10-for-large-language-model-applica tions/

    Owasp top 10 for large language model applications — owasp foundation, 2023. URL https://owasp.org/www-project-top-10-for-large-language-model-applica tions/

    2023

  11. [19]

    URL https://www.sonarsource.com/products/sonarcloud/

    Sonarcloud, 2023. URL https://www.sonarsource.com/products/sonarcloud/

    2023

  12. [20]

    URL https://www.clouddefense.ai/

    Clouddefense.ai, 2023. URL https://www.clouddefense.ai/

    2023

  13. [21]

    Jailbreaking gpt-4’s code interpreter, 07 2023

    nikola. Jailbreaking gpt-4’s code interpreter, 07 2023. URL https://www.lesswron g.com/posts/KSroBnxCHodGmPPJ8/jailbreaking-gpt-4-s-code-interpreter

    2023

  14. [22]

    Tehrani, Eldar Sultanow, William J Buchanan, Houmani Mahkameh, and H

    Madjid G. Tehrani, Eldar Sultanow, William J Buchanan, Houmani Mahkameh, and H. Djaha Fodja Christel. Source code, gpt results: Gptvssast. https://github.com /Sultanow/vulnerability-detector-llm, 2023

    2023

  15. [23]

    Note on the sampling error of the difference between correlated proportions or percentages

    Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12:153–157, 06 1947. https://doi.org/ 10.1007/bf02295996. URL https://link.springer.com/article/10.1007/BF02 295996

    1947 doi

  16. [24]

    URL https://github.com/pikulet/mem-attacks-example /blob/master/buffer_overflow/buffer_overflow.c

    Buffer overflow, 2023. URL https://github.com/pikulet/mem-attacks-example /blob/master/buffer_overflow/buffer_overflow.c

    2023

  17. [25]

    URL https://github.com/doublehops/sql-injecti on-attack-example

    Sql injection attack, 04 2019. URL https://github.com/doublehops/sql-injecti on-attack-example

    2019

  18. [26]

    URL https://learn.snyk.io/lesson/sql-inj ection/

    What is sql injection (sqli)?, 2021. URL https://learn.snyk.io/lesson/sql-inj ection/

    2021

  19. [27]

    URL https://learn.snyk.io/lesson/xss /

    What is cross-site scripting (xss)?, 2021. URL https://learn.snyk.io/lesson/xss /

    2021

  20. [28]

    URL https://learn.snyk.io/lesson/broken-acces s-control/

    Broken access control, 2022. URL https://learn.snyk.io/lesson/broken-acces s-control/

    2022

  21. [29]

    URL https://learn.snyk.io/lesson/insecure-d eserialization/

    Insecure deserialization, 2022. URL https://learn.snyk.io/lesson/insecure-d eserialization/

    2022

  22. [30]

    URL https://www.lunasec.io/docs/blog/log4j-zero-day/

    Log4shell, 12 2021. URL https://www.lunasec.io/docs/blog/log4j-zero-day/

    2021

  23. [31]

    URL https://learn.snyk.io/lesson/unr estricted-file-upload/

    What is unrestricted file upload?, 2023. URL https://learn.snyk.io/lesson/unr estricted-file-upload/

    2023

  24. [32]

    URL https://learn.snyk.io/lesson/i mproper-input-validation/

    What is improper input validation?, 2023. URL https://learn.snyk.io/lesson/i mproper-input-validation/

    2023

  25. [33]

    URL https://learn.snyk.io/lesson/memory-lea ks/

    What are memory leaks?, 2023. URL https://learn.snyk.io/lesson/memory-lea ks/

    2023

  26. [34]

    URL https://learn.snyk.io/lesson/mass-ass ignment/

    What is mass assignment?, 2023. URL https://learn.snyk.io/lesson/mass-ass ignment/

    2023

  27. [35]

    URL https://learn.snyk.io/lesson/ssrf-server-side-r equest-forgery/

    What is ssrf?, 2023. URL https://learn.snyk.io/lesson/ssrf-server-side-r equest-forgery/

    2023

  28. [36]

    URL https://learn.snyk.io/lesson/i nsecure-temporary-file/

    What is an insecure temporary file?, 2023. URL https://learn.snyk.io/lesson/i nsecure-temporary-file/

    2023

  29. [37]

    URL https: //learn.snyk.io/lesson/cleartext-sensitive-information-in-cookie/

    The dangers of storing cleartext sensitive information in a cookie?, 2023. URL https: //learn.snyk.io/lesson/cleartext-sensitive-information-in-cookie/

    2023

  30. [38]

    URL https://learn.snyk.io/lesson/xpath-i njection/

    What is an xpath injection?, 2023. URL https://learn.snyk.io/lesson/xpath-i njection/

    2023

  31. [39]

    URL https://learn.snyk.io/lesson/wea k-password-recovery/

    What is weak password recovery?, 2023. URL https://learn.snyk.io/lesson/wea k-password-recovery/

    2023

  32. [40]

    URL https://learn.snyk.io/lesson/log ging-vulnerabilities/

    What are logging vulnerabilities?, 2023. URL https://learn.snyk.io/lesson/log ging-vulnerabilities/

    2023

  33. [41]

    URL https://learn.snyk.io/lesson/insec ure-randomness/

    What is insecure randomness?, 2023. URL https://learn.snyk.io/lesson/insec ure-randomness/

    2023

  34. [42]

    URL https://learn.snyk.io/lesson/nosql-injec tion-attack/

    Nosql injection attack, 2022. URL https://learn.snyk.io/lesson/nosql-injec tion-attack/

    2022

  35. [43]

    URL https://learn.snyk.io/lesson/malicious-c ode-injection/

    What is code injection?, 2022. URL https://learn.snyk.io/lesson/malicious-c ode-injection/

    2022

  36. [44]

    URL https://learn.snyk.io/lesson/no-rate-limiting/

    No rate limiting, 2022. URL https://learn.snyk.io/lesson/no-rate-limiting/

    2022

  37. [45]

    URL https://learn

    How to manage vulnerable and outdated components, 2022. URL https://learn. snyk.io/lesson/vulnerable-and-outdated-components/

    2022

  38. [46]

    URL https://learn.snyk.io/lesson/insecure-design/

    Insecure design, 2022. URL https://learn.snyk.io/lesson/insecure-design/

    2022

  39. [47]

    URL https://learn.snyk.io/lesson/insecur e-hash/

    What is an insecure hash?, 2022. URL https://learn.snyk.io/lesson/insecur e-hash/

    2022

  40. [48]

    URL https://learn.snyk.io/lesson/redos/

    Redos, 2022. URL https://learn.snyk.io/lesson/redos/

    2022

  41. [49]

    URL https://learn.snyk.io/lesson/xxe/

    Xxe attack, 2022. URL https://learn.snyk.io/lesson/xxe/

    2022

  42. [50]

    URL https://learn.snyk.io/lesson/csrf-attack/

    Csrf attack, 2022. URL https://learn.snyk.io/lesson/csrf-attack/

    2022

  43. [51]

    URL https://learn.snyk.io/lesson/dom-based-xss/

    Dom based xss, 2022. URL https://learn.snyk.io/lesson/dom-based-xss/

    2022

  44. [52]

    URL https://learn.snyk.io/lesson/open-red irect/

    Open redirect vulnerability, 2022. URL https://learn.snyk.io/lesson/open-red irect/

    2022

  45. [53]

    URL https://learn.snyk.io/lesson/directo ry-traversal/

    What is directory traversal?, 2021. URL https://learn.snyk.io/lesson/directo ry-traversal/

    2021

  46. [54]

    URL https://learn.snyk.io/lesson/proto type-pollution/

    What is prototype pollution?, 2021. URL https://learn.snyk.io/lesson/proto type-pollution/

    2021

  47. [55]

    URL https://learn.snyk.i o/lesson/container-does-not-drop-all-default-capabilities/

    Container does not drop all default capabilities, 2021. URL https://learn.snyk.i o/lesson/container-does-not-drop-all-default-capabilities/

    2021

  48. [56]

    URL https://learn.snyk.io/lesson/c ontainer-runs-in-privileged-mode/

    Container runs in privileged mode, 2021. URL https://learn.snyk.io/lesson/c ontainer-runs-in-privileged-mode/

    2021

  49. [57]

    Cwe-121: Stack-based buffer overflow, 2023

    MITRE Corporation. Cwe-121: Stack-based buffer overflow, 2023. URL https: //cwe.mitre.org/data/definitions/121.html. Accessed: 21-08-2023

    2023

  50. [58]

    Cwe-564: Sql injection, 2023

    MITRE Corporation. Cwe-564: Sql injection, 2023. URL https://cwe.mitre.org/ data/definitions/564.html. Accessed: 21-08-2023

    2023

  51. [59]

    Cwe-79: Improper neutralization of input during web page generation (’cross-site scripting’), 2023

    MITRE Corporation. Cwe-79: Improper neutralization of input during web page generation (’cross-site scripting’), 2023. URL https://cwe.mitre.org/data/defin itions/79.html. Accessed: 21-08-2023

    2023

  52. [60]

    Cwe-284: Improper access control, 2023

    MITRE Corporation. Cwe-284: Improper access control, 2023. URL https://cwe. mitre.org/data/definitions/284.html. Accessed: 21-08-2023

    2023

  53. [61]

    Cwe-502: Deserialization of untrusted data, 2023

    MITRE Corporation. Cwe-502: Deserialization of untrusted data, 2023. URL https: //cwe.mitre.org/data/definitions/502.html. Accessed: 21-08-2023

    2023

  54. [62]

    Cwe-434: Unrestricted upload of file with dangerous type, 2023

    MITRE Corporation. Cwe-434: Unrestricted upload of file with dangerous type, 2023. URL https://cwe.mitre.org/data/definitions/434.html. Accessed: 21-08-2023

    2023

  55. [63]

    Cwe-20: Improper input validation, 2023

    MITRE Corporation. Cwe-20: Improper input validation, 2023. URL https://cwe. mitre.org/data/definitions/20.html. Accessed: 21-08-2023

    2023

  56. [64]

    Cwe-401: Missing release of memory after effective lifetime,

    MITRE Corporation. Cwe-401: Missing release of memory after effective lifetime,

  57. [65]

    Cwe-915: Improperly controlled modification of dynamically- determined object attributes, 2023

    MITRE Corporation. Cwe-915: Improperly controlled modification of dynamically- determined object attributes, 2023. URL https://cwe.mitre.org/data/definitio ns/915.html. Accessed: 21-08-2023

    2023

  58. [66]

    Cwe-918: Server-side request forgery (ssrf), 2023

    MITRE Corporation. Cwe-918: Server-side request forgery (ssrf), 2023. URL https: //cwe.mitre.org/data/definitions/918.html. Accessed: 21-08-2023

    2023

  59. [67]

    Cwe-377: Insecure temporary file, 2023

    MITRE Corporation. Cwe-377: Insecure temporary file, 2023. URL https://cwe.mi tre.org/data/definitions/377.html. Accessed: 21-08-2023

    2023

  60. [68]

    Cwe-315: Cleartext storage of sensitive information in a cookie,

    MITRE Corporation. Cwe-315: Cleartext storage of sensitive information in a cookie,

  61. [69]

    Cwe-643: Improper neutralization of data within xpath expres- sions (’xpath injection’), 2023

    MITRE Corporation. Cwe-643: Improper neutralization of data within xpath expres- sions (’xpath injection’), 2023. URL https://cwe.mitre.org/data/definitions/6 43.html. Accessed: 21-08-2023

    2023

  62. [70]

    Accessed: 21-08-2023

    URL https://cwe.mitre.org/data/definitions/315.html . Accessed: 21-08-2023

    2023

  63. [71]

    Cwe-532: Insertion of sensitive information into log file, 2023

    MITRE Corporation. Cwe-532: Insertion of sensitive information into log file, 2023. URL https://cwe.mitre.org/data/definitions/532.html. Accessed: 21-08-2023

    2023

  64. [72]

    Cwe-640: Weak password recovery mechanism for forgotten password, 2023

    MITRE Corporation. Cwe-640: Weak password recovery mechanism for forgotten password, 2023. URL https://cwe.mitre.org/data/definitions/640.html . Accessed: 21-08-2023

    2023

  65. [73]

    Cwe-89: Improper neutralization of special elements used in an sql command (’sql injection’), 2023

    MITRE Corporation. Cwe-89: Improper neutralization of special elements used in an sql command (’sql injection’), 2023. URL https://cwe.mitre.org/data/definitio ns/89.html. Accessed: 21-08-2023

    2023

  66. [74]

    Cwe-330: Use of insufficiently random values, 2023

    MITRE Corporation. Cwe-330: Use of insufficiently random values, 2023. URL https: //cwe.mitre.org/data/definitions/330.html. Accessed: 21-08-2023

    2023

  67. [75]

    Cwe-770: Allocation of resources without limits or throttling,

    MITRE Corporation. Cwe-770: Allocation of resources without limits or throttling,

  68. [76]

    Cwe-94: Improper control of generation of code (’code injec- tion’), 2023

    MITRE Corporation. Cwe-94: Improper control of generation of code (’code injec- tion’), 2023. URL https://cwe.mitre.org/data/definitions/94.html. Accessed: 21-08-2023

    2023

  69. [77]

    Cwe-657: Violation of secure design principles, 2023

    MITRE Corporation. Cwe-657: Violation of secure design principles, 2023. URL https://cwe.mitre.org/data/definitions/657.html. Accessed: 21-08-2023

    2023

  70. [78]

    Accessed: 21-08-2023

    URL https://cwe.mitre.org/data/definitions/770.html . Accessed: 21-08-2023

    2023

  71. [79]

    Cwe-1352: Owasp top ten 2021 category a06:2021 - vulnerable and outdated components, 2023

    MITRE Corporation. Cwe-1352: Owasp top ten 2021 category a06:2021 - vulnerable and outdated components, 2023. URL https://cwe.mitre.org/data/definitions /1352.html. Accessed: 21-08-2023

    2021

  72. [80]

    URL https: //cwe.mitre.org/data/definitions/611.html

    Cwe-611: Improper restriction of xml external entity reference, 2023. URL https: //cwe.mitre.org/data/definitions/611.html. Accessed: 21-08-2023

    2023

  73. [81]

    Cwe-328: Use of weak hash, 2023

    MITRE Corporation. Cwe-328: Use of weak hash, 2023. URL https://cwe.mitre. org/data/definitions/328.html. Accessed: 21-08-2023

    2023

  74. [82]

    Cwe-185: Incorrect regular expression, 2023

    MITRE Corporation. Cwe-185: Incorrect regular expression, 2023. URL https: //cwe.mitre.org/data/definitions/185.html. Accessed: 21-08-2023

    2023

  75. [83]

    Cwe-601: Url redirection to untrusted site (’open redirect’),

    MITRE Corporation. Cwe-601: Url redirection to untrusted site (’open redirect’),

  76. [84]

    URL https://cwe.mitre.org/da ta/definitions/352.html

    Cwe-352: Cross-site request forgery (csrf), 2023. URL https://cwe.mitre.org/da ta/definitions/352.html. Accessed: 21-08-2023

    2023

  77. [85]

    Cwe-80: Improper neutralization of script-related html tags in a web page (basic xss),

  78. [86]

    Accessed: 21-08-2023

    URL https://cwe.mitre.org/data/definitions/80.html . Accessed: 21-08-2023

    2023

  79. [87]

    Towards provably efficient quantum algorithms for large-scale machine learning models

    Jens Eisert, Junyu Liu, Minzhao Liu, Jin-Peng Liu, Ziyu Ye, Yuri Alexeev, and Liang Jiang. Towards provably efficient quantum algorithms for large-scale machine learning models. Research Square (Research Square), 2023. https://doi.org/10.21203/rs. 3.rs-2860733/v1. URL https://...

    2023 doi

  80. [88]

    Accessed: 21-08-2023

    URL https://cwe.mitre.org/data/definitions/601.html . Accessed: 21-08-2023

    2023

  81. [89]

    URL https://cwe.mitre.org/data/defin itions/23.html

    Cwe-23: Relative path traversal, 2023. URL https://cwe.mitre.org/data/defin itions/23.html. Accessed: 21-08-2023

    2023

  82. [90]

    URL https://cwe.mitre.org/data/definitions/1321.ht ml

    Cwe-1321: Improperly controlled modification of object prototype attributes (’proto- type pollution’), 2023. URL https://cwe.mitre.org/data/definitions/1321.ht ml. Accessed: 21-08-2023

    2023

  83. [91]

    URL https://cwe.mitre.or g/data/definitions/250.html

    Cwe-250: Execution with unnecessary privileges, 2023. URL https://cwe.mitre.or g/data/definitions/250.html. Accessed: 21-08-2023

    2023

  84. [92]

    Targeted backdoor attacks on deep learning systems using data poisoning, 2017

    Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, 2017. URL https://arxiv. org/abs/1712.05526

    2017 arXiv

  85. [93]

    Software must be secure by design, and artificial intelligence is no exception — cisa, 08 2023

    CISA. Software must be secure by design, and artificial intelligence is no exception — cisa, 08 2023. URL https://www.cisa.gov/news-events/news/software-must-b e-secure-design-and-artificial-intelligence-no-exception

    2023

  86. [94]

    Planting undetectable backdoors in machine learning models

    Shafi Goldwasser, Michael P Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models. In 2022 IEEE 63rd Annual Sym- posium on Foundations of Computer Science (FOCS) , pages 931–942. IEEE, 2022

    2022

  87. [95]

    Reflections on trusting trust

    Ken Thompson. Reflections on trusting trust. Communications of the ACM , 27(8): 761–763, 1984

    1984

  88. [96]

    Systematic evaluation of backdoor data poisoning attacks on image classifiers

    Loc Truong, Chace Jones, Brian Hutchinson, Andrew August, Brenda Praggastis, Robert Jasper, Nicole Nichols, and Aaron Tuor. Systematic evaluation of backdoor data poisoning attacks on image classifiers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern...

    2020

  89. [97]

    The secret revealer: Generative model-inversion attacks against deep neural networks

    Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 253–261, 2020

    2020

  90. [98]

    Discovery of ai/ml supply chain vulnerabilities within automotive cyber-physical systems

    Daniel Williams, Chelece Clark, Rachel McGahan, Bradley Potteiger, Daniel Cohen, and Patrick Musau. Discovery of ai/ml supply chain vulnerabilities within automotive cyber-physical systems. In 2022 IEEE International Conference on Assured Autonomy (ICAA), pages 93–96. IEEE, 2022

    2022

  91. [99]

    Why api attacks are increasing and how to avoid them, 07 2023

    Maria Korolov. Why api attacks are increasing and how to avoid them, 07 2023. URL https://www.csoonline.com/article/646557/why-api-attacks-are-increasin g-and-how-to-avoid-them.html

    2023

  92. [100]

    URL https://dl.acm.org/doi/10

    I know what you trained last summer: A survey on stealing machine learning models and defences — acm computing surveys, 2023. URL https://dl.acm.org/doi/10. 1145/3595292

    2023

  93. [101]

    Membership inference attacks against machine learning models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. arXiv (Cornell University) , 05

  94. [102]

    Copilot security: A user study, 08 2023

    Owura Asare, N Asokan, and Meiyappan Nagappan. Copilot security: A user study, 08 2023. URL https://arxiv.org/pdf/2308.06587.pdf

    2023 arXiv

  95. [103]

    Secret trove offers rare look into russian cyberwar ambitions, 03 2023

    Craig Timberg, Ellen Nakashima, Hannes Munzinger, and Hakan Tanriverdi. Secret trove offers rare look into russian cyberwar ambitions, 03 2023. URL https://www. washingtonpost.com/national-security/2023/03/30/russian-cyberwarfare-d ocuments-vulkan-files/

    2023

  96. [104]

    Data-free knowl- edge distillation for object detection

    Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez. Data-free knowl- edge distillation for object detection. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision , pages 3289–3298, 2021

    2021

  97. [105]

    Ignore previous prompt: Attack techniques for language models

    F´ abio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527 , 2022

    2022 arXiv

  98. [106]

    ”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2023

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. ”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2023. URL https://arxiv.org/abs/2308.03825

    2023 arXiv

  99. [107]

    Privacy risks of general-purpose language models

    Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP) , pages 1314–1331. IEEE, 2020

    2020

  100. [108]

    Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn

    Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , 06 2022. https://doi.org/10 .1145/3520312.3534862. 7 Appendix 1 im...

    2022

  101. [110]

    Federated large language model : A position paper, 2023

    Chaochao Chen, Xiaohua Feng, Jun Zhou, Jianwei Yin, and Xiaolin Zheng. Federated large language model : A position paper, 2023. URL https://arxiv.org/pdf/2307 .08925.pdf

    2023

  102. [111]

    To repeat or not to repeat: Insights from scaling llm under token-crisis, 05 2023

    Fuzhao Xue, Yao Fu, Wangchushu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling llm under token-crisis, 05 2023. URL https: //arxiv.org/pdf/2305.13230.pdf

    2023 arXiv

  103. [112]

    Training compute-optimal large language models, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models, 2022

    2022

  104. [113]

    Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection

    Yizheng Chen, Zhoujie Ding, Xinyun Chen, and David Wagner. Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. arXiv preprint arXiv:2304.00409, 2023

    2023 arXiv

  105. [2017]

    URL https://ieeexplore.ieee

    https://doi.org/10.1109/sp.2017.41 . URL https://ieeexplore.ieee. org/document/7958568

    2017

  106. [2023]

    Accessed: 21-08-2023

    URL https://cwe.mitre.org/data/definitions/401.html . Accessed: 21-08-2023

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.