Towards Effective Complementary Security Analysis using Large Language Models

Andreas Both; Christian N\"ather; Jan-Philipp Stegh\"ofer; Jonas Wagner; Simon M\"uller

REVIEW 3 major objections 4 minor 1 cited by

Towards Effective Complementary Security Analysis using Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a reassessment stage built on few-shot chain-of-thought prompting with self-consistency lets off-the-shelf LLMs filter 62.5% of SAST false positives on a benchmark, and 78.9% when three models are combined, without…

desk verdict Useful LLM-for-SAST benchmark with a candid limitations section, but the zero-FN numbers are test-set optima, so read the practical claim as conditional. read the letter →

arxiv 2506.16899 v2 pith:BAPGYIHS submitted 2025-06-20 cs.CR cs.AI

Jonas Wagner , Simon Müller , Christian Näther , Jan-Philipp Steghöfer , Andreas Both This is my paper

classification cs.CRcs.AI

keywords LargeLanguageModelsSASTfalsepositivedetectionChain-of-ThoughtpromptingSelf-ConsistencyconservativeanalysisvulnerabilitytriageOWASPBenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes adding an LLM-based reassessment stage after static application security testing (SAST), before human review, to weed out false positives while never discarding a genuine weakness. Using 3-shot chain-of-thought prompting and a self-consistency step that repeats the query five times, the authors show that off-the-shelf GPT-4o and Qwen2.5-32B each flag 62.5% of false positives on the OWASP Benchmark test split at thresholds where no true positive is missed. Combining the top three models' conservative outputs lifts false-positive detection to 78.9%. On a heterogeneous real-world dataset spanning five SAST tools and three languages, the best single model reaches 33.85% and the ensemble 38.46% with a perfect true positive rate. The point is that security teams could treat LLM-flagged false positives as safe to ignore without fine-tuning.

What carries the argument

The mechanism is a conservative LLM reassessment stage: for each SAST finding, the model receives the source file, line, weakness category, CWE-ID, and method name, and is prompted with the paper's 3-shot chain-of-thought template to output an explanation and a confidence score from 0.0 to 10.0. A user-defined threshold separates findings flagged as true positives from those flagged as false positives, and self-consistency repeats the query five times at temperature 0.7 before the scores are combined. Setting the threshold so that no true positive falls below it is what converts the LLM's scores into a conservative filter: findings below the threshold can be dropped without human review. The threshold is the load-bearing tuning knob, and optimal thresholds differ by model and dataset.

What would settle it

Run the paper's exact 3-shot CoT plus self-consistency protocol with GPT-4o at threshold 6 on a fresh SAST report whose findings have been independently verified, and count whether any genuine vulnerability receives a score below 6; a single such case would show the conservative guarantee does not transfer without label-dependent threshold tuning.

Watch

Extended reading notes

Core claim

The central discovery is that conservative analysis is achievable: a black-box LLM, prompted with three in-context examples, asked to reason step by step, and scored on a 0–10 agreement scale, can be tuned to a decision threshold at which it labels only false positives as removable while every genuine weakness in the evaluation set is preserved. At its optimal threshold (6 for GPT-4o, 3 for Qwen2.5-32B-Instruct), self-consistency over five runs raises true negatives to 80 of 128 (62.5%) with zero false negatives. The union of the top three models' conservative predictions reaches 102 of 128 (78.9%) because each model detects some false positives the others miss. The paper also shows the pattern holds outside the benchmark: on a real-world report set, Phi-4 filters 22 of 65 false positives (33.85%) with no false negatives, and the two best models combined reach 25 of 65 (38.46%).

Load-bearing premise

The results depend on the assumption that a threshold with zero false negatives, found using labeled evaluation data, will also avoid false negatives on unseen unlabeled reports.

Editorial extensions

If this is right

If the conservative guarantee holds, SAST reports can be pre-filtered by an LLM before human triage, removing up to 62.5% of false positives on Java benchmark findings and 33.85% on a mixed-language real-world report without expecting any genuine weakness to be lost.
Combining several conservatively filtering LLMs is strictly better: the union of GPT-4o, Qwen2.5-32B-Instruct, and Phi-4 detects 78.9% of benchmark false positives and 38.46% of real-world false positives while keeping the true positive rate at 100%, because each model contributes unique detections.
The approach works without fine-tuning, so it can be deployed with off-the-shelf models, including open-weights Qwen and Phi models that can run in private environments.
Model choice matters more than scale: several large Llama and Gemini variants failed to achieve conservative analysis, while smaller models like Phi-4 and Qwen2.5-32B succeeded.
The prompt and context identified in the paper (SAST-supplied fields plus full source file) can be reused across proprietary and open LLMs, and 3-shot CoT matches 5-shot at 35% lower token cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the per-category analysis (e.g., CWE-501 Trust Boundary Violation shows no overlap among models) hints that routing each finding to the model with the strongest category-level precision could raise the ensemble ceiling further; the paper does not test this routing.
Beyond the paper, because the conservative threshold is the single point of failure, a practical deployment could use an abstention rule: send every finding near the threshold to a human and only auto-drop scores far below it, which would preserve the guarantee even without labeled tuning.
Beyond the paper, the same scoring-and-threshold machinery could be applied to other triage tasks, such as filtering false positives from infrastructure-as-code scanners or SAST tools in CI pipelines, where the cost of a missed vulnerability is higher than the cost of an extra manual check.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is one of the more careful empirical comparisons of LLMs for SAST false-positive filtering, and it earns credit for an honest limitations section. The catch is exactly what the stress-test note says: the thresholds behind the zero-FN results are chosen from the labeled test sets, so the 62.5% and 78.9% numbers are in-sample optima, not validated operating points.

What's new: a systematic head-to-head of many open-weight and proprietary models under a conservative criterion (TPR=100%), the self-consistency comparison, and the ensemble union. The real-world dataset with five SAST tools and infrastructure files is a step beyond the Java-only OWASP benchmark. The authors also publish their result files and prompts on Zenodo, which makes the work independently checkable. That is real evidence and it raises the floor on this kind of study.

Where it's soft: the load-bearing claim is that LLMs can be a conservative pre-filter. That only holds at thresholds selected with access to ground truth. Section V-E says "the chosen thresholds also depend on labeled datasets, making it challenging to guarantee conservative analyses on unlabeled data" — that is the whole problem, and the abstract and conclusion tend to drop the qualifier. GPT-4o then misses one genuine weakness on the real-world set at threshold 2, showing that a threshold that is conservative for one set/model need not transfer. So the zero-FN guarantee is a test-time property, not a deployment property. Minor points: the real-world ground truth is from three experts but no inter-rater reliability is reported, and the few-shot examples are Java-specific, which may understate real-world transfer.

Bottom line: worth reading as a benchmark, not as a recipe. A serious referee should send it out, with a request to validate threshold selection on a held-out split or to propose a label-free heuristic, and to report rater agreement. I'd cite it as an empirical reference, but with the in-sample caveat clearly attached.

Referee Report

3 major / 4 minor

Summary. The manuscript proposes an LLM-based second-stage filter for static application security testing (SAST) reports. The filter uses 3-shot chain-of-thought prompting with a numeric confidence score, and a user-supplied threshold flags each finding as a true positive or false positive. The authors evaluate a broad set of proprietary and open-source LLMs on the OWASP Benchmark test split (403 findings, 128 FPs) with and without self-consistency. They report that GPT-4o and Qwen2.5-32B-Instruct each detect 80/128 FPs (62.5%) with zero false negatives at their selected thresholds, and that the union of GPT-4o, Qwen2.5-32B, and Phi-4 detects 102/128 FPs (78.9%) while preserving all true positives. On a real-world dataset of 114 findings from five SAST tools, Phi-4 detects 33.85% of FPs (22/65) and the ensemble of Phi-4 plus Qwen2.5-32B detects 38.46% (25/65) with zero false negatives. The paper argues that a conservative LLM pre-filter can reduce manual triage without fine-tuning.

Significance. The paper has notable strengths: a diverse model zoo spanning proprietary and open-source LLMs, a standardized prompting protocol, a real-world heterogeneous dataset covering multiple SAST tools and languages, and publicly available result files. If the zero-false-negative guarantee were shown to transfer to unseen data, the reported FP-reduction rates would be practically valuable for SAST triage. However, the central claim is conditional on thresholds selected from the labeled evaluation sets themselves. The headline effectiveness figures are in-sample optima, not validated out-of-sample operating points; the authors explicitly concede this in Section V-E. Because the conservative-analysis guarantee is the paper's main selling point, the missing out-of-sample calibration is a load-bearing weakness rather than a cosmetic issue. With a held-out validation split or a label-free threshold-selection procedure, the paper would be a solid empirical contribution to LLM-assisted security analysis.

major comments (3)

[V-B, V-D, V-E] The central effectiveness figures are obtained by selecting each model's decision threshold from the same labeled evaluation data used for measurement. For each LLM, the 'optimal threshold' in Fig. 4 and Fig. 5 is defined as the highest threshold with zero false negatives on the OWASP test split, and Section V-D performs the analogous selection at threshold 2 on the real-world test set. Consequently, the reported 62.5%, 78.9%, 33.85%, and 38.46% figures are best-case in-sample values, not properties of a fixed decision rule. This is precisely the limitation acknowledged in Section V-E ('The chosen thresholds also depend on labeled datasets, making it challenging to guarantee conservative analyses on unlabeled data'). To support RQ1 and RQ2, the authors should either select thresholds on a held-out validation split and evaluate on a disjoint test split, provide a label-free threshold-setting procedure, or reframe the results as exploratory upper bounds.
[V-D and Conclusion] The transferability of a conservative threshold is contradicted by the authors' own real-world experiment. GPT-4o, which achieved zero false negatives at its selected threshold on OWASP, misses one genuine weakness on the real-world dataset at threshold 2, while Qwen2.5-32B and Phi-4 remain conservative. This shows that a threshold that is conservative for one model/dataset need not be conservative for another, which undermines the concluding statement that 'once an LLM is proven to filter conservatively, its performance can be enhanced through threshold optimization.' The paper needs to specify how a user without labels can set or adapt the threshold in the Fig. 1 workflow, or provide evidence of a fixed threshold that transfers across datasets and models.
[V-C] The ensemble results inherit the same in-sample selection issue. The union of true-negative sets is formed by taking each model's individually test-optimal threshold, and the resulting 102/128 on OWASP and 25/65 on the real-world dataset are unions of sets that were each optimized on the labels of the same test instances. This is an optimistic estimate of any practical ensemble. A valid ensemble claim would require a fixed ensemble decision rule (e.g., majority vote or a union rule whose threshold is fixed on a validation split) evaluated on a disjoint test set, with the zero-false-negative property verified for that fixed rule.

minor comments (4)

[III-D and IV] Section III-D states that the OWASP training subset contains 1,557 samples, but Section IV refers to '1,577 items' in the preliminary study; this numerical discrepancy should be corrected.
[Fig. 5] The second row of Fig. 5 lists four metrics per threshold, but the panel header only reads 'Metrics (%)'; the mapping between the four numbers and TPR, FPR, Precision, and F2-score should be made explicit in the caption or legend.
[V-D and VI] The real-world dataset contains only 114 findings, and no inter-rater agreement measure is reported for the three security experts who produced the ground truth. Reporting agreement (e.g., Cohen's kappa or Fleiss' kappa) would strengthen the generalizability claim.
[Abstract and Conclusion] The Abstract and Conclusion present the headline percentages without the caveat that they depend on label-based threshold selection; given Section V-E, those sections should carry the same limitation explicitly.

Circularity Check

2 steps flagged · score 6.0 of 10

Zero-FN and FP-filtering numbers are in-sample optima: thresholds are selected using test-set labels before the reported effectiveness is read off.

fitted input called prediction [Section V-B, Fig. 5 (Self-Consistency results), also Fig. 4]
"There, a white dot marks the best threshold for each LLM —that is, the highest decision threshold at which no FNs occurred. For GPT-4o, the optimal threshold is 6, yielding a TN count of 80 (62.5% detected FPs) with zero FNs."

The 'best threshold' is defined as the highest threshold with zero FNs on the same OWASP test split whose 128 FPs are then counted as the effectiveness. Because the operating point is selected using the test-set FN labels, TPR=100% holds by construction at that point; the 62.5% TN figure is the value of the curve at a label-chosen point, not the performance of a fixed decision rule selected before seeing labels. Section V-E confirms this by conceding that 'the chosen thresholds also depend on labeled datasets, making it challenging to guarantee conservative analyses on unlabeled data.' Thus the headline 'without missing genuine weaknesses' is a fitted optimum, not a prediction.
fitted input called prediction [Section V-D, Conservative Analysis of Real-World Security Findings]
"In contrast to the results observed on the OWASP Benchmark test split, all LLMs performed best at a threshold of 2. GPT-4o identified 24 out of the 65 FPs but missed one genuine weakness and therefore did not achieve a conservative analysis on the real-world dataset. Both Qwen2.5-32B-Instruct and Phi-4 successfully performed conservative analyses, with Phi-4 correctly filtering out 33.85% of all FPs (22 out of 65)."

On the real-world set, 'performed best' is again determined after inspecting the labeled 49-TP/65-FP ground truth. Selecting threshold 2 because it yields zero FNs for Qwen2.5-32B-Instruct and Phi-4 makes the conservative-analysis claim a restatement of the selection criterion. The reported 33.85% and 38.46% FP-detection rates are operating points chosen with label access, so they are in-sample optima rather than validated operating points for unlabeled triage. The paper's own V-E limitation statement explicitly acknowledges that thresholds depend on labeled datasets, undermining the transferability of the conservative guarantee.

full rationale

The paper's central capability claim — that LLMs filter approximately 62.5% and 78.9% of FPs on OWASP, and 33.85% and 38.46% on a real-world set, without missing genuine weaknesses — is not a free prediction of a fixed decision rule. Section V-B defines each model's 'optimal threshold' as the highest threshold with zero FNs on the OWASP test split, and Section V-D does the same on the labeled real-world set, finding threshold 2. The TPR=100% property is therefore built into the operating-point selection, and the TN counts are simply the values at that label-chosen point. Section V-E explicitly concedes: 'The chosen thresholds also depend on labeled datasets, making it challenging to guarantee conservative analyses on unlabeled data.' The union result in Section V-C inherits this property, since the union of zero-FN sets trivially has zero FNs. This is not full definitional circularity: the LLMs' score distributions are independent outputs, and the threshold is a user input in the workflow. However, the central effectiveness claims reduce, by construction, to threshold fitting on the evaluation labels, making the conservative guarantee an in-sample optimum rather than a validated prediction. There is no load-bearing self-citation chain; the [18] citation concerns preliminary prompt selection and is not what forces the reported numbers.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The evaluation rests on benchmark and expert labels as ground truth, on the assumption that label-selected thresholds can support a conservative guarantee in practice, and on a hand-crafted prompt whose few-shot examples were tuned on the training split. The per-model thresholds and the chosen examples are fitted elements. No new physical or formal entities are introduced.

free parameters (3)

Decision threshold per LLM on the OWASP test split (after self-consistency) = GPT-4o: 6; Phi-4: 2; Qwen2.5-32B-Instruct: 3
Chosen as the highest threshold with zero false negatives on the labeled test split (Fig. 5); the reported 62.5% and 78.9% effectiveness numbers are measured at these fitted thresholds.
Decision threshold per LLM on the real-world dataset = Phi-4: 2; Qwen2.5-32B-Instruct: 2; GPT-4o did not achieve a conservative threshold
Thresholds were again selected by inspecting false-negative counts on the labeled real-world findings; the 33.85% and 38.46% figures are measured at these thresholds.
3-shot chain-of-thought example selection = Not numerically specified
The few-shot examples were selected during the preliminary study on the training split and then fixed; this hand-fitted choice can overfit the benchmark domain, and the paper notes the examples are Java-specific.

assumptions (4)

domain assumption OWASP Benchmark (v1.2) labels accurately represent the true vulnerability status of the SpotBugs findings used in the evaluation.
The paper computes TPR, FPR, precision and F2 from these labels and uses them to select thresholds; if the mapping from benchmark test cases to SpotBugs findings is imperfect, the conservative result would not transfer.
domain assumption The manual labels of the 114 real-world findings by three senior security experts are a reliable ground truth.
No inter-rater agreement metric is reported; the dataset is small (49 true positives, 65 false positives) and spans three programming languages plus infrastructure files.
domain assumption The 80/20 split of the OWASP data keeps the reported test performance independent of the prompt-design choices made in the preliminary study.
The preliminary study used the training split, but the OWASP Benchmark is public and may be in LLM training data, which the paper itself flags as a threat to validity.
domain assumption The self-consistency aggregation procedure used to combine the five score samples is a valid realization of the method from Wang et al.
The main text does not specify the aggregation function, so the comparison with self-consistency assumes a standard implementation that is not fully described.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Effective Complementary Security Analysis using Large Language Models." pith.science (2026). https://pith.science/paper/BAPGYIHS

@misc{pith2026250616899,
  author       = {Pith},
  title        = {Pith review of: Towards Effective Complementary Security Analysis using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BAPGYIHS}},
  note         = {Machine review of arXiv:2506.16899}
}

read the original abstract

A key challenge in security analysis is the manual evaluation of potential security weaknesses generated by static application security testing (SAST) tools. Numerous false positives (FPs) in these reports reduce the effectiveness of security analysis. We propose using Large Language Models (LLMs) to improve the assessment of SAST findings. We investigate the ability of LLMs to reduce FPs while trying to maintain a perfect true positive rate, using datasets extracted from the OWASP Benchmark (v1.2) and a real-world software project. Our results indicate that advanced prompting techniques, such as Chain-of-Thought and Self-Consistency, substantially improve FP detection. Notably, some LLMs identified approximately 62.5% of FPs in the OWASP Benchmark dataset without missing genuine weaknesses. Combining detections from different LLMs would increase this FP detection to approximately 78.9%. Additionally, we demonstrate our approach's generalizability using a real-world dataset covering five SAST tools, three programming languages, and infrastructure files. The best LLM detected 33.85% of all FPs without missing genuine weaknesses, while combining detections from different LLMs would increase this detection to 38.46%. Our findings highlight the potential of LLMs to complement traditional SAST tools, enhancing automation and reducing resources spent addressing false alarms.

Figures

Figures reproduced from arXiv: 2506.16899 by the authors.

**Figure 1.** Complementary Security Analysis Process was randomly split into a training subset (80%, 1,557 samples) and a testing subset (20%, 403 samples—275 TPs and 128 FPs). Both subsets include findings from eleven distinct vulnerability categories, each identified by unique CWE-IDs (e.g., CWE-501: Trust Boundary Violation) [18]. To demonstrate that our generalized framework not only improves security analysis on benchmark … view at source ↗

**Figure 2.** Chain-of-Thought Prompt Template tokens [18]. For that reason, we decided to continue using 3-shot over 5-shot CoT prompting in this work. Moreover, we observed that applying 3-shot CoT prompting five times repeatedly (following the SC approach) further enhanced performance compared to solely using CoT prompting; however, this improvement comes at the expense of five times the resource demands, including quintupled… view at source ↗

**Figure 3.** True Negative and False Negative Ratio of LLM-based Assessments at Threshold = 1 (y-axis inverted) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: True Negative Ratio of LLM Assessments across all Thresholds [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Self-Consistency (n=5) Results of GPT-4o, Phi-4 and Qwen2.5 32B [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: TN Classification Overlaps of Self-Consistency Results [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Using LLMs to Adjudicate Static-Analysis Alerts with Error Reduction Techniques
cs.SE 2026-07 conditional novelty 5.5 of 10

Mid-tier reasoning LLMs with consistency checks and LLM reasoning evaluation adjudicate static-analysis alerts at ≥98% recall and ≥94.8% specificity across Juliet, FormAI, and SV-COMP.

Reference graph

Works this paper leans on

29 extracted references · 10 canonical work pages · cited by 1 Pith paper

[1]

Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities,

Z. Guo, T. Tan, S. Liu, X. Liu, W. Lai et al., “Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities,” IEEE TSE, vol. 49, no. 12, pp. 5154–5188, 2023

work page 2023
[2]

“False negative - that one is going to kill you

A. S. Ami, K. Moran, D. Poshyvanyk, and A. Nadkarni, ““False negative - that one is going to kill you”: Understanding industry per- spectives of static analysis based security testing,” in IEEE Symposium on Security and Privacy . IEEE, 2024, pp. 3979–3997

work page 2024
[3]

Static program analysis,

A. Møller and M. I. Schwartzbach, “Static program analysis,” Oc- tober 2018, department of Computer Science, Aarhus University, http://cs.au.dk/˜amoeller/spa/

work page 2018
[4]

OW ASP Benchmark Project,

OW ASP Foundation, “OW ASP Benchmark Project,” 2016. [Online]. Available: https://web.archive.org/web/20240522054757/https: //owasp.org/www-project-benchmark/

arXiv 2016
[5]

Learning a classifier for false positive error reports emitted by static code analysis tools,

U. Koc, P. Saadatpanah, J. S. Foster, and A. A. Porter, “Learning a classifier for false positive error reports emitted by static code analysis tools,” in 1st ACM SIGPLAN International Workshop on Machine Learning and Programming Languages . ACM, 2017, pp. 35–42

work page 2017
[6]

An Empirical Assessment of Machine Learning Approaches for Triaging Reports of a Java Static Analysis Tool,

U. Koc, S. Wei, J. S. Foster, M. Carpuat, and A. A. Porter, “An Empirical Assessment of Machine Learning Approaches for Triaging Reports of a Java Static Analysis Tool,” in 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, 2019, pp. 288–299

work page 2019
[7]

Transformer-Based Language Models for Software Vul- nerability Detection,

C. Thapa, S. I. Jang, M. E. Ahmed, S. Camtepe, J. Pieprzyk, and S. Nepal, “Transformer-Based Language Models for Software Vul- nerability Detection,” in Proceedings of the 38th Annual Computer Security Applications Conference . USA: ACM, 2022, pp. 481–496

work page 2022
[8]

Using ChatGPT as a Static Application Security Testing Tool,

A. Bakhshandeh, A. Keramatfar, A. Norouzi, and M. M. Chekidehkhoun, “Using ChatGPT as a Static Application Security Testing Tool,” 2023, arXiv:2308.14434 [cs]

arXiv 2023

Show all 29 references

[9]

Software Vulnerability and Functionality Assessment using LLMs,

R. I. T. Jensen, V . Tawosi, and S. Alamir, “Software Vulnerability and Functionality Assessment using LLMs,” 2024, arXiv:2403.08429 [cs]

2024 arXiv
[10]

Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far Are We?

C. Wen, Y . Cai, B. Zhang, J. Su, Z. Xuet al., “Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far Are We?” ACM Transactions on Knowledge Discovery from Data , vol. 18, no. 7, pp. 1–34, 2024

2024
[11]

Assisting Static Analysis with Large Language Models: A ChatGPT Experiment,

H. Li, Y . Hao, Y . Zhai, and Z. Qian, “Assisting Static Analysis with Large Language Models: A ChatGPT Experiment,” in 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . ACM, 2023, pp. 2107–2111

2023
[12]

Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detection,

X. Zhou, D.-M. Tran, T. Le-Cong, T. Zhang, I. C. Irsan et al. , “Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detection,” Jul. 2024, arXiv:2407.16235 [cs]

2024 arXiv
[13]

Harnessing Large Language Mod- els for Software Vulnerability Detection: A Comprehensive Bench- marking Study,

K. Tamberg and H. Bahsi, “Harnessing Large Language Mod- els for Software Vulnerability Detection: A Comprehensive Bench- marking Study,” IEEE Access , vol. 13, pp. 29 698–29 717, 2025, arXiv:2405.15614 [cs]

2025 arXiv
[14]

Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine,

H. Nori, Y . T. Lee, S. Zhang, D. Carignan, R. Edgar et al. , “Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine,” 2023, arXiv:2311.16452 [cs]

2023 arXiv
[15]

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, and B. m. o. Ichter, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” 2023, arXiv:2201.11903 [cs]

2023 arXiv
[16]

Large Language Models are Zero-Shot Reasoners,

T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large Language Models are Zero-Shot Reasoners,” 2023, arXiv:2205.11916

2023 arXiv
[17]

Self- Consistency Improves Chain of Thought Reasoning in Language Models,

X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi et al. , “Self- Consistency Improves Chain of Thought Reasoning in Language Models,” 2023, arXiv:2203.11171 [cs]

2023 arXiv
[18]

Towards efficient complementary security analysis using large language models,

J. Wagner, “Towards efficient complementary security analysis using large language models,” May 2025. [Online]. Available: https://doi.org/10.5281/zenodo.15378450

2025 doi
[19]

A review of the f-measure: Its history, properties, criticism, and alternatives,

P. Christen, D. J. Hand, and N. Kirielle, “A review of the f-measure: Its history, properties, criticism, and alternatives,” ACM Comput. Surv. , vol. 56, no. 3, Oct. 2023. [Online]. Available: https://doi.org/10.1145/3606367

2023 doi
[20]

GPT-4 Technical Report,

OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad et al. , “GPT-4 Technical Report,” 2024, arXiv:2303.08774 [cs]

2024 arXiv
[21]

Hello GPT-4o,

OpenAI Team, “Hello GPT-4o,” May 2024, Blog Post. [Online]. Available: https://web.archive.org/web/20240815014626/https: //openai.com/index/hello-gpt-4o/

2024
[22]

The Llama 3 Herd of Models,

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle et al., “The Llama 3 Herd of Models,” 2024, arXiv:2407.21783 [cs]

2024 arXiv
[23]

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

Gemini Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” 2024, arXiv:2403.05530 [cs]

2024 arXiv
[24]

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,

M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah et al. , “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” 2024, arXiv:2404.14219 [cs]

2024 arXiv
[25]

Phi-4 Technical Report,

M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan et al. , “Phi-4 Technical Report,” Dec. 2024, arXiv:2412.08905 [cs]

2024 arXiv
[26]

Qwen Technical Report,

J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang et al., “Qwen Technical Report,” Sep. 2023, arXiv:2309.16609 [cs]

2023 arXiv
[27]

Qwen2 Technical Report,

A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou et al., “Qwen2 Technical Report,” Sep. 2024, arXiv:2407.10671 [cs]

2024 arXiv
[28]

Qwen2.5 Technical Report,

Qwen, A. Yang, B. Yang, B. Zhang, B. Hui et al., “Qwen2.5 Technical Report,” Jan. 2025, arXiv:2412.15115 [cs]

2025 arXiv
[29]

Qwen2.5-Coder Technical Report,

B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu et al. , “Qwen2.5-Coder Technical Report,” Nov. 2024, arXiv:2409.12186 [cs]

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities,

Z. Guo, T. Tan, S. Liu, X. Liu, W. Lai et al., “Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities,” IEEE TSE, vol. 49, no. 12, pp. 5154–5188, 2023

work page 2023

[2] [2]

“False negative - that one is going to kill you

A. S. Ami, K. Moran, D. Poshyvanyk, and A. Nadkarni, ““False negative - that one is going to kill you”: Understanding industry per- spectives of static analysis based security testing,” in IEEE Symposium on Security and Privacy . IEEE, 2024, pp. 3979–3997

work page 2024

[3] [3]

Static program analysis,

A. Møller and M. I. Schwartzbach, “Static program analysis,” Oc- tober 2018, department of Computer Science, Aarhus University, http://cs.au.dk/˜amoeller/spa/

work page 2018

[4] [4]

OW ASP Benchmark Project,

OW ASP Foundation, “OW ASP Benchmark Project,” 2016. [Online]. Available: https://web.archive.org/web/20240522054757/https: //owasp.org/www-project-benchmark/

arXiv 2016

[5] [5]

Learning a classifier for false positive error reports emitted by static code analysis tools,

U. Koc, P. Saadatpanah, J. S. Foster, and A. A. Porter, “Learning a classifier for false positive error reports emitted by static code analysis tools,” in 1st ACM SIGPLAN International Workshop on Machine Learning and Programming Languages . ACM, 2017, pp. 35–42

work page 2017

[6] [6]

An Empirical Assessment of Machine Learning Approaches for Triaging Reports of a Java Static Analysis Tool,

U. Koc, S. Wei, J. S. Foster, M. Carpuat, and A. A. Porter, “An Empirical Assessment of Machine Learning Approaches for Triaging Reports of a Java Static Analysis Tool,” in 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, 2019, pp. 288–299

work page 2019

[7] [7]

Transformer-Based Language Models for Software Vul- nerability Detection,

C. Thapa, S. I. Jang, M. E. Ahmed, S. Camtepe, J. Pieprzyk, and S. Nepal, “Transformer-Based Language Models for Software Vul- nerability Detection,” in Proceedings of the 38th Annual Computer Security Applications Conference . USA: ACM, 2022, pp. 481–496

work page 2022

[8] [8]

Using ChatGPT as a Static Application Security Testing Tool,

A. Bakhshandeh, A. Keramatfar, A. Norouzi, and M. M. Chekidehkhoun, “Using ChatGPT as a Static Application Security Testing Tool,” 2023, arXiv:2308.14434 [cs]

arXiv 2023