Pith.

Software Engineering · arXiv:2611.19086 v1 · submitted 14 Nov 2026 · 99 references

Beyond Benchmarks: Evaluating Large Language Models and Agentic Patterns for Static Code Analysis Warning Classification

Lakmal Deshapriya, Sherlock A. Licorish, Brendon J. Woodford, Grant Dick

The paper says

“Context: Static Code Analysis (SCA) tools support software verification and validation, reducing the time and cost of code reviews.”

more of the abstract

However, the best SCA tools in the practitioner community generate a high volume of false positives, which developers tend to ignore, leading to genuine warnings being overlooked. Accordingly, studies increasingly adopt Large Language Models (LLMs) to identify and reduce false positives. Existing studies have produced promising results on public datasets, but underperformed on unseen datasets. Moreover, researchers do not leverage the full potential of Agentic patterns in LLMs or incorporate established code quality standards for warning classification. Objectives: This study utilises four manually annotated datasets to evaluate the performance of three contemporary LLMs for SCA warning classification in real-world applications, and incorporates Agentic patterns and established coding standards to enhance models' performance, addressing gaps identified in existing studies. Method: Three LLMs were selected using the LiveCodeBench benchmark, and the Structured Chain-of-Thought (SCoT) prompting technique was used to evaluate these LLMs. Furthermore, we incorporated Agentic patterns, such as Debating and Self-Consistency, to potentially improve performance in SCA warning classification. Results: Our results show that the performance of different LLMs varies with the characteristics of the dataset, ranging from 0.52 to 0.96 F1-Score for the best models. Additionally, the use of Agentic patterns does not improve LLMs' performance in most scenarios. Conclusions: SCA warning classification using LLMs works best only with certain warning types. We recommend evaluating LLMs in real-world applications rather than using publicly available datasets/benchmarks to understand their full utility.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 17 Nov 2026 · read the full text (168,366 characters) · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A careful four-dataset evaluation finds LLMs classify static-analysis warnings worse on unseen code than public benchmarks suggest, with debating and self-consistency patterns giving no reliable improvement. This line is wrong

10 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

The paper, first character to last

The reviewer was given the whole text, 168,366 characters. Not checked: figures, code, what the 99 references say.

10 minor objections

The paper says, at Section 2.2, Table 1

Pith says: minor objection 1 of 10

The goals G1-G4 are defined with associated metrics, but the paper does not consistently report all metrics for all experiments. For example, in RQ2 results (Figure 6), only Precision, Recall, MCC, F1-Score, BA, and time are shown, but the goals are not explicitly mapped. Consider adding a table that maps each experiment to the goals and metrics.

The paper says, at Section 4, Table 2

Pith says: minor objection 2 of 10

The false positive percentage in the final dataset varies widely (70.50%, 61.54%, 20.00%, 14.77%). This is a strength, but the paper should discuss how class imbalance affects the metrics and whether any techniques (e.g., class weighting) were considered. The use of MCC and BA partially addresses this, but a brief discussion would help.

show 8 more

The paper says, at Section 5.1, Table 4

Pith says: minor objection 3 of 10

The slicing approach 'TEN_LINE_DOWN' appears in Appendix C for rule java:S1451 but is not listed in Table 4. The table should include all approaches used.

The paper says, at Section 6.1, Table 12

Pith says: minor objection 4 of 10

The Cochran's Q test is applied to accuracy, but accuracy is not the primary metric for imbalanced datasets. The paper should consider using a metric more robust to imbalance (e.g., MCC or BA) for the statistical test, or justify why accuracy is appropriate here.

The paper says, at Section 6.1, Figure 5

Pith says: minor objection 5 of 10

The boxplots in Figure 5 are not clearly labeled. It is difficult to distinguish which box corresponds to which ML model. The figure should include a legend or clearer labels.

The paper says, at Section 7.1

Pith says: minor objection 6 of 10

The paper states 'our approach performs better than existing studies that used non-publicly available datasets for their evaluations.' This claim is not directly supported by the data in Appendix A, as the comparison is across different datasets and metrics. Please clarify or remove.

The paper says, at Section 8, Internal threats

Pith says: minor objection 7 of 10

The paper mentions that reliability checks were conducted only for Datasets 1 and 3, citing Deshapriya et al. [46]. This is a limitation for generalizing to Datasets 2 and 4. The paper should state this more prominently and consider whether any additional checks were performed for those datasets.

The paper says, at Appendix A

Pith says: minor objection 8 of 10

The performance values for our study in Appendix A include some entries that appear to be percentages (e.g., Balanced Accuracy 60.746 for Gemini on SonarQube_Apache_Tomcat) but are not formatted consistently. Please verify and format uniformly.

The paper says, at General

Pith says: minor objection 9 of 10

The paper uses the term 'Agentic patterns' throughout, but the standard term in the literature is 'agentic patterns' or 'multi-agent patterns'. Consider using 'agentic patterns' for consistency.

The paper says, at Section 5.2

Pith says: minor objection 10 of 10

The prompt template in Appendix G is useful, but the paper does not specify how the coding standards are incorporated into the prompt for each warning type. The example shows a placeholder '{coding_standards}', but the actual mapping is not described. Please clarify.

What would settle it

If an independent replication using the same datasets and prompts found that agentic patterns (Debating, Self-Consistency) consistently improve F1-score by more than 0.05 across multiple LLMs and warning types, or that LLMs match or exceed the best ML model's performance on all four datasets, the paper's central claims would be contradicted.

from Pith's summary of the paper

How the review went

5 passes by deepseek-v4-flash · 17 Nov 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic concern withdrawn: Pith's reviewer misread the paper 24 s · 44,811 tokens read, 4,408 written
  3. Referee 10 minor 12 s · 45,196 tokens read, 2,210 written
  4. Check against the paper kept 0 of 5 objections 22 s · 46,046 tokens read, 3,132 written
  5. Editor wrote the verdict above 19 s · 45,500 tokens read, 3,275 written

Not run on this paper: the simulated authors' defence.

Pith's passes raised 5 objections. A check against the paper's text kept 0 and withdrew 5 (3 the paper already says; 2 Pith's reviewer misread the paper).

The review in full

The review in brief, machine-written

This paper reports a careful empirical evaluation of three LLMs—O4-mini, Gemini-2.5-Pro, and the open-weight OpenReasoning-Nemotron-32B—on four manually annotated datasets of PMD and SonarQube warnings the models had not been trained on. The headline finding is a credible negative result: on these unseen datasets, F1 scores range from 0.52 to 0.96, and several datasets fall well below the near-perfect scores prior work reported on public benchmarks. The paper also finds that two agentic patterns, debating and self-consistency with majority voting, do not reliably improve classification, and that simpler ML models trained on the same datasets match or exceed the LLMs.

The work is methodologically serious. Labels were assigned against documented coding standards with a reported inter-rater agreement check, model choice is anchored to a contamination-free leaderboard, and the comparisons use Cochran's Q and pairwise McNemar tests with correction. The per-rule slicing schemes and modified rule descriptions are given in appendices, and a replication package with raw results and source code is provided. The paper's own threats-to-validity section is candid, which earns trust.

The soft spots are modest and mostly acknowledged. First, the program slicing is load-bearing—the LLM only ever sees the slice—yet slice quality rests on the authors' own manual verification rather than independent validation; I would want to see that confirmed before betting on the results. Second, the ML comparison is summarised with a single boxplot figure; the training details live in a companion publication, so the claim that ML beats LLMs is asserted here more than demonstrated. Third, RQ2 covers only two rules, which the authors flag as limiting generalisation of the agentic-pattern conclusion. One claim to temper: "first to systematically demonstrate" performance on unseen datasets sits a little awkwardly against the paper's own review of Wen et al. and Iranmanesh et al., both of whom report weaker results on more realistic code.

Who this is for: practitioners deciding whether to invest in an LLM-based warning classifier, and researchers working on the benchmark-versus-real-world gap. The negative result is worth having, and the dataset and protocol are useful assets. I would send this to peer review.

The stress test's concern

Withdrawn: Pith's reviewer misread the paper.

The paper says

“We selected few Machine Learning (ML) approaches as baseline for our study as ML approaches were popular in academia for SCA warning classification before LLMs were introduced [74, 78, 90, 91] . We used four Code Representation Learning (CRL) models (Code2Vec, CodeBERT, FastText and Word2Vec) to generate numerical representations of source code program slices to ML algorithms. We utilised six ML algorithms such as BGRU, BiLSTM, GRU, LSTM, Random Forest, and TextCNN). Figure 5 shows the results for ML approaches trained on the same datasets from our previous study [92]. Datasets were trained with multiple ML models 10 times each and graphs in Figure 5 show boxplots for F1-Score values for each iteration. According to the Figure 5, the best ML model is varies based on the dataset. However, some ML models performed better than LLM approaches evaluated on the same datasets shows that LLM models produce lower results, mainly for Datasets 2-4.”

The referee's report

Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.

Pith's reading of the paper (a machine's summary, not the authors')

The paper investigates whether large language models (LLMs) can reliably classify static code analysis (SCA) warnings as true or false positives when applied to code the LLMs have not seen during training. The authors created four manually annotated datasets from two source code corpora (Stack Overflow snippets and Apache Tomcat) scanned with two SCA tools (PMD and SonarQube). They evaluated three top-performing LLMs (O4-mini, Gemini-2.5-Pro, OpenReasoning-Nemotron-32B) using Structured Chain-of-Thought prompting and also tested agentic patterns (Debating and Self-Consistency). The central finding is that LLM performance varies widely across datasets (F1-scores from 0.52 to 0.96), is often lower than on public benchmarks, and that agentic patterns do not improve classification in most scenarios. The paper further reports that traditional machine learning models trained on the same datasets outperform the LLMs. The authors conclude that SCA warning classification with LLMs works best only for certain warning types and that real-world evaluation is essential.

What carries the argument. The central mechanism is the use of program slicing tailored to each SCA rule (e.g., ISSUE_LINE_ONLY, AST_FORWARD, DATA_DEPENDENCY) to extract minimal relevant code context for the LLM, combined with Structured Chain-of-Thought (SCoT) prompting that embeds coding standards and rule descriptions. The agentic patterns evaluated are Debating (multiple LLM agents with different roles) and Self-Consistency (multiple runs with majority voting). The evaluation uses four manually annotated datasets with varying false-positive rates (14.77% to 70.50%) and multiple metrics (MCC, F1, Balanced Accuracy, time).

Core claim. The paper claims that LLMs for SCA warning classification perform significantly worse on unseen real-world datasets than on public benchmarks, and that agentic patterns (Debating and Self-Consistency) do not improve performance in most scenarios. The authors report F1-scores ranging from 0.52 to 0.96 across four datasets, with the best performance on a PMD/Stack Overflow dataset (0.96 F1) and much lower scores on other datasets. They also find that traditional ML models (e.g., Random Forest) trained on the same data outperform the LLMs. The paper argues that public benchmarks may overestimate LLM capability due to potential training data overlap, and that incorporating coding standards into prompts does not close the gap.

The premise it rests on. The paper assumes that the program slicing approaches defined for each SCA rule (Table 4, Appendices B and C) extract sufficient and correct context for the LLM to classify every warning, but this was only manually verified by the authors (100% agreement on a 10% sample) and not independently validated or quantitatively measured for slice quality.

If the paper is right (Pith's inferences, not the authors')

If the paper's findings hold, practitioners should not rely on public benchmark performance to select LLMs for SCA warning classification; real-world evaluation on their own codebases is necessary.

The use of agentic patterns (Debating, Self-Consistency) for SCA warning classification appears to add cost and latency without reliable performance gains, so practitioners should not assume these patterns will help.

Traditional machine learning models (e.g., Random Forest) may be more cost-effective and performant than LLMs for this task, especially when training data is available.

SCA warning classification with LLMs works well only for certain warning types (e.g., CommentRequired in PMD), suggesting that rule-specific approaches may be needed rather than a one-size-fits-all LLM.

The paper's datasets and methodology provide a template for evaluating LLMs on unseen SCA warnings, which could become a standard practice in the field.

The paper's finding that ML models outperform LLMs on the same datasets suggests that the complexity of LLMs may be unnecessary for this binary classification task, and that simpler models with engineered features may capture the relevant patterns more efficiently.

The variability in LLM performance across datasets (F1 from 0.52 to 0.96) implies that the characteristics of the codebase (e.g., code snippet vs. full project, SCA tool used) strongly influence LLM effectiveness, which may extend to other code understanding tasks beyond SCA warning classification.

The failure of agentic patterns to improve performance may indicate that the SCA warning classification task does not benefit from multi-step reasoning or debate, possibly because the necessary information is already present in the prompt and additional LLM calls introduce noise rather than insight.

The paper's reliance on manual program slicing per SCA rule (Table 4, Appendices B-C) suggests that automating slice generation could be a key enabler for practical deployment, but the paper does not evaluate the quality or completeness of these slices systematically.

Check this review

What was read

The passes were given all 168,366 characters of the text of the paper's PDF.

Fingerprint of that text (SHA-256): 8346cd79ea9840e2a14252498b5898cdfcda3e981741a5de39a1bf7590073759

Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:

PYTHONPATH=. python scripts/review_input_check.py 2611.19086

The passes, in the order they ran

  1. Reader deepseek-v4-flash · medium effort · $0.0048 · instructions of the Reader, fingerprint f9ead7300382…
  2. Skeptic deepseek-v4-flash · medium effort · $0.0026 · instructions of the Skeptic, fingerprint 8f88c0f61488…
  3. Referee deepseek-v4-flash · medium effort · $0.0045 · instructions of the Referee, fingerprint a852d99aea0d…
  4. Check against the paper deepseek-v4-flash · medium effort · $0.0048 · instructions of the Check against the paper, fingerprint 6c7a6883d551…
  5. Editor deepseek-v4-flash · medium effort · $0.0026 · instructions of the Editor, fingerprint e7211f6a3d27…
  6. Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0046 · instructions of the Summary for the page, fingerprint e4f56bf568e7…

The passes cost $0.024 in model calls together.

Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.0ef1a750a3e3

Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.

curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum

prints the fingerprint of the reader's instructions. The same works for every pass.

The receipt

The review's fingerprint was taken on 17 Nov 2026. The review you are reading still matches it.

Fingerprint of the review (SHA-256): dc11573720f2533e456f659521ed46726c6ef4f4a80e523b143b4734eb980788

curl -s 'https://pith.science/api/v1/verdicts/2611.19086?form=canonical' | sha256sum

prints the fingerprint of the review as it stands now. The rule, in four steps.

A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, from the text of the paper's PDF.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Context: Static Code Analysis (SCA) tools support software verification and validation, reducing the time and cost of code reviews. However, the best SCA tools in the practitioner community generate a high volume of false positives, which developers tend to ignore, leading to genuine warnings being overlooked. Accordingly, studies increasingly adopt Large Language Models (LLMs) to identify and reduce false positives. Existing studies have produced promising results on public datasets, but underperformed on unseen datasets. Moreover, researchers do not leverage the full potential of Agentic patterns in LLMs or incorporate established code quality standards for warning classification. Objectives: This study utilises four manually annotated datasets to evaluate the performance of three contemporary LLMs for SCA warning classification in real-world applications, and incorporates Agentic patterns and established coding standards to enhance models' performance, addressing gaps identified in existing studies. Method: Three LLMs were selected using the LiveCodeBench benchmark, and the Structured Chain-of-Thought (SCoT) prompting technique was used to evaluate these LLMs. Furthermore, we incorporated Agentic patterns, such as Debating and Self-Consistency, to potentially improve performance in SCA warning classification. Results: Our results show that the performance of different LLMs varies with the characteristics of the dataset, ranging from 0.52 to 0.96 F1-Score for the best models. Additionally, the use of Agentic patterns does not improve LLMs' performance in most scenarios. Conclusions: SCA warning classification using LLMs works best only with certain warning types. We recommend evaluating LLMs in real-world applications rather than using publicly available datasets/benchmarks to understand their full utility.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-17

The authors' main claim, as the classifier picked it out 0.99

“Conclusions: SCA warning classification using LLMs works best only with certain warning types.”

How the main claim is stated: asserted 0.99

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.86 · human participants 0.09 · controlled comparison 0.47 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.64 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Context: Static Code Analysis (SCA) tools support software verification and validation, reducing the time and cost of code reviews. background 1.00 0.04 0.01 asserted 1.00
However, the best SCA tools in the practitioner community generate a high volume of false positives, which developers tend to ignore, leading to genuine warnings being overlooked. background 0.96 0.06 0.02 asserted 1.00
Accordingly, studies increasingly adopt Large Language Models (LLMs) to identify and reduce false positives. background 1.00 0.04 0.02 asserted 0.99
Existing studies have produced promising results on public datasets, but underperformed on unseen datasets. gap 0.71 0.05 0.02 asserted 1.00
Moreover, researchers do not leverage the full potential of Agentic patterns in LLMs or incorporate established code quality standards for warning classification. gap 1.00 0.06 0.02 asserted 1.00
Objectives: This study utilises four manually annotated datasets to evaluate the performance of three contemporary LLMs for SCA warning classification in real-world applications, and incorporates Agentic patterns and established coding standards to enhance models' performance, addressing gaps identified in existing studies. contribution 1.00 0.86 0.03 asserted 0.99
Method: Three LLMs were selected using the LiveCodeBench benchmark, and the Structured Chain-of-Thought (SCoT) prompting technique was used to evaluate these LLMs. method 1.00 0.91 0.02 asserted 0.99
Furthermore, we incorporated Agentic patterns, such as Debating and Self-Consistency, to potentially improve performance in SCA warning classification. method 0.97 0.89 0.02 hedged possibility 0.95
Results: Our results show that the performance of different LLMs varies with the characteristics of the dataset, ranging from 0.52 to 0.96 F1-Score for the best models. result 1.00 0.97 0.07 asserted 1.00
Additionally, the use of Agentic patterns does not improve LLMs' performance in most scenarios. result 1.00 0.95 0.11 asserted 1.00
Conclusions: SCA warning classification using LLMs works best only with certain warning types. interpretation 0.82 0.80 0.81 asserted 0.99
We recommend evaluating LLMs in real-world applications rather than using publicly available datasets/benchmarks to understand their full utility. future work 1.00 0.18 0.15 asserted 0.74

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-17: 1 request, 8,911 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

References (99)
  1. [1] R. Yedida, H. J. Kang, H. Y. Tu, X. Q. Yang, D. V. Lo, and T. Menzies, “How to Find Actionable Static Analysis Warnings: A Case Study With FindBugs,” IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2856-2872, 2023 on Pith
  2. [2] C. Wen, Y. Cai, B. Zhang, J. Su, Z. Xu, D. Liu, S. Qin, Z. Ming, and T. Cong, “Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far Are We?,” ACM Transactions on Knowledge Discovery from Data, vol. 18, no. 7, pp. 1-34, 2024 on Pith
  3. [3] G. Liargkovas, E. Panourgia, and D. Spinellis, “Quieting the Static: A Study of Static Analysis Alert Suppressions,” arXiv preprint arXiv:2311.07482, 2023 on PithPith's reviewarXiv
  4. [4] J. Chen, H. Xiang, Z. Zhao, L. Li, Y. Zhang, B. Ding, Q. Li, and S. Xiong, “Utilizing Precise and Complete Code Context to Guide LLM in Automatic False Positive Mitigation,” arXiv preprint arXiv:2411.03079, 2024 on PithPith's reviewarXiv
  5. [5] L. Flynn, and W. Klieber, “Using llms to automate static-analysis adjudication and rationales,” CrossTalk: The Journal of Defense Software Engineering, May, 2024 on Pith
  6. [6] M. Iranmanesh, S. M. Sabet, S. Marefat, A. J. Ghasr, A. Wilson, I. Sharafaldin, and M. A. Tayebi, “ZeroFalse: Improving Precision in Static Analysis with LLMs,” arXiv preprint arXiv:2510.02534, 2025 on PitharXiv
  7. [7] Z. Zheng, K. Ning, Q. Zhong, J. Chen, W. Chen, L. Guo, W. Wang, and Y. Wang, “Towards an understanding of large language models in software engineering tasks,” Empirical Software Engineering, vol. 30, no. 2, pp. 50, 2025 on Pith
  8. [8] H. Jelodar, M. Meymani, and R. Razavi-Far, “Large language models (llms) for source code analysis: applications, models and datasets,” arXiv preprint arXiv:2503.17502, 2025 on PithPith's reviewarXiv
  9. [9] N. Huynh, and B. Lin, “Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications,” arXiv preprint arXiv:2503.01245, 2025 on PithPith's reviewarXiv
  10. [10] J. Svärd, and M. Teräs, “Enhancing Code Static Analysis with LLMs: An Experimentwith SpotBugs and Gemini,” M.S. thesis, Mid Sweden University, Östersund, Sweden, 2025 on Pith
  11. [11] C. Song, T. Ristenpart, and V. Shmatikov, “Machine learning models that remember too much,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications security, 2017, pp. 587-601 on Pith
  12. [12] S. Chatterjee, “Learning and memorization,” in International conference on machine learning, 2018, pp. 755-763 on Pith
  13. [13] A. J. Simmons, S. Barnett, J. Rivera-Villicana, A. Bajaj, and R. Vasa, “A large-scale comparative analysis of coding standard conformance in open-source data science projects,” in Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Me... on Pith
  14. [14] A. Kupari, N. Giacaman, and V. Terragni, “Adoption and Evolution of Code Style and Best Programming Practices in Open-Source Projects,” in 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2025, pp. 187-198 on Pith
  15. [15] A. Gullí, Agentic Design Patterns, 2025. 38 on Pith
  16. [16] L. Deshapriya, S. A. Licorish, and B. J. Woodford, “Taking a Closer Look at Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards,” arXiv preprint arXiv:2603.00821, 2026 on PitharXiv
  17. [17] L. Deshapriya, S. Licorish, and B. Woodford, “Taking a Closer Look at Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards - Replication Package,” Zenodo, 2025. https://doi.org/10.5281/zenodo.15770053 on Pithdoi:10.5281/zenodo.15770053
  18. [18] L. Deshapriya, S. Licorish, and B. Woodford, “AgenticSCA: Using Large Language Models for SCA Warning Classification: Will Agentic patterns improve performance? - Replication Package,” Zenodo, 2026. https://doi.org/10.5281/zenodo.18820913 on Pithdoi:10.5281/zenodo.18820913
  19. [19] D. Chicco, and G. Jurman, “The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,” BioData Mining, vol. 16, no. 1, pp. 4, 2023 on Pith
  20. [20] R. Diallo, C. Edalo, and O. O. Awe, "Machine learning evaluation of imbalanced health data: a comparative analysis of balanced accuracy, MCC, and F1 score," Practical Statistical Learning and Data Science Methods: Case Studies from LISA 2020 Global Network, USA, pp. 283-312: S... on Pith
show 79 more
  1. [21] X. Guo, Y. Yin, C. Dong, G. Yang, and G. Zhou, “On the class imbalance problem,” in 2008 Fourth international conference on natural computation, 2008, pp. 192-201 on Pith
  2. [22] J. M. Johnson, and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,” Journal of big data, vol. 6, no. 1, pp. 1-54, 2019 on Pith
  3. [23] A. Kharkar, R. Z. Moghaddam, M. Jin, X. Liu, X. Shi, C. Clement, N. Sundaresan, and I. C. Soc, “Learning to Reduce False Positives in Analytic Bug Detectors,” in ACM/IEEE 44th International Conference on Software Engineering (ICSE), Pittsburgh, PA, 2022, pp. 1307-1316 on Pith
  4. [24] X. Ge, C. Fang, Q. Zhang, D. Wu, B. Yu, Q. Zheng, A. Guo, S. Lin, Z. Zhao, and Y. Liu, “Pre-trained model-based actionable warning identification: A feasibility study,” arXiv preprint arXiv:2403.02716, 2024 on PithPith's reviewarXiv
  5. [25] G. D. Apostolidis, I. Kalouptsoglou, M. Siavvas, D. Kehagias, and D. Tzovaras, “AI- Enhanced Static Analysis: Reducing False Alarms Using Large Language Models,” in 2025 IEEE International Conference on Smart Computing (SMARTCOMP), 2025, pp. 288-293 on Pith
  6. [26] E. Sharkey, and P. Treleaven, “Optimising Large Language Models: Taxonomy and Techniques,” Available at SSRN, 2025 on Pith
  7. [27] C. Arora, A. I. Sayeed, S. Licorish, F. Wang, and C. Treude, “Optimizing Large Language Model Hyperparameters for Code Generation,” arXiv preprint arXiv:2408.10577, 2024 on PithPith's reviewarXiv
  8. [28] C. Wang, Z. Yang, S. Gao, C. Gao, T. Peng, H. Huang, Y. Deng, and M. Lyu, “RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry,” in Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025, pp. 93-104 on Pith
  9. [29] Y. Xia, J. Kim, Y. Chen, H. Ye, S. Kundu, C. C. Hao, and N. Talati, “Understanding the performance and estimating the cost of llm fine-tuning,” in 2024 IEEE International Symposium on Workload Characterization (IISWC), 2024, pp. 210-223 on Pith
  10. [30] I. Vieira, W. Allred, S. Lankford, S. Castilho, and A. Way, “How much data is enough data? Fine-tuning large language models for in-house translation: Performance evaluation across multiple dataset sizes,” arXiv preprint arXiv:2409.03454, 2024 on PithPith's reviewarXiv
  11. [31] C. Arora, A. I. Sayeed, S. Licorish, F. Wang, and C. Treude, “Optimizing LLMs for Code Generation: Which Hyperparameter Settings Yield the Best Results?,” in 2024 31st Asia-Pacific Software Engineering Conference (APSEC), 2024, pp. 281-290. 39 on Pith
  12. [32] Y. Wu, Y. Gu, X. Feng, W. Zhong, D. Xu, Q. Yang, H. Liu, and B. Qin, “Extending context window of large language models from a distributional perspective,” arXiv preprint arXiv:2410.01490, 2024 on PithPith's reviewarXiv
  13. [33] P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” arXiv preprint arXiv:2402.07927, 2024 on PithPith's reviewarXiv
  14. [34] B. Chen, Z. Zhang, N. Langrené, and S. Zhu, “Unleashing the potential of prompt engineering for large language models,” Patterns, vol. 6, no. 6, 2025 on Pith
  15. [35] S. Vatsal, and H. Dubey, “A survey of prompt engineering methods in large language models for different nlp tasks,” arXiv preprint arXiv:2407.12994, 2024 on PithPith's reviewarXiv
  16. [36] M. Labonne. "Decoding Strategies in Large Language Models," 27/02/2026; https://huggingface.co/blog/mlabonne/decoding-strategies on Pith
  17. [37] J. Cheng, M. Marone, O. Weller, D. Lawrie, D. Khashabi, and B. Van Durme, “Dated data: Tracing knowledge cutoffs in large language models,” arXiv preprint arXiv:2403.12958, 2024 on PithPith's reviewarXiv
  18. [38] M. Levy, A. Jacoby, and Y. Goldberg, “Same task, more tokens: the impact of input length on the reasoning performance of large language models,” arXiv preprint arXiv:2402.14848, 2024 on PithPith's reviewarXiv
  19. [39] R. Eskenasy. "What is CAG? Using external context to guide and improve AI responses," 7/10/2025; https://arato.ai/resources/hub/what-is-cag-context-augmented- generation/ on Pith
  20. [40] Y. Wang, and H. Li, “Code completion by modeling flattened abstract syntax trees as graphs,” in Proceedings of the AAAI conference on artificial intelligence, 2021, pp. 14015-14023 on Pith
  21. [41] M. Heffernan, K. Wilken, and G. Shobaki, “Data-dependency graph transformations for superblock scheduling,” in 2006 39th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO'06), 2006, pp. 77-88 on Pith
  22. [42] J. Ferrante, K. J. Ottenstein, and J. D. Warren, “The program dependence graph and its use in optimization,” ACM Transactions on Programming Languages and Systems (TOPLAS), vol. 9, no. 3, pp. 319-349, 1987 on Pith
  23. [43] "Joern - The Bug Hunter's Workbench," 27/02/2026; https://joern.io/ on Pith
  24. [44] B. Xu, A. Yang, J. Lin, Q. Wang, C. Zhou, Y. Zhang, and Z. Mao, “Expertprompting: Instructing large language models to be distinguished experts,” arXiv preprint arXiv:2305.14688, 2023 on PithPith's reviewarXiv
  25. [45] A. T. Kalai, O. Nachum, S. S. Vempala, and E. Zhang, “Why language models hallucinate,” arXiv preprint arXiv:2509.04664, 2025 on PithPith's reviewarXiv
  26. [46] L. Deshapriya, S. A. Licorish, and B. J. Woodford, “Understanding Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards,” Science of Computer Programming, pp. 103470, 2026 on Pith
  27. [47] S. Meldrum, S. A. Licorish, C. A. Owen, and B. T. R. Savarimuthu, “Understanding stack overflow code quality: A recommendation of caution,” Science of Computer Programming, vol. 199, pp. 102516, 2020 on Pith
  28. [48] Y. Wei, X. Sun, L. Bo, S. Cao, X. Xia, and B. Li, “A comprehensive study on security bug characteristics,” Journal of Software: Evolution and Process, vol. 33, no. 10, pp. e2376, 2021 on Pith
  29. [49] V. Piantadosi, S. Scalabrino, and R. Oliveto, “Fixing of security vulnerabilities in open source projects: A case study of apache http server and apache tomcat,” in 2019 12th IEEE Conference on software testing, validation and verification (ICST), 2019, pp. 68-78. 40 on Pith
  30. [50] S. A. Licorish, and M. Wagner, “Dissecting Copy/Delete/Replace/Swap mutations: Insights from a GIN Case Study,” in GECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference, 2022, pp. 1940-1945 on Pith
  31. [51] S. A. Licorish, and M. Wagner, “On the utility of marrying GIN and PMD for improving Stack Overflow code snippets,” arXiv preprint arXiv:2202.01490, 2022 on PithPith's reviewarXiv
  32. [52] S. A. Licorish, and M. Wagner, “Combining GIN and PMD for code improvements,” in GECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference, 2022, pp. 790-793 on Pith
  33. [53] S. A. Licorish, and T. Nishatharan, “Contextual profiling of stack overflow java code security vulnerabilities initial insights from a pilot study,” in 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), 2021, pp. 1060-1068 on Pith
  34. [54] M. Alqaradaghi, and T. Kozsik, “Comprehensive Evaluation of Static Analysis Tools for Their Performance in Finding Vulnerabilities in Java Code,” IEEE Access, vol. 12, pp. 55824-55842, 2024, 2024 on Pith
  35. [55] H. Liu, S. Chen, R. Feng, C. Liu, K. Li, Z. Xu, L. Nie, Y. Liu, and Y. Chen, “A Comprehensive Study on Quality Assurance Tools for Java,” in 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), Seattle, WA, 2023, pp. 285-297 on Pith
  36. [56] J. Yeboah, and S. Popoola, “Efficacy of Static Analysis Tools for Software Defect Detection on Open-Source Projects,” in Proceedings - 2023 International Conference on Computational Science and Computational Intelligence, CSCI 2023, 2023, pp. 1588-1593 on Pith
  37. [57] E. Zolduoarrati, S. A. Licorish, and N. Stanger, “Impact of individualism and collectivism cultural profiles on the behaviour of software developers: A study of stack overflow,” Journal of Systems and Software, vol. 192, pp. 111427, 2022 on Pith
  38. [58] F. Faul, E. Erdfelder, A.-G. Lang, and A. Buchner, “G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences,” Behavior research methods, vol. 39, no. 2, pp. 175-191, 2007 on Pith
  39. [59] R. Latpate, J. Kshirsagar, V. K. Gupta, and G. Chandra, Advanced sampling methods: Springer, 2021 on Pith
  40. [60] M. Idris, I. Syarif, and I. Winarno, “Development of vulnerable web application based on OWASP API security risks,” in 2021 International Electronics Symposium (IES), 2021, pp. 190-194 on Pith
  41. [61] P. Becker, L. Olsina, and M. F. Papa, “Approach to Improving Java Source Code Considering Non-compliance with a Java Style Guide,” in Argentine Congress of Computer Science, 2022, pp. 123-139 on Pith
  42. [62] A. Honkaranta, T. Leppänen, and A. Costin, “Towards practical cybersecurity mapping of stride and cwe—a multi-perspective approach,” in 2021 29th Conference of Open Innovations Association (FRUCT), 2021, pp. 150-159 on Pith
  43. [63] S. Simonetto, and P. Bosch, “Comprehensive threat analysis and systematic mapping of CVEs to MITRE framework,” in 1st International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security, NLPAICS 2024, 2024 on Pith
  44. [64] V. Arnaoudova, M. Di Penta, and G. Antoniol, “Linguistic antipatterns: What they are and how developers perceive them,” Empirical Software Engineering, vol. 21, pp. 104-158, 2016 on Pith
  45. [65] L. Murphy, T. Alliyu, A. Macvean, M. B. Kery, and B. A. Myers, “Preliminary analysis of REST API style guidelines,” Ann Arbor, vol. 1001, no. 48109, pp. 4, 2017. 41 on Pith
  46. [66] K. A. Sedek, N. Osman, M. N. Osman, and J. H. Kamaruzaman, “Developing a Secure Web Application Using OWASP Guidelines,” Comput. Inf. Sci., vol. 2, no. 4, pp. 137-143, 2009 on Pith
  47. [67] K. A. Neuendorf, The content analysis guidebook: sage, 2017 on Pith
  48. [68] A. Jordan, D. Kunkel, J. Manganello, and M. Fishbein, Media messages and public health: New York: Routledge, 2009 on Pith
  49. [69] E. Zolduoarrati, S. A. Licorish, and N. Stanger, “Harmonising contributions: exploring diversity in software engineering through CQA mining on stack overflow,” ACM transactions on software engineering and methodology, vol. 33, no. 7, pp. 1-54, 2024 on Pith
  50. [70] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37-46, 1960 on Pith
  51. [71] J. R. Landis, and G. G. Koch, “The measurement of observer agreement for categorical data,” biometrics, pp. 159-174, 1977 on Pith
  52. [72] L. Marks, Y. Zou, and A. E. Hassan, “Studying the fix-time for bugs in large open source projects,” in Proceedings of the 7th International Conference on Predictive Models in Software Engineering, 2011, pp. 1-8 on Pith
  53. [73] C. Zhang, H. Liu, J. Zeng, K. Yang, Y. Li, and H. Li, “Prompt-enhanced software vulnerability detection using chatgpt,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 276-277 on Pith
  54. [74] S. Yerramreddy, A. Mordahl, U. Koc, S. Wei, J. S. Foster, M. Carpuat, and A. A. Porter, “An empirical assessment of machine learning approaches for triaging reports of static analysis tools,” Empirical Software Engineering, vol. 28, no. 2, 2023 on Pith
  55. [75] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” arXiv preprint arXiv:1801.01681, 2018 on PithPith's reviewarXiv
  56. [76] M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “Deep learning similarities from different representations of source code,” in Proceedings of the 15th international conference on mining software repositories, 2018, pp. 542-553 on Pith
  57. [77] S. Salimi, and M. Kharrazi, “VulSlicer: Vulnerability detection through code slicing,” Journal of Systems and Software, vol. 193, pp. 111450, 2022 on Pith
  58. [78] Y. Yang, M. Wen, X. Gao, Y. Zhang, and H. Sun, “Reducing false positives of static bug detectors through code representation learning,” in 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2024, pp. 681-692 on Pith
  59. [79] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024 on PithPith's reviewarXiv
  60. [80] C. Ling, X. Zhao, J. Lu, C. Deng, C. Zheng, J. Wang, T. Chowdhury, Y. Li, H. Cui, and X. Zhang, “Domain specialization as the key to make large language models disruptive: A comprehensive survey,” ACM Computing Surveys, vol. 58, no. 3, pp. 1- 39, 2025 on Pith
  61. [81] D. Anisuzzaman, J. G. Malins, P. A. Friedman, and Z. I. Attia, “Fine-tuning large language models for specialized use cases,” Mayo Clinic Proceedings: Digital Health, vol. 3, no. 1, pp. 100184, 2025 on Pith
  62. [82] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171, 2022. 42 on PithPith's reviewarXiv
  63. [83] Y. Nong, M. Aldeen, L. Cheng, H. Hu, F. Chen, and H. Cai, “Chain-of-thought prompting of large language models for discovering and fixing software vulnerabilities,” arXiv preprint arXiv:2402.17230, 2024 on PithPith's reviewarXiv
  64. [84] J. Li, G. Li, Y. Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp. 1-23, 2025 on Pith
  65. [85] Y. He, S. Li, J. Liu, W. Wang, X. Bu, G. Zhang, Z. Peng, Z. Zhang, Z. Zheng, and W. Su, “Can large language models detect errors in long chain-of-thought reasoning?,” arXiv preprint arXiv:2502.19361, 2025 on PithPith's reviewarXiv
  66. [86] J. Bae, S. Kwon, and S. Myeong, “Enhancing software code vulnerability detection using gpt-4o and claude-3.5 sonnet: A study on prompt engineering techniques,” Electronics, vol. 13, no. 13, pp. 2657, 2024 on Pith
  67. [87] M. Aslam, “Cochran’s Q test for analyzing categorical data under uncertainty,” Journal of Big Data, vol. 10, no. 1, pp. 147, 2023 on Pith
  68. [88] M. W. Tate, and S. M. Brown, “Note on the Cochran Q test,” Journal of the American Statistical Association, vol. 65, no. 329, pp. 155-160, 1970 on Pith
  69. [89] F. Dexter, B. J. Hindman, and K. N. Thenuwara, “Evaluation using Cochran’s Q tests of the information content of individual items’ scores in the nurse anesthetist work habits instrument for use in providing feedback,” Perioperative Care and Operating Room Management, pp. 100519, 2025 on Pith
  70. [90] Y. Tan, and J. Tian, “A Method for Processing Static Analysis Alarms Based on Deep Learning,” Applied Sciences, vol. 14, no. 13, pp. 5542, 2024 on Pith
  71. [91] D. H. Nguyen, A. Seo, N. P. Nnamdi, and Y. Son, “False Alarm Reduction Method for Weakness Static Analysis Using BERT Model,” Applied Sciences (Switzerland), vol. 13, no. 6, 2023 on Pith
  72. [92] L. Deshapriya, S. Licorish, and B. Woodford, “Using machine learning approaches to classify false positives generated by static analysis tools - An empirical study using small datasets - Replication Package,” Zenodo, 2025. https://doi.org/10.5281/zenodo.17861561 on Pithdoi:10.5281/zenodo.17861561
  73. [93] X. Ge, C. Fang, X. Li, W. Sun, D. Wu, J. Zhai, S.-w. Lin, Z. Zhao, Y. Liu, and Z. Chen, “Machine learning for actionable warning identification: A comprehensive survey,” ACM Computing Surveys, vol. 57, no. 2, pp. 1-35, 2024 on Pith
  74. [94] R. Croft, M. A. Babar, and M. M. Kholoosi, “Data quality for software vulnerability datasets,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 121-133 on Pith
  75. [95] Y. Lin, Y. Li, M. Gu, H. Sun, Q. Yue, J. Hu, C. Cao, and Y. Zhang, “Vulnerability dataset construction methods applied to vulnerability detection: A survey,” in 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), 2022, pp. 141-146 on Pith
  76. [96] M. Elizabeth, M. Veyret, M. Couceiro, O. Dušek, and L. M. R. Barahona, "Exploring ReAct prompting for task-oriented dialogue: Insights and shortcomings." pp. 143-153 on Pith
  77. [97] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022 on PithPith's reviewarXiv
  78. [98] Y. Liu, T. Yang, S. Huang, Z. Zhang, H. Huang, F. Wei, W. Deng, F. Sun, and Q. Zhang, “Calibrating llm-based evaluator,” in Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (lrec-coling 2024), 2024, pp. 2638-2656 on Pith
  79. [99] R. Khanmohammadi, E. Miahi, M. Mardikoraem, S. Kaur, I. Brugere, C. Smiley, K. S. Thind, and M. M. Ghassemi, “Calibrating LLM Confidence by Probing Perturbed 43 Representation Stability,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing... on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment