Pith. sign in

REVIEW 4 major objections 6 minor 50 references

StaAgent: An Agentic Framework for Testing Static Analyzers

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An agentic LLM pipeline can expose flawed rule implementations in static analyzers, finding 64 problematic rules across five tools, 53 of them missed by the leading baseline.

desk verdict A useful LLM-driven extension of metamorphic testing for static analyzers, but the 64-bug headline is only as solid as the weak semantic-equivalence oracle and a demonstrably wrong example. read the letter →

arxiv 2507.15892 v1 pith:FMRXJ5B3 submitted 2025-07-20 cs.SE

Elijah Nnorom , Md Basim Uddin Ahmed , Jiho Shin , Hung Viet Pham , Song Wang This is my paper
classification cs.SE
keywords staticanalysismetamorphictestinglargelanguagemodelsagenticframeworksemanticequivalencemutationanalyzerbugsJava
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StaAgent asks whether a static analyzer's verdict changes when it is given two Java programs that should behave identically. The paper tries to show that an agentic pipeline, in which one LLM agent writes a buggy seed from a rule's documentation, a second validates the bug with executable tests, a third rewrites the seed into semantically equivalent mutants, and a fourth compares analyzer output across the pair, can expose real flaws in analyzer rule implementations without hand-curated test suites. The evaluation reports 64 problematic rules in the latest versions of SpotBugs, SonarQube, ErrorProne, Infer, and PMD, with 53 of the 64 not detected by the Statfier baseline. Two of the reported bugs have already been fixed and three more confirmed by developers. If the finding holds, it gives static-analyzer maintainers a scalable way to find under-tested rules, and it warns users that a clean analyzer pass is not evidence that a bug class is absent.

What carries the argument

Metamorphic testing is the engine: feed an analyzer two programs that are semantically equivalent, and any difference in its verdict is a symptom of a faulty rule. StaAgent operationalizes that with four LLM agents. A Seed Generation Agent turns a rule's title and description into a small compilable buggy Java program; a Code Validation Agent writes JUnit tests, runs them, and has the LLM decide from the execution trace whether the bug matches the rule; a Mutation Generation Agent applies nine semantic-preserving operators (dead stores, duplicated assignments, unreachable if/switch/for/while branches, renaming, and do-while conversion); and an Analyzer Evaluation Agent runs the static analyzer on seed and mutants, raising Type1 or Type2 alarms. A mutant is admitted only if it compiles and produces identical outputs, exceptions, and execution traces on the same test cases, so the entire inference rests on that equivalence check.

What would settle it

For a concrete check, take the paper's SpotBugs example: the seed computes Math.abs(input.hashCode()) and the mutant adds an unreachable switch. Run both on an input whose hashCode equals Integer.MIN_VALUE and compare return values and exceptions; any divergence would refute that pair's equivalence and collapse that report. More generally, replay every reported seed/mutant pair under randomized differential testing with far more inputs than the LLM wrote, or under a formal equivalence checker; any observable divergence on a behavior the analyzer tracks marks that report a false positive.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that simple semantic-preserving transformations—inserting an unreachable switch, duplicating an assignment, renaming a variable—are enough to make mature static analyzers contradict themselves. The authors define Type1 as the analyzer detecting the bug in the seed but not in all equivalent mutants, and Type2 as the analyzer missing the seed and at least one mutant; they count both as evidence that a rule implementation is overly specific, brittle, or fundamentally inadequate. Across five Java analyzers and five LLMs, the pipeline surfaced 64 unique problematic rules after manual validation by two authors: 28 in SpotBugs, 18 in SonarQube, 6 in ErrorProne, 4 in Infer, and 8 in PMD. Of these, 53 were not found when the Statfier baseline was run on the same analyzer versions, and reporting the bugs to developers has already produced two fixes and three confirmations.

Load-bearing premise

The load-bearing premise is that a rewritten program is truly identical in behavior to the original whenever it gives the same outputs, exceptions, and execution traces on the tests the language model wrote; if those tests miss a difference that matters to the analyzer, some reported bugs could be false alarms.

Editorial extensions

If this is right

  • A clean report from a static analyzer is weaker evidence than users assume: even trivial rewrites like adding an unreachable branch can make a rule miss a bug it just found in the seed.
  • StaAgent and Statfier find largely different bugs, so semantic-preserving metamorphic testing is not yet saturated; combining seed sources broadens coverage.
  • Because seeds are generated from rule documentation rather than harvested from test suites, adding a new rule or analyzer requires no manual test collection, only a rule description and an LLM.
  • The 68.86% mutant validation rate means LLM-generated equivalent mutants are usable at scale, but only with compile checking and test-based equivalence filtering; without that filtering, false alarms multiply.
  • The two developer fixes and three confirmations show that at least some of the 64 reports are genuine rule defects, not artifacts of the harness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 64-bug count is probably a lower bound on real rule fragility: the pipeline discards seeds and mutants that fail compile or equivalence checks, and the rule set excludes style, framework-specific, and test-code rules, so the same machinery on other rule families would likely surface additional inconsistencies.
  • Because the same family of LLMs writes the seeds and the equivalence tests, the discovered bugs are biased toward patterns those models can reproduce; grammar-based or search-based mutant generators could reveal a complementary set of rule flaws that LLMs cannot express, so the method's coverage and its blind spots are entangled.
  • A Type2 report (missed seed and mutants) is effectively a ready-made regression test: even before maintainers fix the underlying rule, each Type2 pair can be added to the analyzer's test suite to prevent the gap from widening.
  • The equivalence oracle could be strengthened independently of the LLM layer: feeding each seed/mutant pair to a symbolic executor or a differential tester with broader inputs would separate genuine rule defects from cases where the mutant only seems equivalent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces StaAgent, a four-agent LLM pipeline that performs metamorphic testing of Java static analyzers. The Seed Generation Agent creates bug-inducing programs from analyzer rule descriptions, the Code Validation Agent generates and evaluates tests that trigger the seed bugs, the Mutation Generation Agent produces supposedly semantically equivalent mutants via nine mutation operators, and the Analyzer Evaluation Agent compares analyzer behavior on seeds and mutants, flagging Type1 (inconsistent detection) and Type2 (false negatives across variants) issues. The evaluation uses five LLMs (CodeLlama, DeepSeek, Codestral, Qwen, GPT-4o) and five analyzers (SpotBugs, SonarQube, ErrorProne, Infer, PMD), reporting 64 manually verified problematic rules, 53 of which are not detected by the Statfier baseline, with two bugs already fixed and three confirmed by developers.

Significance. If the 64 identified defects are genuine, this is a valuable contribution: it demonstrates a scalable, largely automated way to test static analyzer rules across multiple tools, localizes flaws to specific rules, and complements the existing Statfier approach. The authors' effort to report issues to developers and to release an artifact is commendable, and the comparison against Statfier on the latest analyzer versions is a useful empirical data point. However, the significance is directly bounded by the trustworthiness of the semantic-equivalence oracle and by the transparency of the manual verification procedure; the current evidence for the central count is weaker than the paper's conclusions suggest.

major comments (4)
  1. [Section 3.4] The definition of semantic equivalence as 'identical outputs, exceptions, and execution traces' on the same LLM-generated test set is too weak for the metamorphic-testing premise. If the seed and a mutant differ on some input not covered by the generated tests, the analyzer may legitimately behave differently, and a Type1 or Type2 alarm becomes a false positive. This risk is concrete for specific operators: Op2's floating-point reassociation (e.g., x = 1.0 + 0.1 - 0.1) need not preserve the exact value for all inputs, and Op9's for-to-do-while conversion changes behavior when the original loop condition is initially false. The paper reports neither the size nor the coverage of the test sets, nor any independent audit of the accepted mutants. Because every reported bug depends on this oracle, I ask the authors to either restrict the mutation operators to provably semantics-preserving transformations, use a stronger equivalence check (e.g., formal comparison or substantially larger coverage-guided test suites), and report a manual audit of accepted mutants per operator and per model.
  2. [Section 5.4, Table 3] The central claim of 64 problematic rules rests on manual verification, but the protocol is not described. The text states that 'two of the authors manually validated each bug instance,' but it does not explain how a Type1/Type2 alarm was classified as a true positive, whether the semantic equivalence of each seed-mutant pair was independently re-established during that validation, how disagreements between the two authors were resolved, or what inter-rater agreement was achieved. Without this information, and without the actual 64 seed-mutant pairs, their tests, the analyzer outputs, and the validation logs, an independent reader cannot assess the main count. The anonymous artifact link is mentioned but the artifact contents are not described; please include a complete replication package with these records.
  3. [Section 5.3, Figure 4] Figure 4 reports the percentage of mutants that pass the LLM-based validation, but this is not an independent measure of semantic equivalence. The manual inspection described in this section is limited to compile-failed mutants; the authors do not manually audit a sample of the mutants that passed validation to measure how often the LLM-generated tests actually fail to discriminate between a seed and a non-equivalent mutant. The claim that 68.86% of generated mutants are valid should be supported by a ground-truth audit, ideally stratified by mutation operator and by LLM, with precision and recall of the equivalence oracle.
  4. [Tables 3 and 4] The relationship between the per-model detections in Table 3 and the unique-bug totals in Table 4 is not explained. Several rows in Table 3 have very high false-positive rates (e.g., DeepSeek on SpotBugs Type1: 35 detections with 5 verified, 85.71% FP; Codestral on SonarQube Type2: 21 detections with 2 verified, 90.48% FP), yet Table 4 reports 22 unique Type1 and 6 Type2 bugs for SpotBugs and 18 unique bugs for SonarQube. Please clarify how per-model detections are deduplicated across models, how 'same root cause' is judged, and how many unique verified bugs each model contributes. This is essential for understanding whether the 64 count is a robust aggregate or is driven by a handful of high-precision model-tool combinations.
minor comments (6)
  1. [Section 3.3, Listing 2] The example is actually valid: the Java hashCode of 'polygenelubricants' is Integer.MIN_VALUE, so the assertion result >= 0 fails as the text claims. I recommend adding a short comment or footnote to the listing stating this fact, because it is not obvious to many readers.
  2. [Table 1 and Section 3.4] The mutation-operator numbering is inconsistent: Table 1 lists Op2 as 'duplication' and Op3 as 'unreachable if,' while Section 3.4 describes Op2 as 'Obfuscating' and Op3 as 'Duplication.' This should be reconciled.
  3. [Table 1] The caption says 'ten mutation operators' but the table lists nine; the text in Section 3.4 also says nine. Correct the caption.
  4. [Section 7] The threat-to-validity paragraph says 'we examine our approach on three widely adopted static analyzers,' but the study evaluates five analyzers. This appears to be a leftover from an earlier draft and should be corrected.
  5. [Table 3] The '#Tests' column for CodeLlama on SonarQube is 88 while '#Comp Seeds' is 65; please clarify whether multiple tests are counted per seed and what the unit of the '#Tests' column is.
  6. [Figure 3] Figure 3 reports aggregate accuracy values for 'valid seeds' but does not state the sample size or a per-analyzer breakdown; please include these details in the caption or surrounding text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the 64-bug claim is an empirical, externally checked result; the test-based equivalence check is a validity threat, not a self-referential reduction.

full rationale

StaAgent's pipeline does not fit parameters to the data it later reports as predictions. Seed programs are generated from rule descriptions, mutants are produced by nine fixed and explicitly enumerated operators, and the analyzer's behavior on seeds versus mutants is an independent measurement. The semantic-equivalence gate in Section 3.4 ('If the test results, i.e., output, exceptions, and execution traces, are identical, the mutant is considered a valid semantically equivalent variant') is the only place where the framework's own generated artifacts validate one another. This is a weak oracle and a genuine threat to validity -- LLM-generated tests may not separate equivalent from non-equivalent mutants -- but it is not circularity in the derivation sense: the analyzer verdicts are not used to define equivalence, and the final 64-bug count is further filtered by manual verification by two authors and by developer responses (two fixed, three confirmed). The mutation operators are reused from prior work including one self-citation (Wei et al. [42] and Statfier [46]), but they are fully specified in Table 1 and their use does not presuppose the target result. The comparison against Statfier on the same analyzer versions is an external baseline. The 'polygenelubricants' example in Listing 2 appears factually wrong (that string has Java hashCode 0, so the assertion result >= 0 would pass), which undermines that illustrative validation but is an error in an example, not a circular step. Overall, no load-bearing reduction of the reported bugs to the framework's own inputs was found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on no fitted numeric parameters. The listed hyperparameters affect pipeline throughput but were not tuned to match target bug counts. The key non-empirical assumptions are the test-based equivalence criterion for mutants and the reliability of manual validation.

free parameters (4)
  • Generation temperature = 0.75
    Chosen by hand for code generation tasks; affects diversity of seeds and mutants.
  • Validation temperature = 0.1
    Chosen by hand to make LLM judgments deterministic; affects whether seeds and mutants are retained.
  • Refinement iteration cap = 5
    Maximum feedback loops for compiling code and tests before discarding; affects how many seeds survive the pipeline.
  • Mutants per operator = 3
    Number of mutants generated per seed per operator; directly affects the volume of metamorphic comparisons and hence the number of bugs exposed.
assumptions (4)
  • domain assumption A mutant that passes the same tests with identical outputs is semantically equivalent to the seed.
    Used in Section 3.4 to retain mutants for metamorphic testing; if false, Type1 and Type2 alarms can be false positives.
  • domain assumption Rule documentation for each static analyzer accurately describes intended bug detection behavior.
    Seed generation prompts and manual bug validation rely on rule descriptions from official documentation in Sections 3.1 and 5.4.
  • domain assumption Manual review by two authors is a reliable ground truth for whether a detected issue is a true bug.
    Used to filter LLM-reported bugs into the final 64; no inter-rater agreement metric or independent replication is provided in Section 5.4.
  • domain assumption The selected subset of rules, covering correctness, security, and performance but excluding style, framework, and test-code rules, is representative enough to support cross-analyzer conclusions.
    Selection criteria in Section 3.1 affect which rules are tested and therefore the bug counts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StaAgent: An Agentic Framework for Testing Static Analyzers." pith.science (2026). https://pith.science/paper/FMRXJ5B3

@misc{pith2026250715892,
  author       = {Pith},
  title        = {Pith review of: StaAgent: An Agentic Framework for Testing Static Analyzers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FMRXJ5B3}},
  note         = {Machine review of arXiv:2507.15892}
}
read the original abstract

Static analyzers play a critical role in identifying bugs early in the software development lifecycle, but their rule implementations are often under-tested and prone to inconsistencies. To address this, we propose StaAgent, an agentic framework that harnesses the generative capabilities of Large Language Models (LLMs) to systematically evaluate static analyzer rules. StaAgent comprises four specialized agents: a Seed Generation Agent that translates bug detection rules into concrete, bug-inducing seed programs; a Code Validation Agent that ensures the correctness of these seeds; a Mutation Generation Agent that produces semantically equivalent mutants; and an Analyzer Evaluation Agent that performs metamorphic testing by comparing the static analyzer's behavior on seeds and their corresponding mutants. By revealing inconsistent behaviors, StaAgent helps uncover flaws in rule implementations. This LLM-driven, multi-agent framework offers a scalable and adaptable solution to improve the reliability of static analyzers. We evaluated StaAgent with five state-of-the-art LLMs (CodeL-lama, DeepSeek, Codestral, Qwen, and GPT-4o) across five widely used static analyzers (SpotBugs, SonarQube, ErrorProne, Infer, and PMD). The experimental results show that our approach can help reveal 64 problematic rules in the latest versions of these five static analyzers (i.e., 28 in SpotBugs, 18 in SonarQube, 6 in ErrorProne, 4 in Infer, and 8 in PMD). In addition, 53 out of the 64 bugs cannot be detected by the SOTA baseline. We have reported all the bugs to developers, with two of them already fixed. Three more have been confirmed by developers, while the rest are awaiting response. These results demonstrate the effectiveness of our approach and underscore the promise of agentic, LLM-driven data synthesis to advance software engineering.

Figures

Figures reproduced from arXiv: 2507.15892 by the authors.

Figure 1
Figure 1. The workflow of StaAgent Generation Agent, which generates bug-inducing seed programs (Section 3.2). Next, the Code Validation Agent leverages an LLM to create tests that trigger the seed bugs and verifies whether the detected bugs align with their respective rule descriptions, based on both the seed code and test outcomes (Section 3.3). The Muta￾tion Generation Agent then uses the LLM to produce semantically equiva… view at source ↗
Figure 2
Figure 2. An example bug detection rule from Spotbugs [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of the “Valid Seeds” generated by each [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: The number of Type1 bugs across the analyzers detected by applying each mutation operator. StaAgent identified 43 Type1 and 21 Type2 faulty rule imple￾mentation bugs in the five tools. Codestral, Qwen2.5, and GPT￾4o showed strong performance, while DeepSeek and CodeLla…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 33 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  3. [3]

    QiHong Chen, Jiawei Li, Jiecheng Deng, Jiachen Yu, Justin Tian Jin Chen, and Iftekhar Ahmed. 2024. A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why? arXiv preprint arXiv:2411.01414 (2024)

    arXiv 2024

  4. [4]

    John Joon Young Chung, Ece Kamar, and Saleema Amershi. 2023. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprint arXiv:2306.04140 (2023)

    arXiv 2023

  5. [5]

    Roland Croft, Dominic Newlands, Ziyu Chen, and M Ali Babar. 2021. An empirical study of rule-based and learning-based approaches for static application security testing. In Proceedings of the 15th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM) . 1–12

    work page 2021

  6. [6]

    Pascal Cuoq, Benjamin Monate, Anne Pacalet, Virgile Prevosto, John Regehr, Boris Yakobowski, and Xuejun Yang. 2012. Testing static analyzers with randomly generated programs. In NASA Formal Methods Symposium . Springer, 120–125

    work page 2012

  7. [7]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shu- vendu K Lahiri. 2024. Llm-based test-driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering (2024)

    work page 2024

  8. [8]

    Markus Fleischmann, David Kaindlstorfer, Anastasia Isychev, Valentin Wüstholz, and Maria Christakis. 2024. Constraint-based test oracles for program analyz- ers. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 344–355

    work page 2024

Show all 50 references
  1. [9]

    Joseph Gatto, Parker Seegmiller, Timothy E Burdick, and Sarah Masud Preum

  2. [10]

    Asem Ghaleb and Karthik Pattabiraman. 2020. How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection. In Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis. 415–427

    2020

  3. [11]

    Qiuhan Gu. 2023. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 2201–2203

    2023

  4. [12]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196 (2024)

    2024 arXiv

  5. [13]

    Weigang He, Peng Di, Mengli Ming, Chengyu Zhang, Ting Su, Shijie Li, and Yulei Sui. 2024. Finding and understanding defects in static analyzers by constructing automated oracles. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1656–1678

    2024

  6. [14]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-Coder Technical Report. arXiv preprint arXiv:2409.12186 (2024)

    2024 arXiv

  7. [15]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  8. [16]

    Sathvik Joel, Jie JW Wu, and Fatemeh H Fard. 2024. A survey on llm-based code generation for low-resource and domain-specific programming languages. arXiv preprint arXiv:2410.03981 (2024)

    2024

  9. [17]

    David Kaindlstorfer, Anastasia Isychev, Valentin Wüstholz, and Maria Christakis

  10. [18]

    Christian Klinger, Maria Christakis, and Valentin Wüstholz. 2019. Differentially testing soundness and precision of program analyzers. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis . 239–250

    2019

  11. [19]

    In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering

    Interrogation testing of program analyzers for soundness and precision issues. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 319–330

  12. [20]

    Zongjie Li, Zhibo Liu, Wai Kin Wong, Pingchuan Ma, and Shuai Wang. 2024. Evaluating C/C++ vulnerability detectability of query-based static application security testing tools. IEEE Transactions on Dependable and Secure Computing 21, 5 (2024), 4600–4618

    2024

  13. [21]

    Haonan Li, Yu Hao, Yizhuo Zhai, and Zhiyun Qian. 2024. Enhancing static analysis for practical bug detection: An llm-integrated approach. Proceedings of the ACM on Programming Languages 8, OOPSLA1 (2024), 474–499

    2024

  14. [22]

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126 (2024)

    2024 arXiv

  15. [23]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2023), 21558–21572

    2023

  16. [24]

    Mohammad Mahdi Mohajer, Reem Aleithan, Nima Shiri Harzevili, Moshi Wei, Alvine Boaye Belle, Hung Viet Pham, and Song Wang. 2024. Effectiveness of chat- gpt for static analysis: How far are we?. InProceedings of the 1st ACM International Conference on AI-Powered Software. 151–160

    2024

  17. [25]

    Noble Saji Mathews and Meiyappan Nagappan. 2024. Test-Driven Develop- ment and LLM-based Code Generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering . 1583–1594

    2024

  18. [26]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2332–2354

    2024

  19. [27]

    Austin Mordahl. 2023. Automatic testing and benchmarking for configurable static analysis tools. In Proceedings of the 32nd ACM SIGSOFT International Sym- posium on Software Testing and Analysis . 1532–1536

    2023

  20. [28]

    Paulo Nunes, Ibéria Medeiros, José C Fonseca, Nuno Neves, Miguel Correia, and Marco Vieira. 2018. Benchmarking static analysis tools for web security. IEEE Transactions on Reliability 67, 3 (2018), 1159–1175

    2018

  21. [29]

    Marcus Nachtigall, Michael Schlichtig, and Eric Bodden. 2022. A large-scale study of usability criteria addressed by static analysis tools. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis . 532–543

    2022

  22. [30]

    Xavier Rival and Kwangkeun Yi. 2020. Introduction to static analysis

    2020

  23. [31]

    Sajeda Parveen and Manar H Alalfi. 2020. A mutation framework for evaluat- ing security analysis tools in IoT applications. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 587–591

    2020

  24. [32]

    Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang, Qipeng Guo, et al. 2025. Case2Code: Scalable Synthetic Data for Code Generation. In Proceedings of the 31st International Conference on Computational Linguistics . 11056–11069

    2025

  25. [33]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  26. [34]

    Florian Tambon, Arghavan Moradi-Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Giuliano Antoniol. 2025. Bugs in large language models generated code: An empirical study. Empirical Software Engineering 30, 3 (2025), 1–48

    2025

  27. [35]

    Boya Sun, Gang Shu, Andy Podgurski, and Brian Robinson. 2012. Extending static analysis by mining project-specific rules. In 2012 34th International Conference on Software Engineering (ICSE) . IEEE, 1054–1063

    2012

  28. [36]

    Tyler W Thomas, Heather Lipford, Bill Chu, Justin Smith, and Emerson Murphy- Hill. 2016. What questions remain? an examination of how developers understand an interactive static analysis tool. In Twelfth Symposium on Usable Privacy and Security (SOUPS 2016)

    2016

  29. [37]

    Jubi Taneja, Zhengyang Liu, and John Regehr. 2020. Testing static analyses for precision and soundness. In Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization . 81–93

    2020

  30. [38]

    Harishankar Vishwanathan, Matan Shachnai, Srinivas Narayana, and Santosh Nagarakatte. 2022. Sound, precise, and fast abstract interpretation with tristate numbers. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). IEEE, 254–265

    2022

  31. [39]

    Zhao Tian, Honglin Shu, Dong Wang, Xuejie Cao, Yasutaka Kamei, and Junjie Chen. 2024. Large Language Models for Equivalent Mutant Detection: How Far Are We?. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1733–1745. Conference’...

    2024

  32. [40]

    Junjie Wang, Yuchao Huang, Song Wang, and Qing Wang. 2022. Find bugs in static bug finders. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension. 516–527

    2022

  33. [41]

    Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. 2024. Planning in natural language improves llm search for code generation. arXiv preprint arXiv:2409.03733 (2024)

    2024 arXiv

  34. [42]

    Moshi Wei, Yuchao Huang, Jinqiu Yang, Junjie Wang, and Song Wang. 2022. Cocofuzzing: Testing neural code models with coverage-guided fuzzing. IEEE Transactions on Reliability 72, 3 (2022), 1276–1289

    2022

  35. [43]

    Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, and Hao Wang. 2025. HARMONIC: Harnessing LLMs for tabular data synthesis and privacy protection. Advances in Neural Information Processing Systems 37 (2025), 100196–100212

    2025

  36. [44]

    Ting Wu, Xuefeng Li, and Pengfei Liu. 2024. Progress or regress? self- improvement reversal in post-training. arXiv preprint arXiv:2407.05013 (2024)

    2024 arXiv

  37. [45]

    Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. 2023. LLM- powered data augmentation for enhanced cross-lingual performance. arXiv preprint arXiv:2305.14288 (2023)

    2023 arXiv

  38. [46]

    Huaien Zhang, Yu Pei, Junjie Chen, and Shin Hwei Tan. 2023. Statfier: Automated testing of static analyzers via semantic-preserving program transformations. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Softwa...

    2023

  39. [47]

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4all: Universal fuzzing with large language models. In Pro- ceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

  40. [48]

    Huaien Zhang, Yu Pei, Shuyun Liang, Zezhong Xing, and Shin Hwei Tan. 2024. Characterizing and Detecting Program Representation Faults of Static Analysis Frameworks. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1772–1784. Rece...

    2024

  41. [49]

    Huaien Zhang, Yu Pei, Shuyun Liang, and Shin Hwei Tan. 2024. Understanding and detecting annotation-induced faults of static analyzers. Proceedings of the ACM on Software Engineering 1, FSE (2024), 722–744

    2024

  42. [2024]

    arXiv preprint arXiv:2411.06549 (2024)

    In-context learning for preserving patient privacy: A framework for synthe- sizing realistic patient portal messages. arXiv preprint arXiv:2411.06549 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.