Pith. sign in

REVIEW 3 major objections 5 minor 77 references

LAMeD: LLM-generated Annotations for Memory Leak Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that LLM-generated annotations for allocation and deallocation functions, fed into static analyzers, double the number of known memory leaks found in seven real C projects (from 5 to 10 for both CodeQL and Cooddy), at the…

desk verdict Useful proof of concept, but the evaluation doesn't isolate LLM quality from annotation volume and the abstract's path-explosion claim contradicts the warning counts. read the letter →

arxiv 2505.02376 v1 pith:D47KTBEM submitted 2025-05-05 cs.SE

Ekaterina Shemetova , Ilya Shenbin , Ivan Smirnov , Anton Alekseev , Alexey Rukhovich , Sergey Nikolenko , Vadim Lomshakov , Irina Piontkovskaya This is my paper · ORCID
classification cs.SE
keywords memoryleakdetectionstaticanalysisLLM-generatedannotationsfunctiondataflowpathexplosionC/C++codezero-shotprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LAMeD asks a large language model to look at each function in a C codebase, with its callees as context, and say which variables receive newly allocated memory and which receive freed memory. Those answers are converted into the annotation format that static analyzers like CodeQL and Cooddy use to track allocation and deallocation, replacing manually written hints. On a manually labeled cJSON benchmark, the configuration chosen for the main runs had precision 0.933 and recall 0.583, and on 43 known leaks across seven C projects it doubled the number of target bugs each analyzer found, from 5 to 10. The cost is a large increase in total warnings: CodeQL went from 139 to 653 and Cooddy from 86 to 391, so the gain in detections comes with more noise for an analyst to sort through. If this holds up, it means zero-shot LLM annotation can substitute for expert-curated analyzer configuration as a first pass on unfamiliar codebases.

What carries the argument

The load-bearing mechanism is the two-stage LLM prompt with call-graph context plus post-filtering, followed by conversion into the analyzer's annotation format. The first prompt extracts allocated_variables and deallocated_variables from a function's source; the second asks whether a returned value points into a structure passed by argument, filtering out cases where allocation does not produce an independently owned pointer. A call-graph and source extraction engine supplies the callee context that lets the model recognize custom allocators such as solv_chksum_create, which a name-and-signature heuristic misses. The converted annotations become AllocSource and FreeSink entries for Cooddy or custom memory models for CodeQL; that interface is where the LLM's semantic guesses enter the formal data-flow machinery.

What would settle it

Instrument every allocation site in the seven test projects with runtime ownership tracing, then compare the LLM's per-function labels against observed ownership; if a large share of generated AllocSource annotations point to functions whose returned memory is meant to be embedded and freed with its container, the warning surge is mostly noise and the detection gain cannot be attributed to correct annotations.

Watch

Extended reading notes

Core claim

The central result is that the generated annotations change what the analyzers see: functions like solv_chksum_create in libsolv, which allocate through a custom wrapper rather than a direct malloc, become visible allocation sources once the LLM names them, and the analyzer then follows the allocated pointer to the missing free. LAMeD found five additional target bugs per analyzer over the unannotated baseline across different projects, while the annotation sets were not a subset of the analyzers' own heuristics: CodeQL's name-and-signature heuristic caught low-level wrappers like cJSON_malloc that LAMeD missed, while LAMeD caught context-dependent allocators the heuristic cannot see. The paper also reports that post-filtering raises annotation precision substantially, that larger models are not automatically better, and that no leak was found in libxml2 despite a large annotation set.

Load-bearing premise

The result holds only if the LLM's zero-shot judgments about which functions allocate or free memory line up with the analyzers' conservative ownership semantics; if those notions drift apart, the extra annotations produce false positives instead of detections.

Editorial extensions

If this is right

  • Static analyzers can be pointed at a new C codebase and, without manual annotation, recover leaks they would otherwise miss, roughly doubling known-bug recall on the tested projects.
  • The warning count will often grow in proportion to the number of annotations, so teams adopting this approach should budget for triage or add a precision filter before treating warnings as findings.
  • Annotation sets can be regenerated after major code changes and reused across many analysis runs, so the LLM cost is paid once per codebase revision rather than per run.
  • Because LAMeD and the analyzers' name-based heuristics find different allocation functions, the two are complementary, and combining them should give better coverage than either alone.
  • The same two-stage annotation generation could be applied to other bug classes where analyzers need source and sink specifications, such as buffer overflows or race conditions, as the paper suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct check of the transfer claim would be to trace every allocation site in the seven projects at runtime and compare the LLM's per-function labels against observed ownership; the paper validates labels only on cJSON, so transferability is an open assumption.
  • The warning surge may itself be usable as a signal: newly fired warnings point at functions whose ownership semantics the LLM labeled incorrectly, which could be mined to build a self-correcting annotation loop.
  • If low-level wrappers like cJSON_malloc are systematically missed because they return a pointer without storing it in a local variable, a cheap remedy is to run name-and-signature heuristics alongside the LLM and merge the annotation sets, since the paper shows the two are partly disjoint.
  • One could test whether few-shot prompting with a handful of verified annotations from the target project improves precision enough to tame the warning explosion; the paper deliberately sticks to zero-shot to keep cross-project comparisons clean.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces LAMeD, a pipeline that uses zero-shot LLM prompting (with a two-step prompt and post-filtering) to generate function-level annotations marking allocation and deallocation points in C/C++ code, then feeds these annotations to static analyzers CodeQL and Cooddy for memory leak detection. The authors evaluate annotation quality on a manually labeled cJSON benchmark (three LLMs, precision/recall) and then apply the best configuration (Codestral with post-filtering) to a real-life dataset of 43 known memory leaks across seven C projects. On this dataset, both CodeQL and Cooddy increase the number of target bugs found from 5 to 10 when LAMeD annotations are added, but the total number of analyzer warnings rises sharply (CodeQL 139 to 653; Cooddy 86 to 391). The paper claims that LLM-generated annotations improve real-life memory leak detection and, in the abstract, that they reduce path explosion.

Significance. If the central claim is established, LAMeD would be a useful proof of concept for replacing manual annotation of memory allocation/deallocation functions, a known bottleneck in static analysis. The paper has notable strengths: the pipeline is zero-shot and does not require per-project fine-tuning; the evaluation covers two different static analyzers; the cJSON ground truth is manually labeled; and the authors plan to release prompts and datasets. The reported doubling of found target bugs (5 to 10 for both analyzers) is a concrete, falsifiable result. However, as discussed below, the experimental design does not currently separate the effect of LLM semantic precision from the effect of simply adding many annotations, and the abstract's path-explosion claim is unsupported by any measurement. These issues are load-bearing for the paper's main claim and require revision.

major comments (3)
  1. [Section 5.3, Table 4] The comparison is confounded by annotation volume. The baseline uses no custom annotations (CodeQL still uses its built-in HeuristicAllocationFunctionByName), while LAMeD adds 375 annotations for CodeQL and 471 for Cooddy across the dataset. There is no control condition that adds a comparably sized set of annotations from simple non-LLM heuristics, such as marking every function whose body calls malloc/calloc/realloc/strdup and returns the pointer, or every function that calls free/delete on a parameter. The observed increase from 5 to 10 found bugs may therefore be due to the number of added allocation/deallocation points rather than to the semantic quality of LLM-generated annotations. The paper's central claim (Section 1) is that LLM-generated annotations improve detection; a control matched on annotation count is necessary to support this specific claim rather than the weaker claim that any broad annotation set helps.
  2. [Abstract and Section 2.1] The claim that LAMeD 'reduces path explosion' is not operationalized or measured anywhere in the paper. Section 5.3 reports a 4.7x increase in CodeQL warnings (139 to 653) and a 4.5x increase for Cooddy (86 to 391), which is at least in tension with the notion of reduced analysis complexity. No proxy for path explosion is reported, such as analysis runtime, number of explored states, or number of unrolled paths. Either provide direct evidence (e.g., wall-clock analysis time before and after annotation insertion) or remove the 'reduces path explosion' claim from the abstract and introductory motivation, since the current data do not support it.
  3. [Section 5.2, Table 2 and Section 5.3] Annotation quality is validated only on the cJSON library (152 functions), and the chosen configuration (Codestral with post-filtering, precision 0.933) is then applied to the other six projects without per-project validation of the generated annotations against actual allocation/deallocation behavior. The paper itself notes in Section 5.3 that some annotation sets are more effective than others and that warning counts vary widely. Because the real-life dataset contains known leak sites, the authors could strengthen the causal chain by sampling the generated annotations per project and estimating precision/recall there, or at least analyzing how annotation errors correlate with the increase in warnings. As written, the connection between LLM semantic accuracy and the detection improvement is assumed rather than demonstrated for most projects. Relatedly, the post-filtering step (Listing 2) relies on the LLM correctly distinguishing allocation to a new variable from assignment to an existing structure; this assumption is not validated independently.
minor comments (5)
  1. [Section 5.1.2] The text says the five authors labeled '30+ methods' of cJSON, but the dataset subsequently comprises 152 functions. Please clarify what the '30+' refers to (e.g., ambiguous cases, functions with nontrivial annotations) to avoid confusion.
  2. [Table 4 caption] The average annotation counts differ between CodeQL and Cooddy for the same projects (e.g., curl: 117 vs 137; libsolv: 88 vs 103). This is explained in Section 5.3 as an effect of removing annotations without the returned-value property for CodeQL, but adding a note to the caption would make the table self-contained.
  3. [Section 5.3] The statement 'it is easy to see a direct relationship between the number of annotations and number of warnings' would be more convincing with a correlation coefficient or a small scatter plot; as written, the relationship is qualitative.
  4. [Throughout] Naming is inconsistent: 'cjson' and 'cJSON', 'Codestral' and 'codestral' appear in different places. Please standardize.
  5. [Section 5.2.1] The sentence 'The model with the largest number of parameters, DeepSeek-R1-70B, did not provide the better results in comparison' is grammatically awkward; consider rephrasing to 'did not provide the best results among the tested models'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LAMeD's detection gains are measured by running unchanged analyzers with independently generated LLM annotations, not by construction from fitted inputs.

full rationale

I found no circular derivation. The LLM-generated annotations are produced by zero-shot prompts (Listings 1-2) from function source code and call-graph context, and the target bug labels (fixed memory leaks taken from GitHub commits) are not inputs to the prompt design, model selection, or post-filtering procedure. The model choice (Codestral with post-filtering) was made on the separate manually labeled cJSON annotation benchmark (Table 2), and the same configuration was then applied to all seven real-life projects; cJSON appears in both stages, but the real-life cjson target bugs are leak fixes rather than annotation labels, so this is a potential selection-overlap concern rather than a reduction of the detection result to a fitted parameter. The detection improvements in Table 4 are measured by running CodeQL and Cooddy with and without the generated annotations; nothing in the paper's equations makes the number of found bugs equal the annotation set or the annotation precision by construction. The paper explicitly acknowledges the warning-growth tradeoff and the 'direct relationship between the number of annotations and number of warnings,' so the annotation-volume confound is disclosed as a limitation rather than relabeled as a prediction. No load-bearing self-citation or imported uniqueness theorem appears; the Cooddy documentation citation [28] is to external prior work by different authors. Overall, the central claim is an empirical outcome of an independent experimental pipeline, not an artifact of definitional circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central result rests on tool behavior assumptions (annotation formats), dataset assumptions (author-created ground truth and manually curated target bugs), and the ad hoc post-filtering prompt. No numeric free parameters are fitted, and no invented entities are introduced.

assumptions (4)
  • domain assumption Cooddy and CodeQL accept external annotations as described and their analysis engines process them as expected.
    The entire evaluation depends on the annotation formats (AllocSource/FreeSink for Cooddy, custom CodeQL models) functioning as documented by the tools. Section 5.1.1.
  • domain assumption The manually annotated cJSON labels produced by five authors and resolved by one expert are correct and complete.
    Used as ground truth for annotation quality (Table 2). The label set is the benchmark for precision/recall; no inter-annotator agreement is reported. Section 5.1.2.
  • domain assumption The target bugs selected from DiverseVul are true memory leaks and the parent commits are valid leak-containing states.
    The real-life dataset is manually curated; the 'found' metric relies on manual confirmation in target files. Section 5.1.2.
  • ad hoc to paper The post-filtering second prompt (Listing 2) correctly distinguishes allocation to a new variable from assignment to an existing structure.
    This validation step is introduced by the authors and applied to reduce false positive annotations; its correctness is only assessed indirectly via cJSON precision/recall. Section 3, Listing 2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LAMeD: LLM-generated Annotations for Memory Leak Detection." pith.science (2026). https://pith.science/paper/D47KTBEM

@misc{pith2026250502376,
  author       = {Pith},
  title        = {Pith review of: LAMeD: LLM-generated Annotations for Memory Leak Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D47KTBEM}},
  note         = {Machine review of arXiv:2505.02376}
}
read the original abstract

Static analysis tools are widely used to detect software bugs and vulnerabilities but often struggle with scalability and efficiency in complex codebases. Traditional approaches rely on manually crafted annotations -- labeling functions as sources or sinks -- to track data flows, e.g., ensuring that allocated memory is eventually freed, and code analysis tools such as CodeQL, Infer, or Cooddy can use function specifications, but manual annotation is laborious and error-prone, especially for large or third-party libraries. We present LAMeD (LLM-generated Annotations for Memory leak Detection), a novel approach that leverages large language models (LLMs) to automatically generate function-specific annotations. When integrated with analyzers such as Cooddy, LAMeD significantly improves memory leak detection and reduces path explosion. We also suggest directions for extending LAMeD to broader code analysis.

Figures

Figures reproduced from arXiv: 2505.02376 by the authors.

Figure 1
Figure 1. Annotating an “operator delete[](void*, std:: nothrow_t const&)” function (from Cooddy docs; FreeSink means freeing a variable with a free function of “type 3”). need for heuristics that vary across codebases. For example, CodeQL marks a function as allocating based on its name and signature, missing many functions. Kernel memory leak detector K-Meld [23] uses kernel-specific heuristics and context-aware rule mining… view at source ↗
Figure 2
Figure 2. The proposed LLM-enhanced memory leak detection pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 34 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Mistral AI. 2024. Codestral-22B-v0.1: A 22-Billion-Parameter Model for Code Generation. https://huggingface.co/mistralai/Codestral-22B-v0.1 Accessed: 2025-03-23

    work page 2024

  3. [3]

    Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohammad- masiha Zahedivafa, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2024. Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning. arXiv:2307.02179 [cs.CL] https://arxiv.org/abs/ 2307.02179

    arXiv 2024

  4. [4]

    Abhinav Anand, Shweta Verma, Krishna Narasimhan, and Mira Mezini. 2024. A Critical Study of What Code-LLMs (Do Not) Learn. arXiv:2406.11930 [cs.SE] https://arxiv.org/abs/2406.11930

    work page Pith review arXiv 2024

  5. [5]

    Hayk Aslanyan, Hovhannes Movsisyan, Hripsime Hovhannisyan, Zhora Gevor- gyan, Ruslan Mkoyan, Arutyun Avetisyan, and Sevak Sargsyan. 2024. Combining Static Analysis with Directed Symbolic Execution for Scalable and Accurate Memory Leak Detection. IEEE Access (2024)

    work page 2024

  6. [6]

    Sam Blackshear and PW O’Hearn. 2017. Open-sourcing RacerD: fast static race detection at scale. code.facebook.com blog post (2017)

    work page 2017

  7. [7]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020). 30https://zenodo.org/communities/llm-4-lsr

    arXiv 2020

  8. [8]

    Derek Bruening and Qin Zhao. 2011. Practical memory checking with Dr. Memory. In International Symposium on Code Generation and Optimization (CGO 2011) . IEEE, 213–223

    work page 2011

Show all 77 references
  1. [9]

    Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.. In OSDI, Vol. 8. 209–224

    2008

  2. [10]

    Cristiano Calcagno and Dino Distefano. 2011. Infer: An Automatic Program Verifier for Memory Safety of C Programs. In NASA Formal Methods , Mihaela Bobaru, Klaus Havelund, Gerard J. Holzmann, and Rajeev Joshi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 459–465

    2011

  3. [11]

    Cristiano Calcagno, Dino Distefano, Peter O’Hearn, and Hongseok Yang. 2009. Compositional shape analysis by means of bi-abduction. In Proceedings of the 36th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages. 289–300

    2009

  4. [12]

    Cristiano Calcagno, Dino Distefano, and Peter O’Hearn. 2015. Open-sourcing Facebook Infer: Identify bugs before you ship. code. facebook. com blog post 11 (2015)

    2015

  5. [13]

    O’Hearn, and Hongseok Yang

    Cristiano Calcagno, Dino Distefano, Peter W. O’Hearn, and Hongseok Yang. 2011. Compositional Shape Analysis by Means of Bi-Abduction. J. ACM 58, 6, Article 26 (dec 2011), 66 pages. https://doi.org/10.1145/2049697.2049700

    2011

  6. [14]

    Patrick J Chapman, Cindy Rubio-González, and Aditya V Thakur. 2024. Interleav- ing Static Analysis and LLM Prompting. In Proceedings of the 13th ACM SIGPLAN International Workshop on the State Of the Art in Program Analysis . 9–17

    2024

  7. [15]

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner

  8. [16]

    Yuxiao Chen, Jingzheng Wu, Xiang Ling, Changjiang Li, Zhiqing Rui, Tianyue Luo, and Yanjun Wu. 2024. When Large Language Models Confront Repository- Level Automatic Program Repair: How Well They Done?. InProceedings of the 2024 IEEE/ACM 46th International Conference on Softwar...

    2024

  9. [17]

    Clang Team. 2024. Clang Static Analyzer. LLVM Project. https://clang-analyzer. llvm.org/ Accessed: 2024-08-23

    2024

  10. [18]

    Dennis de Champeaux, Hermann Kaindl, Joachim Laubsch, and Albert Schap- pert. 1994. Artificial intelligence for object-oriented software engineering. In Addendum to the proceedings on Object-oriented programming systems, languages, and applications (Addendum). 127–130

    1994

  11. [19]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2019

  12. [20]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624 (2024)

    2024 arXiv

  13. [21]

    Dino Distefano, Manuel Fähndrich, Francesco Logozzo, and Peter W O’Hearn

  14. [22]

    Alex Halderman

    Zakir Durumeric, Frank Li, James Kasten, Johanna Amann, Jethro Beekman, Mathias Payer, Nicolas Weaver, David Adrian, Vern Paxson, Michael Bailey, and J. Alex Halderman. 2014. The Matter of Heartbleed. In Proceedings of the 2014 Conference on Internet Measurement Conference (Va...

    2014

  15. [23]

    Navid Emamdoost, Qiushi Wu, Kangjie Lu, and Stephen McCamant. 2021. Detect- ing kernel memory leaks in specialized modules with ownership reasoning. In The 2021 Annual Network and Distributed System Security Symposium (NDSS’21)

    2021

  16. [24]

    Gang Fan, Rongxin Wu, Qingkai Shi, Xiao Xiao, Jinguo Zhou, and Charles Zhang

  17. [25]

    Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Asmita, Ryan Tsang, Najmeh Nazari, Han Wang, and Houman Homayoun

  18. [26]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al . 2020. CodeBERT: A Pre- Trained Model for Programming and Natural Languages. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020...

    2020

  19. [27]

    In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

    Smoke: scalable path-sensitive memory leak detection for millions of lines of code. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 72–82

    2019

  20. [28]

    Alexander Yurievich Gerasimov, Alexey Alexeyevich Kanakhin, Petr Alekseevich Privalov, Andrey Alexandrovich Zhukov, and Evgenii Arkadievich Kaminskii

  21. [29]

    Seyed Mohammad Ghaffarian and Hamid Reza Shahriari. 2017. Software Vul- nerability Analysis and Discovery Using Machine-Learning and Data-Mining Techniques: A Survey. ACM Comput. Surv. 50, 4, Article 56 (aug 2017), 36 pages. https://doi.org/10.1145/3092566

    2017 doi

  22. [30]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  23. [31]

    Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. 2023. How far have we gone in vulnerability detection using large language models. arXiv preprint arXiv:2311.12420 (2023)

    2023 arXiv

  24. [32]

    Yu Hao, Weiteng Chen, Ziqiao Zhou, and Weidong Cui. 2023. E&V: Prompting Large Language Models to Perform Static Analysis by Pseudo-code Execution and Verification. CoRR abs/2312.08477 (2023). https://doi.org/10.48550/ARXIV. 2312.08477 arXiv:2312.08477

    2023 doi arXiv

  25. [33]

    Ben Hardekopf and Calvin Lin. 2011. Flow-sensitive pointer analysis for millions of lines of code. In Proceedings of the 9th Annual IEEE/ACM International Sympo- sium on Code Generation and Optimization (CGO ’11) . IEEE Computer Society, USA, 289–298

    2011

  26. [34]

    Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead. ACM Trans. Softw. Eng. Methodol. (Jan. 2025). https://doi.org/10.1145/3712003 Just Accepted

    2025 doi

  27. [35]

    Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. 2024. A deep dive into large language models for automated bug localization and repair. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1471–1493

    2024

  28. [36]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196 (2024)

    2024 arXiv

  29. [37]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024)

    2024 arXiv

  30. [38]

    joern.io. 2024. Joern: The Bug Hunter’s Workbench . https://github.com/joernio/ joern

    2024

  31. [39]

    Yungbum Jung and Kwangkeun Yi. 2008. Practical memory leak detector based on parameterized procedural summaries. In Proceedings of the 7th international symposium on Memory management . 131–140

    2008

  32. [40]

    Sungmin Kang, Gabin An, and Shin Yoo. 2024. A quantitative and qualitative evaluation of LLM-based explainable fault localization. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1424–1446

    2024

  33. [41]

    Binyuan Hui. 2023. An awesome and curated list of best code-LLM for research

    2023

  34. [42]

    Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. 2024. MEGAnno+: A Human-LLM Collaborative Annotation System. arXiv:2402.18050 [cs.CL] https://arxiv.org/abs/2402.18050

    2024 arXiv

  35. [43]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 (2023)

    2023 arXiv

  36. [44]

    Haonan Li, Yu Hao, Yizhuo Zhai, and Zhiyun Qian. 2024. Enhancing Static Analysis for Practical Bug Detection: An LLM-Integrated Approach. Proceedings of the ACM on Programming Languages 8, OOPSLA1 (2024), 474–499

    2024

  37. [45]

    Wen Li, Haipeng Cai, Yulei Sui, and David Manz. 2020. PCA: memory leak detection using partial call-path analysis. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1621–1625

    2020

  38. [46]

    Anant Kharkar, Roshanak Zilouchian Moghaddam, Matthew Jin, Xiaoyu Liu, Xin Shi, Colin Clement, and Neel Sundaresan. 2022. Learning to reduce false positives in analytic bug detectors. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Penn...

    2022

  39. [47]

    Yanling Luo, Jiawei Wan, and Shengqin She. 2022. Software Security Vulnerability Mining Based on Deep Learning. In Application of Intelligent Systems in Multi- modal Information Analytics, Vijayan Sugumaran, A. G. Sreedevi, and Zheng Xu (Eds.). Springer International Publishin...

    2022

  40. [48]

    Mohammad Mahdi Mohajer, Reem Aleithan, Nima Shiri Harzevili, Moshi Wei, Alvine Boaye Belle, Hung Viet Pham, and Song Wang. 2023. SkipAna- lyzer: An Embodied Agent for Code Analysis with Large Language Mod- els. CoRR abs/2310.18532 (2023). https://doi.org/10.48550/ARXIV.2310.18...

    2023 doi arXiv

  41. [49]

    László Nagy. 2025. Bear: A tool that generates a compilation database for clang tooling. https://github.com/rizsotto/Bear Accessed: 2025-03-23

    2025

  42. [50]

    Arbi Haza Nasution and Aytug Onan. 2024. ChatGPT Label: Comparing the Quality of Human-Generated and LLM-Generated Annotations in Low-resource Language NLP Tasks. IEEE Access (2024)

    2024

  43. [51]

    Ye Liu, Yue Xue, Daoyuan Wu, Yuqiang Sun, Yi Li, Miaolei Shi, and Yang Liu

  44. [52]

    arXiv preprint arXiv:2405.02580 (2024)

    PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generation. arXiv preprint arXiv:2405.02580 (2024)

    2024 arXiv

  45. [53]

    Felipe Pezoa, Juan L Reutter, Fernando Suarez, Martín Ugarte, and Domagoj Vrgoč. 2016. Foundations of JSON schema. In Proceedings of the 25th International Conference on World Wide Web . International World Wide Web Conferences Steering Committee, 263–273

    2016

  46. [54]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

    2019

  47. [55]

    John C Reynolds. 2002. Separation logic: A logic for shared mutable data struc- tures. In Proceedings 17th Annual IEEE Symposium on Logic in Computer Science . IEEE, 55–74

    2002

  48. [56]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  49. [57]

    Nicholas Nethercote and Julian Seward. 2007. Valgrind: a framework for heavy- weight dynamic binary instrumentation.ACM Sigplan notices42, 6 (2007), 89–100

    2007

  50. [58]

    Peter O’Hearn. 2019. Separation logic. Commun. ACM 62, 2 (2019), 86–95

    2019

  51. [59]

    Yulei Sui and Jingling Xue. 2016. SVF: interprocedural static value-flow analysis in LLVM. In Proceedings of the 25th international conference on compiler construction . 265–266

    2016

  52. [60]

    Yulei Sui, Ding Ye, and Jingling Xue. 2012. Static memory leak detection using full- sparse value-flow analysis. In Proceedings of the 2012 International Symposium on Software Testing and Analysis (Minneapolis, MN, USA) (ISSTA 2012). Association for Computing Machinery, New Yo...

    2012

  53. [61]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Miaolei Shi, and Yang Liu. 2024. LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs’ Vulnerability Reasoning. arXiv preprint arXiv:2401.16185 (2024)

    2024 arXiv

  54. [62]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, ...

    2017

  55. [63]

    2012.{AddressSanitizer}: A fast address sanity checker

    Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. 2012.{AddressSanitizer}: A fast address sanity checker. In2012 USENIX annual technical conference (USENIX ATC 12) . 309–318

    2012

  56. [64]

    Yulei Sui, Peng Di, and Jingling Xue. 2016. Sparse flow-sensitive pointer analysis for multithreaded programs. In Proceedings of the 2016 International Symposium on Code Generation and Optimization (Barcelona, Spain) (CGO ’16). Association for Computing Machinery, New York, NY...

    2016

  57. [65]

    Jin Wang, Zishan Huang, Hengli Liu, Nianyi Yang, and Yinhao Xiao. 2023. Defec- thunter: A novel llm-driven boosted-conformer-based code vulnerability detec- tion mechanism. arXiv preprint arXiv:2309.15324 (2023)

    2023 arXiv

  58. [66]

    Jiwei Yan, Jinhao Huang, Chunrong Fang, Jun Yan, and Jian Zhang. 2024. Better Debugging: Combining Static Analysis and LLMs for Explainable Crashing Fault Localization. arXiv preprint arXiv:2408.12070 (2024)

    2024 arXiv

  59. [67]

    Ding Ye, Yulei Sui, and Jingling Xue. 2014. Accelerating Dynamic Detection of Uses of Undefined Values with Static Value-Flow Analysis. In Proceedings of Annual IEEE/ACM International Symposium on Code Generation and Optimization (Orlando, FL, USA) (CGO ’14). Association for C...

    2014

  60. [68]

    Kwangkeun Yi. 2017. Inferbo: Infer-based buffer overrun analyzer. Meta Research. Retrieved Feburary 6 (2017), 2023. https://research.facebook.com/blog/2017/2/ inferbo-infer-based-buffer-overrun-analyzer/

    2017

  61. [69]

    Nalin Wadhwa, Jui Pradhan, Atharv Sonwane, Surya Prakash Sahu, Nagarajan Natarajan, Aditya Kanade, Suresh Parthasarathy, and Sriram Rajamani. 2024. CORE: Resolving Code Quality Issues using LLMs. Proceedings of the ACM on Software Engineering 1, FSE (2024), 789–811

    2024

  62. [70]

    Chong Wang, Jianan Liu, Xin Peng, Yang Liu, and Yiling Lou. 2025. Boosting Static Resource Leak Detection via LLM-based Resource-Oriented Intention Inference. Proceedings of the 47th International Conference on Software Engineering (2025)

    2025

  63. [71]

    Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. 2024. Large language model for vulnerability detection and repair: Literature review and the road ahead.ACM Transactions on Software Engineering and Methodology (2024)

    2024

  64. [75]

    Jie Zhang, Haoyu Bu, Hui Wen, Yu Chen, Lun Li, and Hongsong Zhu. 2024. When llms meet cybersecurity: A systematic literature review. arXiv preprint arXiv:2405.03644 (2024)

    2024 arXiv

  65. [76]

    Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jian- guo Li, and Rui Wang. 2024. Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code. arXiv:2311.07989 [cs.CL] https://arxiv.org/abs/2311.07989

    2024 arXiv

  66. [2019]

    Scaling static analyses at Facebook. Commun. ACM 62, 8 (2019), 62–70

    2019

  67. [2022]

    Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS) 34, 4 (2022), 7–20

    Case study: Source code static analysis for performance issues detection. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS) 34, 4 (2022), 7–20. Conference’17, July 2017, Washington, DC, USA Shemetova et al

    2022

  68. [2023]

    In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses

    DiverseVul: A new vulnerable source code dataset for deep learning based vulnerability detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses . 654–668

  69. [2024]

    Large Language Models for Code Analysis: Do LLMs Really Do Their Job? arXiv:2310.12357 [cs.SE] https://arxiv.org/abs/2310.12357

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.