{"id":"5e38140d-20b1-41f0-a40b-2bc7473567a4","arxiv_id":"2505.02376","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-generated function annotations for C/C++ memory management improved CodeQL and Cooddy memory leak detection from 5 to 10 known bugs across 43 targets, at the cost of a large increase in warnings.","lead":"This paper presents LAMeD, a system that uses large language models to automatically write memory-allocation and deallocation annotations for C/C++ code, then feeds them into static analyzers. In tests on seven open-source projects, the annotated versions caught twice as many known memory leaks as the default analyzers, though they also produced far more warnings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Improvement may be an artifact of annotation volume, not LLM annotation quality; no non-LLM baseline controls for annotation count.","rationale":"The paper's strongest claim is that LLM-generated annotations increase real-life memory leak detections. The reported evidence is a before/after comparison with no control for the number or method of annotations. The reader correctly identifies the semantic-match premise as fragile, but the more fundamental threat is attribution: a twofold recall gain alongside a 4.7-fold warning increase could be produced by any annotation expansion, and the paper provides no comparison to rule-based annotation generation of comparable size. This does not mean the claim is false; the cJSON precision numbers and the concretely reproduced detections are real evidence. However, the central causal claim requires a control condition to distinguish LLM quality from annotation quantity. The proposed test is feasible using the existing dataset and analyzers and would settle whether the LLM is doing the work. The verdict remains conditional, consistent with the reader's assessment, but with the condition sharpened to require a non-LLM baseline.","tokens_in":17566,"tokens_out":4697,"duration_ms":57231,"concrete_test":"Construct a non-LLM baseline annotation generator using simple, deterministic rules: annotate as AllocSource any function whose body (transitively through callees) calls malloc, calloc, realloc, strdup, or aligned_alloc and returns a pointer-type value; annotate as FreeSink any function that calls free or delete on a pointer parameter or on a pointer returned by an annotated allocator. For each project in the real-life dataset, generate annotations with this generator, then tune it (e.g., also include functions with 'alloc' or 'create' in the name returning a pointer) so the per-project annotation count matches LAMeD's counts in Table 4. Run the identical CodeQL and Cooddy configurations used in Section 5.3 and compare (a) target bugs found and (b) warning counts against Table 4.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM-generated annotations improve real-life memory leak detection, but the experimental design does not control for the number of annotations added. In Table 4, LAMeD adds 24–137 annotations per project for CodeQL (total 375) and 10–137 for Cooddy (total 471), while the 'no annotations' baseline is the analyzers' default configuration with few or no custom annotations. The detection gain (CodeQL 5 -> 10, Cooddy 5 -> 10) is accompanied by a 4.7x and 4.5x increase in warnings (139 -> 653 and 86 -> 391). The paper itself notes a 'direct relationship between the number of annotations and number of warnings' (Section 5.3). Because no comparison is made against a similarly sized set of annotations produced by simple heuristics (e.g., functions calling malloc/calloc/realloc/strdup and returning a pointer, or free/delete on parameters), it is unknown whether the extra detections come from the LLM's semantic precision or merely from covering more functions as allocation/deallocation points. The cJSON annotation precision (0.933 for Codestral post-filtered) is measured on one small library and is not validated per project on the other six codebases; Table 3 shows LAMeD produces far more annotations than CodeQL's name/signature heuristic, and Section 5.3 reports that some annotation sets are more effective than others. Thus the causal claim 'LLM-generated annotations improve detection' is underdetermined: the same or better results might be achieved by any broad annotation expansion. This is an internal validity concern, not a disagreement with external consensus; the paper's conclusion goes beyond what the comparison supports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LAMeD, a pipeline that uses zero-shot LLM prompting (with a two-step prompt and post-filtering) to generate function-level annotations marking allocation and deallocation points in C/C++ code, then feeds these annotations to static analyzers CodeQL and Cooddy for memory leak detection. The authors evaluate annotation quality on a manually labeled cJSON benchmark (three LLMs, precision/recall) and then apply the best configuration (Codestral with post-filtering) to a real-life dataset of 43 known memory leaks across seven C projects. On this dataset, both CodeQL and Cooddy increase the number of target bugs found from 5 to 10 when LAMeD annotations are added, but the total number of analyzer warnings rises sharply (CodeQL 139 to 653; Cooddy 86 to 391). The paper claims that LLM-generated annotations improve real-life memory leak detection and, in the abstract, that they reduce path explosion.","tokens_in":17860,"tokens_out":4571,"duration_ms":55349,"significance":"If the central claim is established, LAMeD would be a useful proof of concept for replacing manual annotation of memory allocation/deallocation functions, a known bottleneck in static analysis. The paper has notable strengths: the pipeline is zero-shot and does not require per-project fine-tuning; the evaluation covers two different static analyzers; the cJSON ground truth is manually labeled; and the authors plan to release prompts and datasets. The reported doubling of found target bugs (5 to 10 for both analyzers) is a concrete, falsifiable result. However, as discussed below, the experimental design does not currently separate the effect of LLM semantic precision from the effect of simply adding many annotations, and the abstract's path-explosion claim is unsupported by any measurement. These issues are load-bearing for the paper's main claim and require revision.","major_comments":[{"comment":"The comparison is confounded by annotation volume. The baseline uses no custom annotations (CodeQL still uses its built-in HeuristicAllocationFunctionByName), while LAMeD adds 375 annotations for CodeQL and 471 for Cooddy across the dataset. There is no control condition that adds a comparably sized set of annotations from simple non-LLM heuristics, such as marking every function whose body calls malloc/calloc/realloc/strdup and returns the pointer, or every function that calls free/delete on a parameter. The observed increase from 5 to 10 found bugs may therefore be due to the number of added allocation/deallocation points rather than to the semantic quality of LLM-generated annotations. The paper's central claim (Section 1) is that LLM-generated annotations improve detection; a control matched on annotation count is necessary to support this specific claim rather than the weaker claim that any broad annotation set helps.","section":"Section 5.3, Table 4"},{"comment":"The claim that LAMeD 'reduces path explosion' is not operationalized or measured anywhere in the paper. Section 5.3 reports a 4.7x increase in CodeQL warnings (139 to 653) and a 4.5x increase for Cooddy (86 to 391), which is at least in tension with the notion of reduced analysis complexity. No proxy for path explosion is reported, such as analysis runtime, number of explored states, or number of unrolled paths. Either provide direct evidence (e.g., wall-clock analysis time before and after annotation insertion) or remove the 'reduces path explosion' claim from the abstract and introductory motivation, since the current data do not support it.","section":"Abstract and Section 2.1"},{"comment":"Annotation quality is validated only on the cJSON library (152 functions), and the chosen configuration (Codestral with post-filtering, precision 0.933) is then applied to the other six projects without per-project validation of the generated annotations against actual allocation/deallocation behavior. The paper itself notes in Section 5.3 that some annotation sets are more effective than others and that warning counts vary widely. Because the real-life dataset contains known leak sites, the authors could strengthen the causal chain by sampling the generated annotations per project and estimating precision/recall there, or at least analyzing how annotation errors correlate with the increase in warnings. As written, the connection between LLM semantic accuracy and the detection improvement is assumed rather than demonstrated for most projects. Relatedly, the post-filtering step (Listing 2) relies on the LLM correctly distinguishing allocation to a new variable from assignment to an existing structure; this assumption is not validated independently.","section":"Section 5.2, Table 2 and Section 5.3"}],"minor_comments":[{"comment":"The text says the five authors labeled '30+ methods' of cJSON, but the dataset subsequently comprises 152 functions. Please clarify what the '30+' refers to (e.g., ambiguous cases, functions with nontrivial annotations) to avoid confusion.","section":"Section 5.1.2"},{"comment":"The average annotation counts differ between CodeQL and Cooddy for the same projects (e.g., curl: 117 vs 137; libsolv: 88 vs 103). This is explained in Section 5.3 as an effect of removing annotations without the returned-value property for CodeQL, but adding a note to the caption would make the table self-contained.","section":"Table 4 caption"},{"comment":"The statement 'it is easy to see a direct relationship between the number of annotations and number of warnings' would be more convincing with a correlation coefficient or a small scatter plot; as written, the relationship is qualitative.","section":"Section 5.3"},{"comment":"Naming is inconsistent: 'cjson' and 'cJSON', 'Codestral' and 'codestral' appear in different places. Please standardize.","section":"Throughout"},{"comment":"The sentence 'The model with the largest number of parameters, DeepSeek-R1-70B, did not provide the better results in comparison' is grammatically awkward; consider rephrasing to 'did not provide the best results among the tested models'.","section":"Section 5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical problem, and the authors have released their prompts and plan to release data, which is commendable. My main concern is the absence of a control for annotation volume, which leaves the central contribution (LLM semantic quality vs. raw coverage) underdetermined. The abstract also overclaims 'reduces path explosion,' which no experiment supports. Neither issue is fatal to the underlying idea, but both require additional experiments or substantial claim softening. I would also encourage the authors to consider reporting per-project annotation precision to strengthen the causal story."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nLAMeD is a straightforward proof of concept: prompt an LLM to mark functions that allocate or free memory, convert the output into CodeQL/Cooddy annotations, run the analyzers. On 43 known leaks across seven C projects, it doubles the number of target bugs each analyzer finds (5 to 10 for both). That is a real, useful result, and the paper is honest about the cost: warnings jump from 139 to 653 for CodeQL and 86 to 391 for Cooddy.\n\nThe strongest part is the simple two-stage zero-shot prompt (initial allocation/free identification, then a post-filter asking whether the returned value points into a passed-in structure) plus the use of Joern for callee context. The authors also ship their manual cJSON annotation set, prompts, and real-leak dataset on Zenodo. That reproducibility is more than many papers in this area provide.\n\nThe soft spots are real but not fatal. The main internal validity issue: there is no baseline that adds a similar volume of annotations without LLM semantics. LAMeD adds 10–137 annotations per project; CodeQL's name/signature heuristic adds 0–7. The 5-to-10 gain could come from covering more functions rather than from the LLM's understanding. A cheap heuristic baseline—say, annotate every function that returns a pointer and calls malloc/realloc/strdup somewhere in its callees—would control for annotation count. Without it, the paper's causal claim about LLM semantic precision is underdetermined. A second issue is the abstract: it says LAMeD \"reduces path explosion,\" but the paper's own RQ3 and Table 4 show the opposite (warnings increase sharply). That sentence should go. Minor: the dataset is small, there are no error bars, and there is no comparison to the closest LLM-based resource-leak detectors (SkipAnalyzer, and the ICSE'25 paper by Wang et al., which is cited but not compared).\n\nWho is this for? Researchers working on LLM-assisted static analysis or tool-specific annotation generation will get a useful baseline and a clean dataset. It is not the last word, but it is a fair proof of concept. I'd send it out for peer review. A competent referee should ask for the matched-volume heuristic baseline and the abstract fix; with those, it becomes a solid contribution.","headline":"Useful proof of concept, but the evaluation doesn't isolate LLM quality from annotation volume and the abstract's path-explosion claim contradicts the warning counts.","tokens_in":18441,"tokens_out":2300,"would_cite":true,"duration_ms":26376,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM-generated annotations for allocation and deallocation functions, fed into static analyzers, double the number of known memory leaks found in seven real C projects (from 5 to 10 for both CodeQL and Cooddy), at the…","keywords":["memory leak detection","static analysis","LLM-generated annotations","function annotations","data flow analysis","path explosion","C/C++ code","zero-shot prompting"],"falsifier":"Instrument every allocation site in the seven test projects with runtime ownership tracing, then compare the LLM's per-function labels against observed ownership; if a large share of generated AllocSource annotations point to functions whose returned memory is meant to be embedded and freed with its container, the warning surge is mostly noise and the detection gain cannot be attributed to correct annotations.","tokens_in":1569,"feed_emoji":"🔍","tokens_out":1900,"duration_ms":101834,"temperature":0.7,"pith_summary":"LAMeD asks a large language model to look at each function in a C codebase, with its callees as context, and say which variables receive newly allocated memory and which receive freed memory. Those answers are converted into the annotation format that static analyzers like CodeQL and Cooddy use to track allocation and deallocation, replacing manually written hints. On a manually labeled cJSON benchmark, the configuration chosen for the main runs had precision 0.933 and recall 0.583, and on 43 known leaks across seven C projects it doubled the number of target bugs each analyzer found, from 5 to 10. The cost is a large increase in total warnings: CodeQL went from 139 to 653 and Cooddy from 86 to 391, so the gain in detections comes with more noise for an analyst to sort through. If this holds up, it means zero-shot LLM annotation can substitute for expert-curated analyzer configuration as a first pass on unfamiliar codebases.","feed_headline":"LLM-written hints double memory-leak finds in static analyzers","feed_subtitle":"Adding machine-generated allocation annotations lifted CodeQL and Cooddy detections from 5 to 10 of 43 known leaks.","key_machinery":"The load-bearing mechanism is the two-stage LLM prompt with call-graph context plus post-filtering, followed by conversion into the analyzer's annotation format. The first prompt extracts allocated_variables and deallocated_variables from a function's source; the second asks whether a returned value points into a structure passed by argument, filtering out cases where allocation does not produce an independently owned pointer. A call-graph and source extraction engine supplies the callee context that lets the model recognize custom allocators such as solv_chksum_create, which a name-and-signature heuristic misses. The converted annotations become AllocSource and FreeSink entries for Cooddy or custom memory models for CodeQL; that interface is where the LLM's semantic guesses enter the formal data-flow machinery.","core_discovery":"The central result is that the generated annotations change what the analyzers see: functions like solv_chksum_create in libsolv, which allocate through a custom wrapper rather than a direct malloc, become visible allocation sources once the LLM names them, and the analyzer then follows the allocated pointer to the missing free. LAMeD found five additional target bugs per analyzer over the unannotated baseline across different projects, while the annotation sets were not a subset of the analyzers' own heuristics: CodeQL's name-and-signature heuristic caught low-level wrappers like cJSON_malloc that LAMeD missed, while LAMeD caught context-dependent allocators the heuristic cannot see. The paper also reports that post-filtering raises annotation precision substantially, that larger models are not automatically better, and that no leak was found in libxml2 despite a large annotation set.","pith_inferences":["A direct check of the transfer claim would be to trace every allocation site in the seven projects at runtime and compare the LLM's per-function labels against observed ownership; the paper validates labels only on cJSON, so transferability is an open assumption.","The warning surge may itself be usable as a signal: newly fired warnings point at functions whose ownership semantics the LLM labeled incorrectly, which could be mined to build a self-correcting annotation loop.","If low-level wrappers like cJSON_malloc are systematically missed because they return a pointer without storing it in a local variable, a cheap remedy is to run name-and-signature heuristics alongside the LLM and merge the annotation sets, since the paper shows the two are partly disjoint.","One could test whether few-shot prompting with a handful of verified annotations from the target project improves precision enough to tame the warning explosion; the paper deliberately sticks to zero-shot to keep cross-project comparisons clean."],"forward_implications":["Static analyzers can be pointed at a new C codebase and, without manual annotation, recover leaks they would otherwise miss, roughly doubling known-bug recall on the tested projects.","The warning count will often grow in proportion to the number of annotations, so teams adopting this approach should budget for triage or add a precision filter before treating warnings as findings.","Annotation sets can be regenerated after major code changes and reused across many analysis runs, so the LLM cost is paid once per codebase revision rather than per run.","Because LAMeD and the analyzers' name-based heuristics find different allocation functions, the two are complementary, and combining them should give better coverage than either alone.","The same two-stage annotation generation could be applied to other bug classes where analyzers need source and sink specifications, such as buffer overflows or race conditions, as the paper suggests."],"supporting_citations":[{"why":"Supplies the real-life vulnerability dataset from which the seven C projects and their fixed memory-leak commits were taken.","marker":"[15]"},{"why":"Provides the call-graph and function-source extraction that feeds the LLM prompts with context.","marker":"[38]"},{"why":"Codestral is the backbone model whose post-filtered annotations are used in the real-life leak detection runs.","marker":"[2]"},{"why":"One of the three LLMs compared on the cJSON annotation task; its behavior informs the choice of backbone model.","marker":"[30]"},{"why":"Another comparison model on cJSON; the paper notes it achieved the best precision when post-filtered.","marker":"[37]"},{"why":"Defines the JSON format that structures the LLM output before conversion to analyzer annotations.","marker":"[53]"}],"fun_headline_variants":["LLM annotations double leak finds in static analyzers","LLM hints uncover leaks that heuristic rules miss","Static analyzers get 2x leak detection via LLM annotations","LLM specs for allocators: leak detection jumps from 5 to 10","LLM-written function specs reveal leaks missed by CodeQL"],"cache_read_input_tokens":20480,"weakest_assumption_plain":"The result holds only if the LLM's zero-shot judgments about which functions allocate or free memory line up with the analyzers' conservative ownership semantics; if those notions drift apart, the extra annotations produce false positives instead of detections.","fun_headline_variants_meta":{"raw":{"variants":["LLM annotations double leak finds in static analyzers","LLM hints uncover leaks that heuristic rules miss","Static analyzers get 2x leak detection via LLM annotations","LLM specs for allocators: leak detection jumps from 5 to 10","LLM-written function specs reveal leaks missed by CodeQL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001365,"raw_usage":{"total_tokens":5495,"prompt_tokens":867,"completion_tokens":4628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":4557}},"tokens_in":483,"tokens_out":4628,"duration_ms":35471,"temperature":1.0,"reasoning_tokens":4557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:53:13.179026+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument every allocation site in the seven test projects with runtime ownership tracing, then compare the LLM's per-function labels against observed ownership; if a large share of generated AllocSource annotations point to functions whose returned memory is meant to be embedded and freed with its container, the warning surge is mostly noise and the detection gain cannot be attributed to correct annotations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Codestral is the backbone model whose post-filtered annotations are used in the real-life leak detection runs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the JSON format that structures the LLM output before conversion to analyzer annotations."}],"review_version":1}