{"id":"5d85101c-4a7b-498e-b49d-2a265346d56a","arxiv_id":"2608.09181","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Memoir captures recurring false-positive patterns in static-analysis alerts as structured memories, achieving reported F1 up to 99% on CWE-Bench-Java, though the evolution loop is not evaluated.","lead":"Memoir is a framework that learns reusable semantic memories from past false-positive security alerts and uses them to triage new alerts from SAST tools. The authors report near-perfect F1 on CWE-Bench-Java and large false-positive reductions in a proprietary industrial deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core mechanism depends on LLM-generated semantic memories that are never validated for correctness; if those memories misstate security semantics, clustering, verification, and the 99% F1 all inherit the error. The benchmark's tiny TP count makes perfect precision weak evidence.","rationale":"The reader's weakest assumption and my concern coincide: the semantic memory base is the central epistemic load-bearing element, and it is never independently validated. I agree with the CONDITIONAL verdict because the framework is plausible and the end-to-end numbers are internally consistent except for the abstract/table mismatch, but the missing validation of the intermediate representation is a concrete, addressable gap rather than a fatal flaw. If the proposed memory-correction test passes, the central claim would be substantially stronger; if it fails, the framework's generality claim is unsupported. I am not arguing that the authors are dishonest or that the results are fabricated—only that the current evidence does not rule out the possibility that the same LLM-generated rationales are being used to confirm themselves. The small number of true positives in the test split makes the perfect precision especially fragile, so the requested test is both necessary and sufficient to settle whether Memoir's verification actually protects true vulnerabilities.","tokens_in":14974,"tokens_out":3636,"duration_ms":36710,"concrete_test":"Sample 50 memories from the constructed repository and have two independent security engineers validate each memory's taxonomy, security invariants, and taint breakers against the original code and alert traces, measuring inter-rater agreement. Then delete or correct all memories judged incorrect and rerun the full Memoir pipeline on the same held-out test split; if F1 drops by more than 2 points, or if the previously perfect precision is lost, the reported performance depends on unverified memory correctness. As a complement, construct an adversarial set of true-positive alerts that superficially match retrieved memories' keys but violate their invariants, and check whether the verification stage catches them.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the LLM-generated semantic memories (Section II-C) faithfully capture why each historical alert is a false positive. Every downstream stage—clustering, retrieval, taxonomy/invariant/taint-breaker verification, and the final LLM reasoning (Section II-D)—consumes these memories as ground truth. The evaluation measures only end-to-end FP/TP agreement with benchmark labels (Section IV-A) and never checks memory fidelity. Since the same model family that writes the memories also performs the 'verification', a systematically wrong invariant can be self-consistent and still suppress a true positive. The Limitations (Section VII-C) admit no formal soundness guarantees; the missing piece is not formal proof but any empirical check of the intermediate representation. The test set further weakens confidence: 865 alerts with 95.66% FP split 7:1.5:1.5 leaves ~130 test alerts and only a handful of TPs, so perfect precision carries little statistical weight. The abstract's F1=99.43%/Recall=98.88% also does not match Table I (0.9921/0.9843), which should be resolved before relying on headline numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Memoir, a memory-driven framework for false-positive (FP) reduction in static application security testing (SAST). It converts historical FP alerts into structured semantic memories (taxonomy, security invariants, taint breakers, retrieval keys, verification rules, representative examples) via LLM-guided annotation, clustering, and synthesis. For a new alert, it retrieves relevant memories, performs taxonomy/invariant/taint-breaker verification, and then makes an FP/TP prediction. Verified results are fed back to evolve the memory base. The evaluation on CWE-Bench-Java CodeQL alerts compares Memoir against zero-shot, IRIS, RAG, BugLens, and ZeroFalse, reporting near-perfect precision/recall/F1, with ablations and cross-backbone experiments; an industrial case study reports large FP reductions on CodeQL and XCheck without retraining.","tokens_in":15274,"tokens_out":3833,"duration_ms":33433,"significance":"If the empirical claims hold, Memoir would be a valuable step toward making SAST triage knowledge reusable and evolvable. The paper's strengths include a public benchmark evaluation, a broad baseline comparison, ablations that isolate clustering and verification, cross-backbone generalization, and a code/data release. The central results, however, are supported by a very small test set (roughly 130 alerts with only a handful of true positives), an unvalidated intermediate memory representation generated by the same LLM family used for decision-making, and inconsistent headline numbers between the abstract and Table I. The significance is therefore conditional on addressing these concerns.","major_comments":[{"comment":"The abstract reports F1 = 99.43% and Recall = 98.88% with perfect Precision, whereas Table I reports Precision = 1.0000, Recall = 0.9843, and F1-score = 0.9921 for the same GPT-4o configuration. These numbers are inconsistent. Please state which set is correct and reconcile the abstract and body, since the near-perfect F1 is the paper's headline claim.","section":"Abstract and Table I"},{"comment":"The test set is very small: 865 alerts with 95.66% FP split 7:1.5:1.5 yields about 130 test alerts, of which only roughly 5-6 are true positives. Perfect precision is therefore not strong evidence, because a single misclassified true positive would change the precision substantially. Please report the exact TP/FP counts per split and include confidence intervals or repeated-run variance for the headline metrics.","section":"Section III-C and Section IV-A"},{"comment":"The memories are produced by LLM-guided semantic annotation of why historical alerts are false positives, and the verification stage uses the same model family to check taxonomy consistency, invariants, and taint breakers. The evaluation never validates the fidelity of these intermediate memories against independent ground truth; only the end-to-end FP/TP agreement with benchmark labels is measured. Since a systematically wrong but self-consistent memory could suppress true positives, please provide an empirical check of memory semantics, such as expert evaluation of a sample or a robustness test where memories are generated by one LLM family and verified by another.","section":"Sections II-C, II-D, and VII-C"},{"comment":"The industrial case study reports only aggregate FP reduction and precision improvement, with no information on the number of projects, the total alert volume, or the exact protocol by which the memory base was transferred across tools. The claim of 'without retraining' is a key generalization claim, so please specify whether the same memory repository from the public benchmark was used for both CodeQL and XCheck alerts, how the XCheck alerts were represented, and how many projects/alerts were involved.","section":"Section V"}],"minor_comments":[{"comment":"The decision threshold θ appears in Figure 2 but is never defined in the text; please define it and report its tuned value or range, along with the other tuned parameters (retrieval depth, clustering similarity threshold, memory strength decay).","section":"Section III-F and Figure 2"},{"comment":"All reported metrics appear to be from a single evaluation run; please state the number of runs and report variance or confidence intervals, especially given the small test set.","section":"Section IV"},{"comment":"Please define the abbreviation 'FPR' as false-positive rate in the captions of Tables III and IV, and clarify the denominator used to compute the residual FPR.","section":"Tables III and IV"},{"comment":"The clustering step is described as using 'semantic similarity over invariants, taint breakers, and security-related features' but the underlying representation and similarity measure are not specified; please clarify the clustering algorithm and embedding used.","section":"Section II-C"},{"comment":"The label 'MUTI-STAGE FP VERIFICATION' contains a typo; it should read 'MULTI-STAGE FP VERIFICATION'.","section":"Figure 2"},{"comment":"The industrial dataset description gives no descriptive statistics; please add at least the number of projects and the distribution of alerts per tool, even if the full dataset cannot be released.","section":"Section III-D"}],"recommendation":"major_revision","confidential_remarks":"The abstract/Table I discrepancy and the tiny TP count in the test set are the most serious reliability issues; I would ask for the raw per-split numbers and a corrected abstract before further consideration. The lack of memory-fidelity validation is a deeper concern that may require additional experiments, and the industrial case study needs a more precise protocol description to support the generalization claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take for your file. Memoir is a memory-augmented false-positive triage system. The genuinely new bit is turning historical false positives into structured semantic memories—taxonomy, invariants, taint-breakers, verification rules—rather than raw examples. That is a real step beyond RAG and single-alert adjudicators like ZeroFalse. The ablation shows clustering and verification each matter, which gives me some confidence the machinery does something. Code is linked.\n\nWhat I'd want fixed before trusting the numbers:\n\nFirst, the abstract reports F1 99.43%, recall 98.88%, precision 100%; Table I says 0.9921, 0.9843, 1.0000. That is a direct inconsistency in the headline claim. Two of the three numbers differ. Needs a correction or explanation.\n\nSecond, the test set is tiny. 865 alerts, 95.66% FP, 7:1.5:1.5 split gives about 130 test alerts and roughly six true positives. Precision 1.0 on six TPs is weak evidence. There are no error bars or multiple runs. LLM outputs are stochastic; give us variance.\n\nThird, the evolution story is not actually evaluated. Section II-E describes the feedback loop, but Section III-F says the memory base is fixed during test. So the 'evolving' part is a design, not a measured benefit. The industrial study applies the learned memory across tools, but doesn't test evolution either. That's a gap between the claims and the evidence.\n\nFourth, the stress-test concern about memory fidelity is legitimate. The memories are written by an LLM, clustered, and then verified by an LLM. The evaluation checks end-to-end labels, but never checks whether the invariants stored in the memory are true statements about the code. The authors admit in Section VII-C that there are no formal soundness guarantees, but the missing piece isn't formalism—it's any empirical check that the intermediate representations are correct. A self-consistent wrong invariant could pass verification and suppress a real vulnerability. That said, the held-out test is a real check and the results are not obviously explained by leakage: memory is built only from the training split. So this is a limitation to address, not a fatal flaw.\n\nWho this is for: people working on LLM-based SAST triage and alert fatigue. It deserves a serious referee, but I'd send it back for major revision, mainly on the numbers and the evolution evaluation.","headline":"Memoir is a real attempt at reusable FP memory for SAST triage, but the headline numbers need fixing and the evolution claim outruns the evidence.","tokens_in":15756,"tokens_out":3852,"would_cite":true,"duration_ms":34849,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Memoir claims that false-positive SAST alerts can be distilled into reusable semantic memories, and that retrieving and verifying these memories identifies alert noise with near-perfect precision and recall.","keywords":["static application security testing","false-positive triage","memory-augmented reasoning","semantic memory","LLM-guided annotation","alert verification","Java benchmark alerts","taint analysis"],"falsifier":"Take a held-out set of alerts whose taint-breaking logic is preserved but whose variable names, method names, and statement order are rewritten to defeat lexical similarity; if Memoir's recall drops materially under such semantic-preserving transformations, the learned 'semantic' memories are actually surface patterns, and the paper's central mechanism is not what carries the result.","tokens_in":14781,"feed_emoji":"🛡️","tokens_out":11850,"duration_ms":96462,"temperature":0.7,"pith_summary":"Memoir sets out to convert the history of false-positive alerts from static application security testing into reusable semantic knowledge. Its central claim is that recurring reasons an alert is a false positive—framework-managed escaping, prepared-statement binding, whitelist mapping, path normalization with boundary checks—can be abstracted into structured memories, then retrieved, verified, and refined for each new alert. If that is right, a memory-guided post-processor can cut manual triage effort dramatically while preserving true-positive reports: the paper reports near-perfect precision and recall on the benchmark and large false-positive reductions on production alerts from two different analyzers. The why-care is that SAST tools are only practical when their noise can be suppressed without losing real vulnerabilities.","feed_headline":"Reusable false-positive memories filter SAST alerts at 99.4% F1","feed_subtitle":"Learned memories triage alerts across tools and projects, cutting manual inspection while keeping real vulnerabilities.","key_machinery":"The central object is the semantic memory entry $m = \\langle T, I, B, K, V, E \\rangle$—taxonomy, security invariants, taint breakers, retrieval keys, verification rules, and representative examples—which stores a recurring false-positive rationale rather than a single alert. Construction runs through LLM-guided annotation, semantic clustering, and memory synthesis so that many alerts collapse into one generalized pattern. Identification uses hierarchical retrieval: taxonomy-level filtering first, then semantic ranking of candidate memories, then progressive verification (taxonomy consistency, invariant and taint-breaker checks, and a final LLM reasoning pass) before a prediction is emitted. Evolution closes the loop by adding verified examples, expanding retrieval keys, and updating reliability statistics, so the memory base strengthens with use and decays when stale.","core_discovery":"The paper claims that SAST false-positive identification is not an instance-by-instance reasoning problem but a memory-learning problem. Memoir represents each reusable false-positive pattern as a structured memory containing a taxonomy, security invariants, taint breakers, retrieval keys, verification rules, and representative examples. Historical alerts are annotated by an LLM, clustered by semantic similarity, and synthesized into generalized memories; new alerts are matched against those memories, filtered by taxonomy, ranked by semantic signals, and accepted as false positives only after consistency checks on invariants and taint breakers plus LLM verification. Verified predictions are folded back into the repository so the knowledge evolves. The paper reports an F1-score of 99.43% with 98.88% recall and perfect precision in its abstract, while its main result table reports 0.9921 F1, 0.9843 recall, and 1.0000 precision on 865 benchmark alerts, plus false-positive reductions of 80.0% and 94.6% on two industrial analyzers without retraining.","pith_inferences":["Editorial inference: the same learn-verify-evolve loop could be applied to true-positive alerts, turning confirmed vulnerabilities into reusable detection memories; the paper only stores false-positive patterns, but its memory architecture has no structural reason to exclude true-positive semantics.","Editorial inference: the verification stage implies a testable cross-language prediction—given equivalent sanitization semantics (a Java PreparedStatement versus a C# SqlParameter), memories should transfer without retraining; the paper does not test this.","Editorial inference: the weaker command-injection suppression (residual false-positive rate 28.89%, versus 0-3% for SQL injection and path traversal) suggests the memory base under-represents project-specific sanitization logic, so adding more command-injection examples to the relevant clusters is a cheap, testable improvement.","Editorial inference: a practical risk not quantified in the paper is how quickly the memory base decays when codebases drift without new labels; replaying old memories against upgraded frameworks would measure the evolution module's renewal value."],"forward_implications":["A deployment can run Memoir as a post-processing layer over existing SAST engines, so alert noise falls (80.0% and 94.6% in the industrial study) without replacing the analyzer.","Because verified cases are folded back into memory, the system keeps improving as triage happens; there is no fixed hand-maintained rule set to rewrite when frameworks change.","Semantic memories carry explicit evidence—which invariant was violated or which taint breaker applies—so each false-positive decision comes with a human-auditable rationale, not just a probability.","Substituting different LLMs as the reasoning backbone leaves the memory-based mechanism intact and high-performing, meaning organizations are not locked into one model.","If the memory base is shared across projects and tools, one organization-level triage history can serve multiple analyzers, avoiding repeated rediscovery of the same rationale."],"supporting_citations":[{"why":"supplies the query-language foundation for the analyzer whose alerts Memoir processes.","marker":"[1]"},{"why":"provides the manually verified Java benchmark whose labels define the training, validation, and test splits.","marker":"[6]"},{"why":"supplies the structured-reasoning baseline that Memoir must outperform in the public evaluation.","marker":"[12]"},{"why":"provides the adapted post-refinement baseline compared in the evaluation.","marker":"[13]"},{"why":"provides the strongest single-alert LLM-adjudication baseline, the main comparison point.","marker":"[14]"},{"why":"supplies the default backbone LLM for memory annotation, retrieval scoring, and verification.","marker":"[15]"}],"fun_headline_variants":["Memoir: Memory-driven SAST triage hits 99.4% F1","Evolving memories cut SAST false positives to 99.4% F1","Memory-filtered SAST triage: 99.4% F1, perfect precision","Reusable memory filters SAST alerts at 99.4% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"One load-bearing premise is that the large-language-model explanations of why historical alerts are false positives faithfully capture the real security semantics; if they are wrong, the clustered memories, retrieved evidence, and verification checks are wrong together, and the paper's own limitations section notes that verification provides no formal soundness guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Memoir: Memory-driven SAST triage hits 99.4% F1","Evolving memories cut SAST false positives to 99.4% F1","Memory-filtered SAST triage: 99.4% F1, perfect precision","Reusable memory filters SAST alerts at 99.4% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3322,"prompt_tokens":1053,"completion_tokens":2269,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":2180}},"tokens_in":669,"tokens_out":2269,"duration_ms":15243,"temperature":1.0,"reasoning_tokens":2180,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:55:18.803068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of alerts whose taint-breaking logic is preserved but whose variable names, method names, and statement order are rewritten to defeat lexical similarity; if Memoir's recall drops materially under such semantic-preserving transformations, the learned 'semantic' memories are actually surface patterns, and the paper's central mechanism is not what carries the result.","supporting_citations":[{"cited_title":"QL: Object- oriented queries on relational data,","cited_arxiv_id":null,"evidence_quote":"supplies the query-language foundation for the analyzer whose alerts Memoir processes."},{"cited_title":"CWE-Bench-Java: A benchmark for java vulnerability detec- tion,","cited_arxiv_id":null,"evidence_quote":"provides the manually verified Java benchmark whose labels define the training, validation, and test splits."},{"cited_title":"IRIS: LLM-assisted static analysis for detecting security vulnerabilities,","cited_arxiv_id":null,"evidence_quote":"supplies the structured-reasoning baseline that Memoir must outperform in the public evaluation."},{"cited_title":"Towards more accurate static analysis for taint-style bug detection in linux kernel,","cited_arxiv_id":null,"evidence_quote":"provides the adapted post-refinement baseline compared in the evaluation."}],"review_version":1}