{"id":"b59ae2e9-cd32-4d2f-895c-55ecdd574762","arxiv_id":"2506.15212","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"In a 32-example test, GPT-4 Advanced Data Analysis detected 30 vulnerabilities versus 11 for the OR-combined SAST tools, with a McNemar p-value below 0.05.","lead":"This paper pitted GPT-4's code analysis mode against two traditional security scanners on 32 handpicked vulnerable code examples. GPT-4 found 30 of the 32 bugs, the scanners found 11, and the gap is statistically significant.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is not reproducible from the paper's own tables: Table 3 sums to 35 and implies 33 GPT-correct detections, while Table 2 implies only 9 SAST-correct detections, so the reported McNemar chi-square (20.046) cannot be verified.","rationale":"The reader focused on leakage and labeling as the weakest assumption, but the more immediate problem is internal arithmetic: the core result as printed cannot be reproduced from the paper's own tables. This is load-bearing because the paper's contribution is precisely the quantitative comparison (30/32 vs 11/32, chi-square=20.046). No external data is needed to see the contradiction: Table 3 sums to 35, and Table 2's OR-based SAST count is 9, not 11. The reported p-value is derived from counts that are internally inconsistent, so the central statistical claim is unverifiable as presented. The protocol and memorization concerns remain valid, but they are secondary to the arithmetic. I recommend keeping CONDITIONAL: the paper should be revised to correct the tables, disclose the exact prompts and labeling rubric, and make the raw per-scenario outputs available; if the corrected analysis changes the significance, the claim should be downgraded.","tokens_in":13476,"tokens_out":5128,"duration_ms":47413,"concrete_test":"Recompute the SAST-OR column directly from Table 2 rows 1-32 and rebuild the 2x2 contingency table; then recompute McNemar's test with and without continuity correction. Also run the appendix script against the linked GitHub repository's data file. If the corrected p-value remains below 0.05, the qualitative claim survives but requires a corrected table and disclosed protocol; if it crosses 0.05, the headline claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4 presents the entire evidential basis for the central claim. Table 2 lists per-row GPT and SAST results; OR-aggregating the SonarQube and Cloud Defence columns yields 9 SAST-correct rows (1, 2, 9, 11, 19, 25, 27, 28, 29), not 11. Table 3 nevertheless reports a 2x2 table with a=11, b=0, c=22, d=2, which sums to 35 scenarios and implies GPT-4 was correct on a+c=33 scenarios, contradicting the abstract's 30/32 and Table 2's 30. The reported McNemar chi-square of 20.046 with p=0.000007562 corresponds to discordant counts b=0, c=22 (continuity-corrected: (22-1)^2/22 = 20.045), i.e., it uses the inconsistent Table 3 counts. The appendix code is syntactically broken (e.g., 'from statsmodels.stats.contingency_tablesimport McNEmar', 'McNamara(table, ...)', 'read_excel' of an unspecified file) and initializes a=d=0, so it cannot validate the computation. The qualitative direction may survive a correction: recomputing from Table 2 gives b=0, c=21, chi-square=(21-1)^2/21 = 19.05, p about 1.3e-5, still below 0.05. But the exact numbers in the abstract and results section are unsupported until the contingency table is corrected and the protocol (prompts, labels) is disclosed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a controlled comparison of GPT-4 (Advanced Data Analysis) with two SAST tools, SonarQube/SonarCloud and Cloud Defence, on 32 curated code security scenarios. Each scenario is scored binomially, the two SAST outputs are OR-aggregated into a single baseline, and McNemar's test is applied to the paired outcomes. The paper claims GPT-4 detected 30/32 scenarios (93.75%) versus 11/32 for SAST with a statistically significant difference, and discusses LLM-specific security risks and integration considerations for DevSecOps.","tokens_in":13768,"tokens_out":4707,"duration_ms":44475,"significance":"The question of whether an LLM can outperform traditional SAST on realistic vulnerability snippets is timely and practically relevant. The authors chose an appropriate paired statistical test, use an OR-combined SAST baseline that is favorable to the classical tools, and make their raw records available in a public repository; these are genuine strengths. The significance of the reported effect depends on the credibility of the binary labels and on the absence of prompt leakage and training-data memorization; as written, both are unresolved, so the result should be treated as a preliminary empirical claim rather than a settled comparative benchmark.","major_comments":[{"comment":"The headline statistics are internally inconsistent. Table 3 has a=11, b=0, c=22, d=2, which sums to 35 scenarios and implies GPT-4 was correct on 33; the abstract and Table 2 state 30 correct out of 32. OR-aggregating the SonarQube and Cloud Defence columns in Table 2 gives 9 SAST-correct scenarios (IDs 1, 2, 9, 11, 19, 25, 27, 28, 29), not 11. The reported chi-square 20.046 is exactly the continuity-corrected value for b=0, c=22, so the test was run on the inconsistent Table 3 counts. The authors must rebuild the contingency table from Table 2 and update the abstract, Section 4, and the McNemar result accordingly; the qualitative conclusion would survive correction, but the precise numbers in the abstract are currently unsupported.","section":"Section 4, Tables 2 and 3"},{"comment":"The detection protocol is not reproducible. The paper does not give the exact prompts sent to GPT-4, the temperature or number of runs, the ordering of scenarios, or whether the code snippets were accompanied by their CWE names. It also does not describe how the binary 'correct detection' labels were assigned or independently verified. Since every downstream statistic depends on these labels, the authors must disclose the full protocol and an annotation rubric.","section":"Sections 3.1 and 4"},{"comment":"The 32 samples are taken from public GitHub repositories and Snyk lessons, so some snippets may be present in GPT-4's training data. The paper does not check for memorization or test variants. This limits external validity; at minimum the authors should state whether CWE identifiers or vulnerability categories were withheld from the prompts and should repeat the comparison on modified or newly generated unseen examples.","section":"Section 3.2 and repository [22]"},{"comment":"The provided Python script cannot reproduce the result. It contains syntax errors in the statsmodels import and in the function call ('McNEmar'/'McNamara'), reads an unspecified Excel file, and initializes cells a and d to 0. A corrected, runnable script with the actual contingency data should be supplied so that the chi-square and p-value can be verified.","section":"Appendix"}],"minor_comments":[{"comment":"The sentence 'adopting a significance level of 0.05%' should read 0.05 (5%), and the text 'using the in the appendix' is incomplete.","section":"Section 4"},{"comment":"The experimental design is described as a 'Chi-Squared Test for Independence', but the correct description is McNemar's paired test; the later text is correct.","section":"Section 3.1"},{"comment":"Several CWE mappings are questionable (e.g., Log4J mapped to CWE-502 and ReDoS to CWE-185); please verify against current CWE entries or use the canonical CWE for each vulnerability.","section":"Table 2"},{"comment":"Future research item 5, 'LLMs-trained using Fault-Tolerant Quantum Computers', is an incomplete dangling bullet; it should either be removed or written out as a full item.","section":"Section 5"},{"comment":"The phrase 'by an accuracy of 94%' is imprecise; report the raw 30/32 figure and sample size in the same sentence.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is on a useful topic, but the empirical core is not yet journal-ready: the central table must be corrected and the protocol needs complete disclosure. If the authors deliver the corrected counts and reproducible prompts, the paper could become a compact empirical note; in its current form it is not a reliable data point. I would not recommend acceptance at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the paper. The punchline: the headline result is likely right in direction but wrong in arithmetic. GPT-4 does seem to detect more of these 32 scenarios than the OR-pooled SAST baseline, and McNemar would still be significant if you recompute from Table 2. But the paper reports numbers that don't add up: Table 3 sums to 35, and Table 2 implies 9 SAST-correct rows, not 11. The chi-square 20.046 is computed from the bad table, not the raw data.\n\nWhat's new and useful: the 32-scenario set spans a reasonable spread of CWEs, the SAST baseline is two real tools OR-aggregated (generous to SAST, which is fine), and the paired McNemar analysis is the right tool even if the implementation is sloppy. This is a modest extension of earlier LLM-vs-SAST comparisons (Cheshkov, LLift), but it adds a direct datapoint on GPT-4 Advanced Data Analysis, which is fine.\n\nThe soft spots are real and not minor. The protocol is under-described: prompts sent to GPT-4 are not given, so we can't rule out leakage or that the model was steered toward the CWE name. The binary labeling rubric is absent—who decided \"correct detection\" and using what criterion? The code samples come from public GitHub and Snyk lessons, so training-data contamination is a live concern. And the appendix code is broken: misspelled imports, a nonsense function call, and a contingency table initialized with zeros for a and d. That doesn't inspire confidence in the rest of the computation.\n\nThat said, the qualitative conclusion is robust to the arithmetic fix: from Table 2, b=0, c=21, continuity-corrected chi-square ≈ 19.0, p ~ 1e-5. So the paper's main claim—that GPT-4 outperforms these SAST tools on this benchmark—survives. But the exact numbers should not be cited until corrected.\n\nThe paper is worth a serious referee: the research question is relevant, the design is basically sound, and the problems are fixable. But it needs major revision: disclose prompts and labeling, fix the tables, make the analysis code run, and discuss contamination. I would not cite it as is.\n\nRecommendation: send to peer review with a strong request for revision. If the authors deliver the corrected data and protocol, this becomes a useful empirical datapoint for the LLM-as-scanner literature.","headline":"The paper's central claim is probably true but the reported statistics are internally inconsistent, so it needs major revision before the numbers can be trusted.","tokens_in":14363,"tokens_out":3296,"would_cite":false,"duration_ms":30657,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports a controlled comparison in which GPT-4-Advanced Data Analysis correctly detected 30 of 32 curated coding vulnerabilities, while two standard SAST tools combined detected 11, a statistically significant difference by…","keywords":["large language models","vulnerability detection","GPT-4 Advanced Data Analysis","static application security testing","McNemar's test","secure software development","zero-day vulnerabilities","code analysis"],"falsifier":"Re-run the comparison on 32 newly written, non-public vulnerable snippets, asking GPT-4 to detect bugs from the code alone and scoring with a pre-registered rubric; if GPT-4 no longer outperforms the same SAST baseline by a wide margin, the reported advantage is an artifact of memorization or prompt leakage.","tokens_in":13249,"feed_emoji":"🐛","tokens_out":9623,"duration_ms":76666,"temperature":0.7,"pith_summary":"This paper is a controlled comparison asking whether a large language model can detect software vulnerabilities as well as or better than traditional static analyzers. The authors selected 32 known coding pitfalls with sample code, ran each through GPT-4 (Advanced Data Analysis) and through two SAST tools (SonarQube and Cloud Defence), aggregated the SAST results with a logical OR, and scored each detection as binary. GPT-4 correctly flagged 30 of the 32 scenarios, while the aggregated SAST baseline flagged 11. McNemar's paired test yielded a chi-square of 20.046 with p = 0.000007562, which the authors read as a statistically significant advantage for GPT-4. The result matters because it suggests LLM-enhanced scanning could complement or replace parts of traditional SAST pipelines, with cost and speed implications for secure development.","feed_headline":"GPT-4 finds 30 of 32 coding bugs; SAST tools find 11","feed_subtitle":"Paired statistical test says the language model beats two static analyzers combined on common vulnerability classes.","key_machinery":"The central machinery is the paired comparison protocol: 32 curated vulnerable code snippets drawn from public examples, a binary 'correct detection' label for each tool, an OR-aggregation of the two SAST tools into a single baseline, and McNemar's test applied to the resulting 2x2 table of concordant and discordant outcomes. McNemar's test is the identity that carries the statistical claim, since it evaluates whether the proportion of GPT-4-only detections significantly exceeds the proportion of SAST-only detections.","core_discovery":"The paper's central claim is that GPT-4-Advanced Data Analysis detects coding vulnerabilities more effectively than the combined output of two established SAST tools on a shared benchmark of 32 security scenarios. In the paper's own accounting, GPT-4 achieved 30 correct detections (93.75%) against 11 for the SAST baseline, and the paired McNemar test on the discordant pairs (22 GPT-4-only, 0 SAST-only) gives a chi-square of 20.046 with a p-value of 0.000007562, rejecting the null hypothesis that GPT-4 is no better than SAST.","pith_inferences":["Given the public provenance of the snippets (GitHub and Snyk lessons), GPT-4's training data may contain these exact examples, so the 93.75% figure could overstate performance on genuinely novel code.","The paper does not report the exact prompts sent to GPT-4 or the labeling rubric, so a replication with masked prompts and pre-registered labels could narrow or widen the gap.","A natural extension would be to measure false positives and scoring confidence, since binary correct/incorrect hides how often the LLM flags benign code."],"forward_implications":["If the result holds, LLM-based scanning could serve as a complement to SAST in CI/CD pipelines, catching vulnerability classes that static analyzers miss.","Security teams may be able to reduce tooling costs if a general-purpose LLM matches or beats specialized SAST licenses on common bug classes.","The performance gap suggests investing in security-tuned LLMs and cleaner vulnerability datasets could yield further gains.","Because the model's knowledge has a training cutoff, the same detection edge may not extend to vulnerabilities discovered after that cutoff."],"supporting_citations":[{"why":"Supplies one of the two SAST tools whose outputs are OR-aggregated into the baseline.","marker":"[19]"},{"why":"Supplies the second SAST tool used in the aggregated baseline.","marker":"[20]"},{"why":"Provides the source code snippets and the recorded GPT-4 and SAST responses that constitute the dataset.","marker":"[22]"},{"why":"Provides McNemar's paired test used to assess the statistical significance of the detection difference.","marker":"[23]"}],"fun_headline_variants":["GPT-4 finds 30 of 32 coding bugs; SAST finds only 11","LLM beats combined SAST tools on 32 bug scenarios","GPT-4 outperforms SAST in 32-case bug detection study","GPT-4 93.75% vs SAST 34.38% on bug tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison assumes that the 32 public code samples and the binary 'correct detection' labels measure vulnerability-detection skill without leakage: the snippets may already be in GPT-4's training memory, the prompts may hint at the bug class, and no rubric for labeling is documented.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 finds 30 of 32 coding bugs; SAST finds only 11","LLM beats combined SAST tools on 32 bug scenarios","GPT-4 outperforms SAST in 32-case bug detection study","GPT-4 93.75% vs SAST 34.38% on bug tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1474,"prompt_tokens":840,"completion_tokens":634,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":548}},"tokens_in":456,"tokens_out":634,"duration_ms":6043,"temperature":1.0,"reasoning_tokens":548,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:40:25.381341+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on 32 newly written, non-public vulnerable snippets, asking GPT-4 to detect bugs from the code alone and scoring with a pre-registered rubric; if GPT-4 no longer outperforms the same SAST baseline by a wide margin, the reported advantage is an artifact of memorization or prompt leakage.","supporting_citations":[],"review_version":2}