{"id":"5dbbdb73-794a-4d60-ab32-30ce394a4a6c","arxiv_id":"2411.08813","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"CyberSecEval's insecure-code tests partly measure refusal to follow bad instructions and susceptibility to leading comments, not natural insecure code generation.","lead":"A new audit finds that Meta's CyberSecEval benchmark for code security in AI models is skewed: many prompts force the model to write unsafe code, and code comments hint at the answer. Removing those flawed prompts and cues raises models' apparent security scores by 8 to 22 percentage points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 10.4 percentage point Instruct gain rests on removing all 450 GPT-4o-flagged prompts, although the paper's own manual validation found only 23.5% were truly non-compliant; the effect may be largely an artifact of deleting compliant prompts whose insecure responses were genuine failures.","rationale":"The reader's weakest assumption identified the same load-bearing concern: the Instruct analysis removes all GPT-4o-flagged prompts despite manual validation indicating that only 23.5% of a 50-sample subset were truly non-compliant. That concern is central because the paper's strongest claim, that removing 'prompts that mandated insecure practices' produces a 10.4 percentage point increase, depends entirely on the flagged set being predominantly genuine. If most flagged prompts are false positives, removing them cannot be credited to the stated mechanism, and the direction of the bias is favorable to the paper's thesis, since a compliant prompt with an insecure response is a real failure that should lower the pass rate. The paper deserves credit for the concrete examples and for attempting manual validation, but the validation rate itself undermines the quantitative conclusion rather than supporting it. I also note that the Autocomplete analysis lacks a reported validation precision, making its 17.7 percentage point claim even harder to evaluate; this is a second, related weakness but not a separate central claim. The correct response is not rejection of the qualitative critique, which may well be correct, but a conditional verdict requiring the benchmark to be re-run on the manually validated subset with uncertainty quantification. This does not change the reader's CONDITIONAL verdict, so the recommended verdict is UNCHANGED.","tokens_in":8165,"tokens_out":2928,"duration_ms":27383,"concrete_test":"Manually audit all 450 flagged Instruct prompts, identify the subset that truly cannot be complied with without violating ICD rules, and re-run the benchmark with only that confirmed subset removed; if the resulting pass-rate increase is substantially below 10.4 percentage points or within single-run noise, the headline quantitative claim is unsupported. Report bootstrap confidence intervals for the delta and, as a secondary check, report the manual validation rate for the Autocomplete rewrites and recompute that delta using only rewrites confirmed to preserve behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 reports that GPT-4o flagged 516 of 1,916 Instruct prompts, a second pass removed 66, leaving 450 flagged prompts, and a manual check of 50 flagged samples found it was impossible to comply with only 23.5% of them without violating the static analysis rules. The paper then removes all 450 flagged prompts and attributes the resulting 8.3-13.1 percentage point (average 10.4) increase in 'secure' labels to removing prompts that mandated insecure practices. This attribution is not supported: if roughly 76.5% of flagged prompts were false positives, those prompts were compliant, so an insecure response to them is a genuine benchmark failure; deleting such failures inflates the pass rate in exactly the direction of the paper's conclusion. The same structural problem afflicts Section 2.3: the 12.2-22.2 percentage point increase after stripping comments and identifiers depends on GPT-4o rewrites, but the paper never reports the manual validation rate for those rewrites, so the increase could reflect altered code behavior or validity rather than removal of superficial cues. The paper's own limitations state that experiments were run once with no error bars, and Appendix E notes Figure 1b only shows four models due to API issues. The qualitative direction of the critique may be correct, but the central quantitative magnitudes are not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper critiques Meta's CyberSecEval insecure code detection methodology and proposes an LLM-aided approach to benchmark critique. It makes three main claims: (i) Meta's static analysis ruleset (89 Semgrep rules, 8 languages) is substantially smaller than an industry-standard repository, and static analysis lacks contextual understanding; (ii) a substantial portion of the Instruct benchmark prompts mandate insecure practices, so these prompts test refusal skills rather than susceptibility to generating insecure code; and (iii) Autocomplete prompts contain comments/identifiers that cue insecure code, inflating measured vulnerability. The authors use GPT-4o to flag non-compliant prompts and to strip identifiers/comments, then re-run the benchmarks and report increases in 'secure' pass rates of 8.3–13.1 percentage points (average 10.4) for Instruct and 12.2–22.2 percentage points (average 17.7) for Autocomplete. They conclude that CyberSecEval's scores are inflated by instruction-following and superficial cues.","tokens_in":8390,"tokens_out":7610,"duration_ms":55354,"significance":"If the qualitative claims were established, the paper would make a useful contribution to the design of cybersecurity benchmarks, highlighting two confounds: prompts that explicitly demand insecure constructs and prompts that leak the expected answer through comments/identifiers. The paper ships code and data and demonstrates a workflow for LLM-assisted benchmark auditing, which is timely. However, the central quantitative magnitudes are not currently supported: the Instruct effect size is computed by removing all 450 GPT-4o-flagged prompts despite the authors' own manual validation of 50 samples finding only 23.5% truly non-compliant, and the Autocomplete effect size is computed from GPT-4o rewrites whose manual validation outcome is not reported. The single-run results without error bars further weaken the numerical claims. The qualitative direction of the critique is plausible, but the paper's main numbers should not be taken at face value.","major_comments":[{"comment":"Section 2.2: The manual validation of 50 flagged prompts found only 23.5% truly non-compliant, yet the authors removed all 450 flagged prompts when computing the 8.3–13.1 percentage point increase in pass rates. If 76.5% of the flagged prompts were false positives, those prompts are compliant, so an insecure response to them is a genuine benchmark failure; deleting them inflates the pass-rate gain in exactly the direction of the paper's conclusion. The reported average increase of 10.4 percentage points is therefore not established as the effect of removing prompts that mandate insecure practices.","section":"2.2"},{"comment":"Section 2.2: The sentence '23.5% of prompts tested LLM's refusal skills' is inaccurate because 23.5% is the proportion of the 50 sampled flagged prompts that were non-compliant, not the proportion of all Instruct prompts. Extrapolating to the full flagged set yields roughly 106 prompts (23.5% of 450), or about 5.5% of the 1,916 total prompts; the paper should report this extrapolation and its uncertainty rather than the raw 23.5% figure.","section":"2.2"},{"comment":"Section 2.3: The reported 12.2–22.2 percentage point increase in pass rates after stripping comments and identifiers rests on GPT-4o rewrites, but the paper does not report the outcome of the manual validation of 50 rewritten samples. Without knowing how often the rewrites preserved the original code's behavior and validity, the observed increase may reflect altered code behavior or invalid code rather than removal of superficial cues. The authors should report the validation results and, if incomplete, recompute the effect on rewrites that preserve meaning.","section":"2.3"},{"comment":"All experiments were run once with no error bars, confidence intervals, or significance tests, as the authors acknowledge. Because the central claims are quantitative effect sizes on a benchmark, the absence of variability measures is load-bearing; the paper should report variance across repeated runs or bootstrap intervals, or explicitly limit the paper's claims to the qualitative direction of the effect.","section":"Section 2 and checklist item 7"}],"minor_comments":[{"comment":"Abstract and Conclusion: 'an 10.4 percentage point' should be 'a 10.4 percentage point'.","section":"Abstract and Conclusion"},{"comment":"Appendix G: 'could assisst' contains a typo; should be 'could assist'.","section":"Appendix G"},{"comment":"Section 2.1: The raw rule counts (2,116 vs. 89) are not a direct measure of coverage or quality; the paper should consider reporting the overlap in CWEs or rules to support the claim that the industry repository 'significantly outscales' Meta's.","section":"2.1"},{"comment":"Section 2.1: The paper first states there are 189 static analysis rules and then refers to 89 Semgrep rules; please clarify whether the 189 figure includes other rule types and avoid the apparent inconsistency.","section":"2.1"},{"comment":"Appendix E: The note that Figure 1b displays only four models due to API issues appears only in an appendix; this limitation should be stated in the main text near the figure to avoid misleading readers.","section":"Appendix E"},{"comment":"Section 2.2: The flagging prompt uses Meta's own static analysis rules as the ground truth for 'insecure practices' while the paper critiques those rules; consider addressing this circularity and clarifying how the manual validation mitigates (or does not mitigate) it.","section":"2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's qualitative observations about CyberSecEval's prompt design are plausible and worth publishing after revision, but the headline quantitative results are compromised by the mismatch between the GPT-4o flagging and the manual validation. The authors should be asked to re-analyze the Instruct data using only the manually validated non-compliant prompts (or a corrected classifier), and to report the Autocomplete validation outcome explicitly. Given the small scale and single-run methodology, the paper may be more appropriate for a workshop than a main-track journal submission, but that is an editorial decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper finds something real about CyberSecEval, but the headline numbers don't hold up. The qualitative examples—prompts that cannot be satisfied without violating Meta's own static analysis rules, and Autocomplete prompts whose comments and identifiers cue insecure code—are worth reading and appear to be new. The LLM-aided audit method is a reasonable idea, and the authors are transparent about their prompts and limitations. That's the good part.\n\nThe soft spot is load-bearing. In Section 2.2 the authors flag 450 Instruct prompts as non-compliant, then report that a manual check of 50 of them found only 23.5% were actually impossible to comply with. They then remove all 450 prompts and attribute the resulting 8.3–13.1 percentage point increase in 'secure' labels to removing prompts that mandated insecure practices. That attribution is not defensible. Roughly 76.5% of the flagged set were false positives—prompts you could comply with without tripping the rules. An insecure response to those is a genuine benchmark failure, and deleting them inflates the pass rate in exactly the direction of the conclusion. The correct analysis would use the manually validated subset, or at least show how the effect size varies with the precision of the filter. There's also a mild circularity in using Meta's rules to judge compliance; the manual validation was the right way to break that, but it's not actually used in the main analysis.\n\nThe same problem afflicts Section 2.3: the 12.2–22.2 pp increase after stripping comments and identifiers rests on GPT-4o rewrites, but the manual validation of 50 rewrites is never reported, so we don't know whether the rewrites changed code behavior or validity.\n\nSmaller issues: the paper says '23.5% of prompts tested LLMs' refusal skills,' but that's 23.5% of the flagged set, which is about 5.5% of the whole Instruct dataset—misleading phrasing. The static-analysis rule comparison (89 vs 2,116 rules) is apples-to-oranges, since Meta's rules are targeted and the general repo includes many irrelevant ones, though the rand() point about context is fair. The authors also state they ran everything once with no error bars, and Figure 1b only shows four models because of API issues.\n\nWho is this for? Anyone using CyberSecEval for model safety comparisons or release decisions, and people working on LLM-based benchmark auditing. The qualitative critique is worth engaging, and the paper deserves a serious referee. But the quantitative claims need to be redone with a validated filter, a reported validation rate for the rewrites, and error bars. If I were editing, I'd send it to review and expect heavy revision.","headline":"Real qualitative issues in CyberSecEval, but the reported effect sizes are not supported by the paper's own validation data.","tokens_in":8954,"tokens_out":4099,"would_cite":true,"duration_ms":32344,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that CyberSecEval's insecure-code scores are inflated by unanswerable prompts and leading code cues, by about 10 and 18 percentage points respectively.","keywords":["cybersecurity evaluation","LLM benchmark critique","insecure code detection","CyberSecEval","static analysis","prompt compliance","refusal behavior","autocomplete benchmark"],"falsifier":"Manually audit all 516 flagged Instruct prompts (or a much larger random sample) to determine which ones truly make secure compliance impossible, then recompute the pass-rate change using only that confirmed set; if the confirmed set is much smaller than 516, the 10.4 percentage point effect shrinks, and if it nearly vanishes, the paper's main correction is an artifact of the critic's imprecision.","tokens_in":7943,"feed_emoji":"🔐","tokens_out":5177,"duration_ms":42156,"temperature":0.7,"pith_summary":"This paper argues that Meta's CyberSecEval insecure-code benchmarks overstate how often LLMs generate insecure code, because parts of the benchmark measure something else. In the Instruct dataset, many prompts explicitly demand an insecure practice, so a model that refuses is counted as failing; removing those prompts raises the secure pass rate by about 10 percentage points. In the Autocomplete dataset, code comments and identifier names hint at the insecure practice, so models are being led; removing those cues raises the pass rate by about 18 percentage points. The paper offers this as a case study in using an LLM as a critique assistant for benchmark analysis.","feed_headline":"Insecure-code benchmark may overstate LLM risk by 18 points","feed_subtitle":"Critique finds CyberSecEval prompts demand unsafe code or leak hints, inflating measured insecurity.","key_machinery":"The load-bearing mechanism is the LLM-aided audit loop: a large language model (GPT-4o) is prompted in a first pass to flag Instruct prompts that force a violation of the Insecure Code Detector rules, a second pass reviews those flags to remove cases where secure compliance is possible, and a separate rewrite step strips comments and identifiers from Autocomplete prompts. The measured 'pass percentage', the share of model outputs the Insecure Code Detector marks as secure, is compared before and after each adjustment, and the gap is the claimed inflation.","core_discovery":"The paper's central claim is that CyberSecEval's insecure-code detection results are inflated by two artifacts that measure compliance and cue-following rather than insecure code generation. In the Instruct benchmark, prompts that explicitly demand an insecure practice (e.g., using rand() and malloc()) cannot be satisfied without tripping Meta's own static-analysis rules, so a model that refuses is counted as vulnerable; after dropping flagged prompts, the share of samples marked secure rose by 10.4 percentage points on average. In the Autocomplete benchmark, comments and identifier names such as '// Use webcrypto if available' lead the model toward the insecure API, and after GPT-4o anonymized these cues, the secure share rose by 17.7 percentage points. The paper also claims Meta's static-analysis ruleset is far smaller than an industry-standard Semgrep set (89 rules across 8 languages vs. 2,116 rules across 28 languages) and that rule-based flagging ignores context, such as flagging rand() even outside cryptographic use.","pith_inferences":["If only the 23.5% confirmed non-compliant prompts from the manual audit were removed, the 10.4 percentage point Instruct correction would likely shrink; the study removed all LLM-flagged prompts despite low measured precision.","The Autocomplete effect may be partly a realistic finding: real codebases contain comments and descriptive names, so an LLM that copies insecure patterns from them is arguably failing a practical security test, not just an artifact.","The same audit recipe could be applied to other safety benchmarks to test whether prompts are unanswerable or self-answering, and to jailbreak suites where prompts may leak the target behavior.","Re-running the two-pass flagger with several different LLMs and measuring inter-model agreement would show whether the reported corrections are stable or model-specific."],"forward_implications":["CyberSecEval's insecure-code pass rates should be interpreted as lower bounds on secure behavior, not direct rates of insecure code generation.","Instruct benchmarks should be screened so each prompt can be answered without violating the evaluation's own rules; otherwise refusal looks like vulnerability.","Autocomplete benchmarks should anonymize comments and identifiers or draw on code that could not have been memorized, so leading cues do not carry the score.","LLM-aided critique can audit benchmarks at scale, but its flags need human validation before being used to recompute scores."],"supporting_citations":[{"why":"Defines the CyberSecEval insecure-code methodology, including the Insecure Code Detector and the Instruct and Autocomplete benchmarks that this paper critiques.","marker":"[1]"},{"why":"Extends the methodology in CyberSecEval 2 and supplies benchmark results that the paper re-runs and adjusts.","marker":"[2]"},{"why":"Extends the methodology in CyberSecEval 3 and is part of the target of the critique.","marker":"[6]"},{"why":"Supplies the Common Weakness Enumeration, the 50 insecure coding practices that Meta's static-analysis rules are designed to detect.","marker":"[3]"},{"why":"Provides the CWE-338 weakness behind the rand() example, showing why context-free flagging of rand() is treated as a limitation.","marker":"[4]"},{"why":"The industry-standard Semgrep repository used as the comparison yardstick showing Meta's ruleset is far smaller in scope.","marker":"[5]"}],"fun_headline_variants":["CyberSecEval critique: flawed prompts inflate LLM risk scores","Benchmark flaws may inflate LLM insecurity by 18 points","LLM risk benchmark biased by prompt tricks study finds","Meta's cyber eval overstates unsafe code by 18 points","Insecure-code test gaming: how prompts skew LLM results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes the LLM critic's flagging after the second pass correctly identifies prompts that cannot be followed without tripping the static rules; the paper's own manual check of 50 flagged samples found only 23.5% met that criterion, yet all flagged samples were removed when computing the reported changes.","fun_headline_variants_meta":{"raw":{"variants":["CyberSecEval critique: flawed prompts inflate LLM risk scores","Benchmark flaws may inflate LLM insecurity by 18 points","LLM risk benchmark biased by prompt tricks study finds","Meta's cyber eval overstates unsafe code by 18 points","Insecure-code test gaming: how prompts skew LLM results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":2942,"prompt_tokens":808,"completion_tokens":2134,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":2047}},"tokens_in":424,"tokens_out":2134,"duration_ms":14376,"temperature":1.0,"reasoning_tokens":2047,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:17:46.886866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually audit all 516 flagged Instruct prompts (or a much larger random sample) to determine which ones truly make secure compliance impossible, then recompute the pass-rate change using only that confirmed set; if the confirmed set is much smaller than 516, the 10.4 percentage point effect shrinks, and if it nearly vanishes, the paper's main correction is an artifact of the critic's imprecision.","supporting_citations":[{"cited_title":"Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models,","cited_arxiv_id":null,"evidence_quote":"Extends the methodology in CyberSecEval 3 and is part of the target of the critique."},{"cited_title":"Common Weakness Enumeration","cited_arxiv_id":null,"evidence_quote":"Supplies the Common Weakness Enumeration, the 50 insecure coding practices that Meta's static-analysis rules are designed to detect."},{"cited_title":"Common Weakness Enumeration","cited_arxiv_id":null,"evidence_quote":"Provides the CWE-338 weakness behind the rand() example, showing why context-free flagging of rand() is treated as a limitation."},{"cited_title":"Semgrep: Lightweight static analysis for many languages","cited_arxiv_id":null,"evidence_quote":"The industry-standard Semgrep repository used as the comparison yardstick showing Meta's ruleset is far smaller in scope."}],"review_version":1}