{"id":"620d2182-0b1c-41b7-a21a-dd346945674c","arxiv_id":"2501.08200","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CWEval is a new benchmark that simultaneously checks functional correctness and security of AI-generated code with dynamic test oracles, exposing a large correct-but-insecure gap in current LLMs.","lead":"This paper introduces a benchmark, CWEval, that tests whether code written by AI assistants is not only correct but also secure, using runtime tests instead of static rule checkers. The authors find that today's models often produce code that passes functional tests but contains security vulnerabilities, and that older benchmarks missed many of these cases.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Security oracle completeness is unvalidated: finite hand-crafted security tests per task make func-sec@k an upper bound, not a proven accuracy.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the finite security oracle is assumed to be a complete vulnerability detector, and this assumption is validated only against the authors' own reference implementations. That is the main soft spot in the paper's central argument. The open-redirect example in Fig. 2 makes the issue concrete—two security test inputs cannot cover the space of redirect-bypass strings, and the paper itself shows in Fig. 1 that semantically equivalent insecure implementations can evade static rules. An outcome-driven oracle is better than static analysis in principle, but 'outcome-driven' does not by itself establish completeness; a dynamic oracle is only as strong as its test inputs. The func-sec@k gap and the 'serious inaccuracy of previous evaluations' claim both rely on this completeness. I considered other candidate concerns: the SafeCoder citation [23] is indeed wrong, and the 'best performing temperature' presentation in Fig. 3 could slightly overstate the gap, but neither changes the core empirical observation as strongly as oracle completeness. The proposed concrete test—independent expert audit of samples that pass the current security oracles—would settle whether the completeness assumption holds. Since the reader already conditioned the verdict on this exact issue, my stress-test does not move the verdict; it remains CONDITIONAL, with the benchmark design credible but the oracle validity evidence incomplete.","tokens_in":11606,"tokens_out":3290,"duration_ms":32768,"concrete_test":"Hold out 20 CWEval tasks (including open-redirect and at least one each of injection, path traversal, DoS, and memory-safety). Collect all LLM-generated samples that pass the current security oracles (e.g., 100 per task across models). Have two security experts independently audit each sample for any vulnerability reachable under the task specification, using a predefined rubric and including known CWE bypass payloads not in the original oracle. If any expert-confirmed vulnerability is found in a sample that CWEval labels secure, the oracle is incomplete; recompute func-sec@10 with an expanded oracle and report the revised gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim in RQ.1—func-sec@10 gap of roughly 25–36 percentage points—treats the security oracle T_s^P as a complete vulnerability detector. In Section III-B, an implementation is labeled insecure iff it fails at least one security test, and secure iff it passes all of them. Completeness is load-bearing because any functional-but-vulnerable implementation that evades the finite test inputs is counted as secure, inflating func-sec@k and understating the security risk. The only validation offered is that the authors' own reference implementations pass/fail as expected (Section III-C and Section IV step 3); this checks internal consistency, not coverage of the implementation space LLMs actually produce. The open-redirect task in Fig. 2 has just two security inputs; variants like 'http://a.example.com.attack.com' or 'http://attack.com/#a.example.com' are exactly the class shown in Fig. 1 as false negatives of static analysis, and nothing demonstrates these are covered. 'Outcome-driven' avoids syntactic false positives and false negatives but retains the oracle's input-dependence; it is not automatically complete. Without independent expert labels or held-out vulnerable implementations, the headline claim that CWEval reveals 'a serious inaccuracy of previous evaluations' rests on an unproven oracle completeness assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CWEval, an outcome-driven framework for evaluating both functionality and security of LLM-generated code on the same task set, together with CWEval-bench, a multilingual benchmark of 119 security-critical coding tasks. For each task the authors provide a clear specification, functional and security test oracles, and both secure and insecure reference implementations. The evaluation reports two metrics, func@k and func-sec@k, on several commercial and open-source LLMs, showing a large gap (roughly 25–36 percentage points at k=10) between functional correctness and combined functional/security correctness. The paper also claims that previous static-analysis-based benchmarks (SecurityEval, CyberSecEval) seriously misestimate security, and presents evidence from a case study (Fig. 1) and from reproduction difficulties with CyberSecEval.","tokens_in":11825,"tokens_out":4024,"duration_ms":41583,"significance":"If the claims are supported, CWEval would be a valuable contribution: it addresses a real gap in LLM code-generation evaluation by jointly measuring functionality and security on the same tasks, uses dynamic rather than static oracles, provides reference implementations for reproducibility, and releases a multilingual benchmark. The authors are explicit about the design rationale and the limitations of static analysis, and the benchmark artifact is open-sourced. The empirical finding that LLMs often produce functional but insecure code, and the measured alignment tax induced by security fine-tuning, are practically important. However, the paper's accuracy claims rest on an unvalidated assumption that the finite security oracles completely detect all vulnerabilities, and the head-to-head evidence against previous benchmarks is largely anecdotal rather than a systematic comparison. These issues need to be addressed before the central claims can be fully trusted.","major_comments":[{"comment":"The security oracle is treated as a complete vulnerability detector, but its completeness is never measured. In §III-B, an implementation is labeled insecure iff it fails at least one security test and secure iff it passes all of them; in §IV step 3 the oracles are validated only against the authors' own reference implementations. This validation checks internal consistency (secure reference passes, insecure reference fails), but not coverage of the space of vulnerable implementations that LLMs actually produce. For example, the open-redirect task in Fig. 2 has only two security test inputs, and the variants shown in Fig. 1 as static-analysis false negatives (e.g., 'http://a.example.com.attack.com' or 'http://attack.com/#a.example.com') are precisely the kinds of inputs that a finite oracle may miss. Because any vulnerable implementation that passes all security tests is counted as secure, func-sec@k is an upper bound on the true fraction of secure code, not a measured ground truth. The paper's headline claim that CWEval provides 'high accuracy' and reveals a 'serious inaccuracy of previous evaluations' therefore requires an external validation study: comparing oracle labels against independent security experts, or against a held-out corpus of diverse vulnerable and secure implementations, and reporting oracle precision and recall.","section":"§III-B, §IV step 3, §V-A"},{"comment":"The claim that previous evaluations (SecurityEval, CyberSecEval) are 'seriously inaccurate' is not supported by a systematic head-to-head comparison. The paper shows in Fig. 1 that CodeQL misses two insecure variants and flags a secure implementation as vulnerable for a single task, and it mentions that only 562/1916 CyberSecEval samples are reproducible. While these examples demonstrate that static analysis can be unsound and incomplete, they do not quantify the degree of inaccuracy across the benchmark tasks or on the same LLM-generated code. To substantiate the central claim, the authors should run a previous static-analysis-based tool (or the original SecurityEval pipeline) on the same set of generated implementations used in the CWEval evaluation, compare the resulting labels with CWEval's labels, and manually adjudicate disagreements. Without such a comparison, the strong wording in the abstract and introduction overstates the evidence.","section":"§V-C, abstract, §I"},{"comment":"The SafeCoder fine-tuning experiment is used to argue that separate functionality and security evaluation misses an alignment tax. However, the comparison is made with a single base model (CodeLlama-7b) and a single fine-tuned checkpoint, and the paper does not report confidence intervals or variability across sampling seeds. The observed functionality drop from 30.77 to 15.35 on func@1 is large, but without repeated runs or statistical testing, it is unclear whether this drop is robust. Adding confidence intervals or a small repeated-sampling analysis would strengthen this conclusion, which is one of the paper's key contributions.","section":"§V-C3, Table II"}],"minor_comments":[{"comment":"Reference [23] is cited as 'SafeCoder' in the context of secure code generation, but the full reference is 'A machine-learning-based encoding system to embed safety identification information into QR codes,' which is a completely different paper. The authors should cite the actual SafeCoder work on secure code generation (e.g., by H. Su et al. or the relevant ICCPS/LLM security paper), or clarify the connection.","section":"References, [23]"},{"comment":"The paper reports func@k and func-sec@k without confidence intervals or error bars. Given that n=100 samples are drawn per configuration, the estimates have nontrivial variance; reporting confidence intervals or at least the number of tasks/model pairs would help the reader assess the reliability of the observed gaps.","section":"§V-A, Fig. 3"},{"comment":"The DoS detection uses a timeout threshold, but the choice of threshold is not described or justified. Since the threshold is a free parameter that can affect the security oracle's outcome, the authors should state the value used and, ideally, report sensitivity of the main results to a small range of thresholds.","section":"§III-B"},{"comment":"The left part of Fig. 2 (the SecurityEval sample) is very small and hard to read, especially the CodeQL rules. The figure would be clearer if enlarged or split into two subfigures.","section":"Fig. 2"},{"comment":"The authors state that specifications were tested with 'one or more common LLMs' to check whether they are understandable, but they do not specify which models or how many tasks were iteratively refined. Briefly reporting this process would improve reproducibility.","section":"§IV, step 2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is solid and the benchmark construction is careful, but the accuracy claims hinge on oracle completeness and a direct comparison with prior benchmarks. I would encourage the editor to require the validation study or a clear qualitative framing before publication. The incorrect SafeCoder reference is also worth flagging to the authors early, as it appears to stem from confusion between two unrelated papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a benchmark paper with a genuinely useful idea: evaluate functionality and security on the same security-critical tasks using dynamic, outcome-driven test oracles rather than static analyzers. The paired secure and insecure reference implementations are a good design choice, and the empirical gap between func@10 and func-sec@10 (roughly 25-35 percentage points across models) is a plausible and important observation. The benchmark is open-sourced, and the writing is clear about what was built and how.\n\nThe main soft spot is exactly where the stress-test note points: the security oracles are treated as complete vulnerability detectors, but the authors never measure oracle precision or recall against independent labels or a diverse corpus of implementations. Each task has a small set of hand-crafted security inputs; the open-redirect example has only two. A vulnerable implementation that evades those inputs is counted as secure, so func-sec@k is an upper bound, not a proven accuracy. The claim of high accuracy is not supported by the evidence given; only the authors' own reference implementations are used to validate the oracles.\n\nHere I part company with the harsher reading: this flaw is conservative for the paper's central claim. If insecure implementations are being missed, the true security gap is likely larger, not smaller. So the headline finding - that LLMs produce a substantial amount of functional but insecure code - still holds. What does not hold is the stronger wording about serious inaccuracy of previous evaluations and high accuracy. That needs an external agreement study, even a small one, comparing oracle labels against human security experts or held-out vulnerable implementations.\n\nTwo smaller issues: the SafeCoder citation is wrong (reference [23] points to a QR-code paper, not the actual SafeCoder secure-code-generation work), and the DoS timeout threshold is a free parameter that is not justified. Both are minor relative to the oracle validation.\n\nWho is this for? Researchers building or using LLM security benchmarks, and teams doing fine-tuning for secure code generation. It deserves peer review with a request for the external validation and the citation fix. The benchmark is worth building on even if the absolute numbers shift after better oracle validation.","headline":"A useful, well-designed benchmark for joint functionality/security evaluation whose main quantitative claim is probably real, but whose security oracle completeness is unvalidated and one citation is wrong.","tokens_in":661,"tokens_out":1874,"would_cite":true,"duration_ms":36767,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CWEval claims outcome-driven oracles can measure functionality and security on the same tasks, exposing a large gap in LLM code outputs.","keywords":["CWEval","secure code generation","LLM code generation","outcome-driven evaluation","dynamic analysis","benchmark","vulnerability detection","func-sec metric"],"falsifier":"Take a CWEval task whose security oracle has only a couple of attack inputs, such as the open-redirect example, and fuzz or manually enumerate bypass strings against implementations that pass the oracle; one unlisted malicious URL that still reaches the vulnerable behavior would show that passing the security oracle does not prove security.","tokens_in":11418,"feed_emoji":"🛡️","tokens_out":10186,"duration_ms":94115,"temperature":0.7,"pith_summary":"The paper proposes CWEval, an evaluation framework that judges LLM-generated code on functional correctness and security at the same time, with the same task specifications. It releases CWEval-bench, a set of 119 security-critical tasks covering 31 CWE types across five programming languages. The central empirical finding is that leading LLMs pass functionality tests far more often than they pass joint functionality-plus-security tests: the $\\text{func-sec@}10$ rate trails $\\text{func@}10$ by roughly 25 to 36 percentage points across the examined models. The authors argue this gap is real and was largely invisible to earlier security benchmarks, which used static-analysis rules on weakly specified tasks and could not reliably distinguish secure from insecure implementations. If correct, the work makes a case for making security evaluation dynamic, outcome-driven, and simultaneous with functionality evaluation.","feed_headline":"LLM code that works fails joint security checks by about 30 points","feed_subtitle":"A benchmark that tests correctness and security on the same tasks exposes risks static-only checks miss.","key_machinery":"The load-bearing mechanism is the pair of reference solutions plus the outcome-driven security oracle. Each CWEval task comes with a detailed specification whose security-relevant semantics are explicit, for example that a user-supplied URL will be used for a redirect, while the specification deliberately avoids any security-awareness leakage such as telling the model to be safe. The framework also ships a secure reference implementation and at least one functionally correct insecure reference implementation, which prove the vulnerability exists and that it can be fixed without changing functionality. Instead of matching source code against static patterns, the security oracle executes the generated code and checks dynamic outcomes, which is what lets it catch semantically different implementations that a pattern-based rule would miss or misreport.","core_discovery":"The paper claims to be the first evaluation method, to its knowledge, that evaluates functionality and security of LLM-generated code on the same problem set, and that this joint outcome-driven evaluation gives a different and more accurate picture than static-analysis-based benchmarks. For each task, CWEval defines functionality oracles that check expected outputs and security oracles that observe runtime behavior such as redirect destinations, timeouts, memory-access validity, and side effects. An implementation must pass both sets to be counted as functionally correct and secure. Measured on CWEval-bench, all evaluated models show a substantial drop from $\\text{func@}10$ to $\\text{func-sec@}10$, and a security-focused fine-tuned model that looks good when functionality and security are evaluated separately loses much of its functionality on the joint metric, which the paper presents as evidence that separate evaluations hide an alignment tax.","pith_inferences":["The reported gap is best read as a lower bound on security risk: if a finite security oracle misses a bypass, insecure code gets labeled secure, so the true shortfall between functional and secure code could be larger than measured.","The outcome-driven design is a template rather than a fixed benchmark: adding dynamic oracles for performance, privacy, or licensing behavior to the same task set would extend the approach beyond vulnerability detection.","The multilingual translation workflow suggests the benchmark can grow to more languages and CWE types, but the manual verification of each translated task, not the translation itself, is what will bound that growth."],"forward_implications":["Measuring $\\text{func-sec@}k$ on the same task set gives a direct estimate of how often accepting a functionally correct LLM answer also accepts exploitable code.","Security fine-tuning evaluated against separate functionality and security benchmarks can appear successful while actually teaching models to avoid security-critical code; the joint metric surfaces this avoidance as a functionality drop.","Adding a simple security reminder to the prompt improves $\\text{func-sec@}k$ for most tested models with little functionality cost, suggesting a cheap partial mitigation.","Model comparisons by functionality alone can be misleading: within a family, a smaller model can score higher on $\\text{func@}k$ but lower on $\\text{func-sec@}k$ than a larger model."],"supporting_citations":[{"why":"Provides the earlier static-analysis-based benchmark whose false positives and false negatives the paper uses as motivation.","marker":"[1]"},{"why":"Supplies the sampling and pass@k methodology that the paper adapts for its functionality and joint metrics.","marker":"[10]"},{"why":"Represents the existing security benchmark whose unclear specifications and non-reproducible samples CWEval is designed to overcome.","marker":"[16]"},{"why":"Extends that earlier benchmark and is treated by the paper as another instance of the static-analysis-based approach it critiques.","marker":"[17]"},{"why":"Provides the security fine-tuned model that the paper re-evaluates on CWEval-bench to demonstrate the alignment tax hidden by separate evaluation.","marker":"[23]"},{"why":"Supplies the CWE documentation used to guide the design of the security-critical coding tasks.","marker":"[24]"},{"why":"Supplies the static-analysis rule example used to illustrate how pattern-based checks miss and misreport implementations.","marker":"[25]"}],"fun_headline_variants":["Joint security and functionality tests slash LLM scores by 30 points","Outcome-driven benchmark finds LLM code often works but is insecure","Security-aware evaluation shows 30-point drop in LLM code quality","LLM code passes functionality but fails security in joint tests","CWEval benchmark: functional code often fails security checks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each task's hand-written security oracle contains every input needed to expose the target vulnerability, so any implementation that passes the security tests is counted secure even if an unlisted bypass exists.","fun_headline_variants_meta":{"raw":{"variants":["Joint security and functionality tests slash LLM scores by 30 points","Outcome-driven benchmark finds LLM code often works but is insecure","Security-aware evaluation shows 30-point drop in LLM code quality","LLM code passes functionality but fails security in joint tests","CWEval benchmark: functional code often fails security checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001016,"raw_usage":{"total_tokens":4298,"prompt_tokens":963,"completion_tokens":3335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":3248}},"tokens_in":579,"tokens_out":3335,"duration_ms":22014,"temperature":1.0,"reasoning_tokens":3248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:11.975754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a CWEval task whose security oracle has only a couple of attack inputs, such as the open-redirect example, and fuzz or manually enumerate bypass strings against implementations that pass the oracle; one unlisted malicious URL that still reaches the vulnerable behavior would show that passing the security oracle does not prove security.","supporting_citations":[{"cited_title":"Securityeval dataset: mining vulner- ability examples to evaluate machine learning-based code generation techniques,","cited_arxiv_id":null,"evidence_quote":"Provides the earlier static-analysis-based benchmark whose false positives and false negatives the paper uses as motivation."},{"cited_title":"Safecoder: A machine- learning-based encoding system to embed safety identification informa- tion into qr codes,","cited_arxiv_id":null,"evidence_quote":"Provides the security fine-tuned model that the paper re-evaluates on CWEval-bench to demonstrate the alignment tax hidden by separate evaluation."},{"cited_title":"Cwe - about cwe,","cited_arxiv_id":null,"evidence_quote":"Supplies the CWE documentation used to guide the design of the security-critical coding tasks."},{"cited_title":"Codeql documentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the static-analysis rule example used to illustrate how pattern-based checks miss and misreport implementations."}],"review_version":1}