REVIEW 3 major objections 5 minor 10 cited by
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read CWEval claims outcome-driven oracles can measure functionality and security on the same tasks, exposing a large gap in LLM code outputs.
desk verdict A useful, well-designed benchmark for joint functionality/security evaluation whose main quantitative claim is probably real, but whose security oracle completeness is unvalidated and one citation is wrong. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pair of reference solutions plus the outcome-driven security oracle. Each CWEval task comes with a detailed specification whose security-relevant semantics are explicit, for example that a user-supplied URL will be used for a redirect, while the specification deliberately avoids any security-awareness leakage such as telling the model to be safe. The framework also ships a secure reference implementation and at least one functionally correct insecure reference implementation, which prove the vulnerability exists and that it can be fixed without changing functionality. Instead of matching source code against static patterns, the security oracle executes the generated code and checks dynamic outcomes, which is what lets it catch semantically different implementations that a pattern-based rule would miss or misreport.
What would settle it
Take a CWEval task whose security oracle has only a couple of attack inputs, such as the open-redirect example, and fuzz or manually enumerate bypass strings against implementations that pass the oracle; one unlisted malicious URL that still reaches the vulnerable behavior would show that passing the security oracle does not prove security.
Extended reading notes
Core claim
The paper claims to be the first evaluation method, to its knowledge, that evaluates functionality and security of LLM-generated code on the same problem set, and that this joint outcome-driven evaluation gives a different and more accurate picture than static-analysis-based benchmarks. For each task, CWEval defines functionality oracles that check expected outputs and security oracles that observe runtime behavior such as redirect destinations, timeouts, memory-access validity, and side effects. An implementation must pass both sets to be counted as functionally correct and secure. Measured on CWEval-bench, all evaluated models show a substantial drop from $\text{func@}10$ to $\text{func-sec@}10$, and a security-focused fine-tuned model that looks good when functionality and security are evaluated separately loses much of its functionality on the joint metric, which the paper presents as evidence that separate evaluations hide an alignment tax.
Load-bearing premise
The load-bearing premise is that each task's hand-written security oracle contains every input needed to expose the target vulnerability, so any implementation that passes the security tests is counted secure even if an unlisted bypass exists.
Editorial extensions
If this is right
- Measuring $\text{func-sec@}k$ on the same task set gives a direct estimate of how often accepting a functionally correct LLM answer also accepts exploitable code.
- Security fine-tuning evaluated against separate functionality and security benchmarks can appear successful while actually teaching models to avoid security-critical code; the joint metric surfaces this avoidance as a functionality drop.
- Adding a simple security reminder to the prompt improves $\text{func-sec@}k$ for most tested models with little functionality cost, suggesting a cheap partial mitigation.
- Model comparisons by functionality alone can be misleading: within a family, a smaller model can score higher on $\text{func@}k$ but lower on $\text{func-sec@}k$ than a larger model.
Reading between the lines
- The reported gap is best read as a lower bound on security risk: if a finite security oracle misses a bypass, insecure code gets labeled secure, so the true shortfall between functional and secure code could be larger than measured.
- The outcome-driven design is a template rather than a fixed benchmark: adding dynamic oracles for performance, privacy, or licensing behavior to the same task set would extend the approach beyond vulnerability detection.
- The multilingual translation workflow suggests the benchmark can grow to more languages and CWE types, but the manual verification of each translated task, not the translation itself, is what will bound that growth.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CWEval, an outcome-driven framework for evaluating both functionality and security of LLM-generated code on the same task set, together with CWEval-bench, a multilingual benchmark of 119 security-critical coding tasks. For each task the authors provide a clear specification, functional and security test oracles, and both secure and insecure reference implementations. The evaluation reports two metrics, func@k and func-sec@k, on several commercial and open-source LLMs, showing a large gap (roughly 25–36 percentage points at k=10) between functional correctness and combined functional/security correctness. The paper also claims that previous static-analysis-based benchmarks (SecurityEval, CyberSecEval) seriously misestimate security, and presents evidence from a case study (Fig. 1) and from reproduction difficulties with CyberSecEval.
Significance. If the claims are supported, CWEval would be a valuable contribution: it addresses a real gap in LLM code-generation evaluation by jointly measuring functionality and security on the same tasks, uses dynamic rather than static oracles, provides reference implementations for reproducibility, and releases a multilingual benchmark. The authors are explicit about the design rationale and the limitations of static analysis, and the benchmark artifact is open-sourced. The empirical finding that LLMs often produce functional but insecure code, and the measured alignment tax induced by security fine-tuning, are practically important. However, the paper's accuracy claims rest on an unvalidated assumption that the finite security oracles completely detect all vulnerabilities, and the head-to-head evidence against previous benchmarks is largely anecdotal rather than a systematic comparison. These issues need to be addressed before the central claims can be fully trusted.
major comments (3)
- [§III-B, §IV step 3, §V-A] The security oracle is treated as a complete vulnerability detector, but its completeness is never measured. In §III-B, an implementation is labeled insecure iff it fails at least one security test and secure iff it passes all of them; in §IV step 3 the oracles are validated only against the authors' own reference implementations. This validation checks internal consistency (secure reference passes, insecure reference fails), but not coverage of the space of vulnerable implementations that LLMs actually produce. For example, the open-redirect task in Fig. 2 has only two security test inputs, and the variants shown in Fig. 1 as static-analysis false negatives (e.g., 'http://a.example.com.attack.com' or 'http://attack.com/#a.example.com') are precisely the kinds of inputs that a finite oracle may miss. Because any vulnerable implementation that passes all security tests is counted as secure, func-sec@k is an upper bound on the true fraction of secure code, not a measured ground truth. The paper's headline claim that CWEval provides 'high accuracy' and reveals a 'serious inaccuracy of previous evaluations' therefore requires an external validation study: comparing oracle labels against independent security experts, or against a held-out corpus of diverse vulnerable and secure implementations, and reporting oracle precision and recall.
- [§V-C, abstract, §I] The claim that previous evaluations (SecurityEval, CyberSecEval) are 'seriously inaccurate' is not supported by a systematic head-to-head comparison. The paper shows in Fig. 1 that CodeQL misses two insecure variants and flags a secure implementation as vulnerable for a single task, and it mentions that only 562/1916 CyberSecEval samples are reproducible. While these examples demonstrate that static analysis can be unsound and incomplete, they do not quantify the degree of inaccuracy across the benchmark tasks or on the same LLM-generated code. To substantiate the central claim, the authors should run a previous static-analysis-based tool (or the original SecurityEval pipeline) on the same set of generated implementations used in the CWEval evaluation, compare the resulting labels with CWEval's labels, and manually adjudicate disagreements. Without such a comparison, the strong wording in the abstract and introduction overstates the evidence.
- [§V-C3, Table II] The SafeCoder fine-tuning experiment is used to argue that separate functionality and security evaluation misses an alignment tax. However, the comparison is made with a single base model (CodeLlama-7b) and a single fine-tuned checkpoint, and the paper does not report confidence intervals or variability across sampling seeds. The observed functionality drop from 30.77 to 15.35 on func@1 is large, but without repeated runs or statistical testing, it is unclear whether this drop is robust. Adding confidence intervals or a small repeated-sampling analysis would strengthen this conclusion, which is one of the paper's key contributions.
minor comments (5)
- [References, [23]] Reference [23] is cited as 'SafeCoder' in the context of secure code generation, but the full reference is 'A machine-learning-based encoding system to embed safety identification information into QR codes,' which is a completely different paper. The authors should cite the actual SafeCoder work on secure code generation (e.g., by H. Su et al. or the relevant ICCPS/LLM security paper), or clarify the connection.
- [§V-A, Fig. 3] The paper reports func@k and func-sec@k without confidence intervals or error bars. Given that n=100 samples are drawn per configuration, the estimates have nontrivial variance; reporting confidence intervals or at least the number of tasks/model pairs would help the reader assess the reliability of the observed gaps.
- [§III-B] The DoS detection uses a timeout threshold, but the choice of threshold is not described or justified. Since the threshold is a free parameter that can affect the security oracle's outcome, the authors should state the value used and, ideally, report sensitivity of the main results to a small range of thresholds.
- [Fig. 2] The left part of Fig. 2 (the SecurityEval sample) is very small and hard to read, especially the CodeQL rules. The figure would be clearer if enlarged or split into two subfigures.
- [§IV, step 2] The authors state that specifications were tested with 'one or more common LLMs' to check whether they are understandable, but they do not specify which models or how many tasks were iteratively refined. Briefly reporting this process would improve reproducibility.
Circularity Check
No circularity: CWEval's oracle-based security labels are an operational definition, not a derived prediction; all load-bearing references are external benchmarks and CWE documentation.
full rationale
The paper's derivation chain is: design coding tasks from CWE documentation, define functionality and security test oracles (Section III-B), build reference implementations as sanity checks (Section III-C), then measure LLM outputs with func@k and func-sec@k (Section V). No parameter is fitted to the LLM-generated code, and the security oracle T_s^P is defined independently of the evaluation results. The claim that LLMs produce 'functional but insecure code' is an empirical measurement under the paper's operational definition of insecurity (failing a security test oracle), which is the standard semantics of any test-based benchmark and not a circular derivation. The reference implementations are used only to confirm that the oracle can distinguish a known vulnerable implementation from a known secure one; they do not determine the measured scores. There are no self-citations that are load-bearing; the paper cites external sources such as SecurityEval, CyberSecEval, SafeCoder, CWE, and CodeQL. The reader's concern about oracle completeness is a validity limitation, not circularity: a finite oracle that misses a bypass would undercount vulnerabilities, but that is an empirical coverage question, not an equation reducing the conclusion to its input. The paper does not fit a parameter to a subset of data and then predict a closely related quantity, nor does it import a uniqueness theorem from prior work by the same authors. Therefore, no significant circularity is present.
Assumptions & free parameters
free parameters (1)
- DoS timeout threshold =
not specified
assumptions (4)
- domain assumption Security test oracles are sound and complete: any implementation that fails a security oracle is vulnerable, and any that passes all security oracles is secure for the specified CWE.
- domain assumption The hand-written insecure reference proves the task has real security semantics and that the oracle detects that specific vulnerability.
- domain assumption LLM-translated tasks preserve both functionality and vulnerability semantics across all five languages after manual refinement.
- domain assumption Specifications contain no security-awareness leakage, so model behavior reflects security capability rather than refusal or task avoidance.
Cite this review
Pith. "Pith review of CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation." pith.science (2026). https://pith.science/paper/4Z7RCIJO
@misc{pith2026250108200,
author = {Pith},
title = {Pith review of: CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/4Z7RCIJO}},
note = {Machine review of arXiv:2501.08200}
}
read the original abstract
Large Language Models (LLMs) have significantly aided developers by generating or assisting in code writing, enhancing productivity across various tasks. While identifying incorrect code is often straightforward, detecting vulnerabilities in functionally correct code is more challenging, especially for developers with limited security knowledge, which poses considerable security risks of using LLM-generated code and underscores the need for robust evaluation benchmarks that assess both functional correctness and security. Current benchmarks like CyberSecEval and SecurityEval attempt to solve it but are hindered by unclear and impractical specifications, failing to assess both functionality and security accurately. To tackle these deficiencies, we introduce CWEval, a novel outcome-driven evaluation framework designed to enhance the evaluation of secure code generation by LLMs. This framework not only assesses code functionality but also its security simultaneously with high-quality task specifications and outcome-driven test oracles which provides high accuracy. Coupled with CWEval-bench, a multilingual, security-critical coding benchmark, CWEval provides a rigorous empirical security evaluation on LLM-generated code, overcoming previous benchmarks' shortcomings. Through our evaluations, CWEval reveals a notable portion of functional but insecure code produced by LLMs, and shows a serious inaccuracy of previous evaluations, ultimately contributing significantly to the field of secure code generation. We open-source our artifact at: https://github.com/Co1lin/CWEval .
Figures
Forward citations
Cited by 10 Pith papers
-
Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits
Showing failing security tests to an LLM during repair fixes a subset of insecure candidates, but visible test success leaves a measurable hidden coverage gap.
-
Can AI Write Compliant Code, and to What Extent? Evaluating SOC 2 Compliance of Claude Fable 5, Claude Opus 4.8, and Claude Opus 5 Across Four Use Cases
A single SOC 2 sentence moved LLM-generated code conformance from 47 to 88 percent up to 86 to 100 percent across 24 generations, while MFA hooks, cookie flags, and account lifecycle controls still required explicit naming.
-
Breaking Customized LLMs for Coding: Automated Red Teaming for Instruction Backdoor Attacks
ARIA automatically generates stealthy backdoored system prompts for customized coding LLMs, achieving high attack success while preserving normal task performance.
-
EduPluginBench: Executable Assurance for AI-Generated Educational Plugins
EduPluginBench provides a staged assurance pipeline that catches self-constructed plugin violations conventional checks miss, while fresh model generations and real Moodle fixes reveal sharp transfer limits.
-
CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities
CyberGym-E2E turns 920 real OSS-Fuzz vulnerabilities into end-to-end agent tasks; results show frontier agents patch known bugs well (65.9% S3) but find the exact ground-truth vulnerability less often (22.2% S4).
-
SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows
An agentic workflow with security guidelines and LLM-generated unit tests improves secure code generation on CWEval C tasks from 61% to 76% Func-Sec@1 with Sonnet-3.7, at roughly 98% of its original functionality.
-
Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences
A new dataset and a token-masked preference optimization loss reduce security vulnerabilities in LLM-generated Python code while preserving code quality.
-
Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software
Frontend features of LLM-generated apps can predict hidden backend vulnerabilities that a given model tends to reproduce, enabling black-box attack triage.
-
SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
A 558-task benchmark with SAST and LLM judging shows state-of-the-art LLMs generate vulnerability-free code only about 37% of the time in zero-shot settings.
-
Adversarial Attack Classification and Robustness Testing for Large Language Models for Code
Word-level adversarial changes in prompts, code, and comments degrade code-generation correctness more than sentence-level rewrites, but classification errors and missing error bars weaken the claim.
Reference graph
Works this paper leans on
-
[23]
H. Su, J. Niu, X. Liu, and M. Atiquzzaman, “Safecoder: A machine- learning-based encoding system to embed safety identification informa- tion into qr codes,” Journal of Network and Computer Applications, vol. 227, p. 103874, 2024
work page 2024
-
[1]
M. L. Siddiq and J. C. Santos, “Securityeval dataset: mining vulner- ability examples to evaluate machine learning-based code generation techniques,” in Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security , 2022, pp. 29–33
work page 2022
-
[2]
Program synthesis with large language models,
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021
arXiv 2021
-
[4]
Codegen: An open large language model for code with multi-turn program synthesis,
E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022
arXiv 2022
-
[5]
Treegen: A tree-based transformer architecture for code generation,
Z. Sun, Q. Zhu, Y . Xiong, Y . Sun, L. Mou, and L. Zhang, “Treegen: A tree-based transformer architecture for code generation,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 05, 2020, pp. 8984–8991
work page 2020
-
[6]
Fast and memory-efficient neural code completion,
A. Svyatkovskiy, S. Lee, A. Hadjitofi, M. Riechert, J. V . Franco, and M. Allamanis, “Fast and memory-efficient neural code completion,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021, pp. 329–340
work page 2021
-
[7]
Code prediction by feeding trees to transformers,
S. Kim, J. Zhao, Y . Tian, and S. Chandra, “Code prediction by feeding trees to transformers,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 150–162
work page 2021
-
[8]
M2ts: Multi-scale multi-modal approach based on transformer for source code summarization,
Y . Gao and C. Lyu, “M2ts: Multi-scale multi-modal approach based on transformer for source code summarization,” in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, 2022, pp. 24–35
2022
Show all 24 references
-
[9]
Automated program repair in the era of large pre-trained language models,
C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1482–1494
2023
-
[10]
Evaluating large language models trained on code,
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021
2021 arXiv
-
[11]
Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,
J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” Advances in Neural Information Processing Systems , vol. 36, 2024
2024
-
[12]
Learning performance-improving code edits,
A. Shypula, A. Madaan, Y . Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, “Learning performance-improving code edits,” arXiv preprint arXiv:2302.07867 , 2023
2023 arXiv
-
[13]
Evalu- ating language models for efficient code generation,
J. Liu, S. Xie, J. Wang, Y . Wei, Y . Ding, and L. Zhang, “Evalu- ating language models for efficient code generation,” arXiv preprint arXiv:2408.06450, 2024
2024 arXiv
-
[14]
Asleep at the keyboard? assessing the security of github copilot’s code contribu- tions,
H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contribu- tions,” in 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2022, pp. 754–768
2022
-
[15]
An empirical study of code smells in transformer-based code generation techniques,
M. L. Siddiq, S. H. Majumder, M. R. Mim, S. Jajodia, and J. C. Santos, “An empirical study of code smells in transformer-based code generation techniques,” in 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) . IEEE, 2022, pp. 71–82
2022
-
[16]
Purple llama cyberseceval: A secure coding benchmark for language models,
M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana et al. , “Purple llama cyberseceval: A secure coding benchmark for language models,” arXiv preprint arXiv:2312.04724, 2023
2023 arXiv
-
[17]
Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models,
M. Bhatt, S. Chennabasappa, Y . Li, C. Nikolaidis, D. Song, S. Wan, F. Ahmad, C. Aschermann, Y . Chen, D. Kapilet al., “Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models,” arXiv preprint arXiv:2404.13161 , 2024
2024 arXiv
-
[18]
Large language models for code: Security hardening and adversarial testing,
J. He and M. Vechev, “Large language models for code: Security hardening and adversarial testing,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , 2023, pp. 1865–1879
2023
-
[19]
Instruction tuning for secure code generation,
J. He, M. Vero, G. Krasnopolska, and M. Vechev, “Instruction tuning for secure code generation,” arXiv preprint arXiv:2402.09497 , 2024
2024 arXiv
-
[20]
Training language models to follow instructions with human feedback,
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022
2022
-
[21]
Measuring coding challenge competence with apps,
D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Songet al., “Measuring coding challenge competence with apps,” arXiv preprint arXiv:2105.09938 , 2021
2021 arXiv
-
[22]
Livecodebench: Holistic and contamination free evaluation of large language models for code,
N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024
2024 arXiv
-
[24]
Cwe - about cwe,
“Cwe - about cwe,” https://cwe.mitre.org/about/index.html, (Accessed on 11/18/2024)
2024
-
[25]
Codeql documentation,
“Codeql documentation,” https://codeql.github.com/docs/, (Accessed on 11/18/2024)
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.