Pith. sign in

REVIEW 3 major objections 6 minor 54 references

SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SCGAgent, an agentic workflow that applies secure-coding guidelines one at a time and self-checks functionality with LLM-generated unit tests, raises Claude Sonnet-3.7's Func-Sec@1 on CWEval C tasks from 0.606 to 0.755, matching the best…

desk verdict A solid, honest engineering paper that shows a prompt-only agentic workflow can lift Sonnet-3.7's secure code rate on CWEval by ~15 points, but the headline gap needs error bars and the method works for only one of four base models. read the letter →

arxiv 2506.07313 v1 pith:2EYGPFYW submitted 2025-06-08 cs.CR

Rebecca Saul , Hao Wang , Koushik Sen , David Wagner This is my paper
classification cs.CR
keywords securecodegenerationagenticworkflowLLM-generatedunittestscodingguidelinesCWEfunctionalitypreservationreasoningmodelsCWEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an agentic workflow can give a non-reasoning LLM the secure-code-generation ability that so far has required reasoning models. SCGAgent works by predicting which CWEs a coding task is exposed to, retrieving hand-written secure-coding guidelines for those CWEs, applying each guideline to the generated code one at a time, and restoring any functionality lost along the way by having the LLM generate unit tests and jointly revise code or tests until the code passes. On CWEval's C tasks, SCGAgent raises Sonnet-3.7's Func-Sec@1 (the fraction of generated samples that are both functional and secure) from 0.606 (direct) and 0.619 (security reminder) to 0.755, while Func@1 only falls from 0.871 to 0.852. This makes the security of a non-reasoning model competitive with o4-mini plus a security reminder (0.748), suggesting that orchestration can substitute for expensive reasoning-model training.

What carries the argument

The load-bearing machinery is the enforce-functionality module (Algorithm 2) paired with one-guideline-at-a-time security revision (Algorithm 1). The enforce-functionality module generates unit tests from the task description, executes them against the code, and whenever a test fails asks the LLM whether the failure reflects a bug in the code or a flaw in the test; it then revises the code to pass the test or regenerates the tests, repeating up to three attempts. This joint revision step is what lets SCGAgent escape the functionality-security tradeoff: the paper's ablations show that security guidelines alone raise security to 0.699 but drop functionality to 0.806, that revising code against unchanged LLM-generated tests makes both fall (0.634 Func-Sec@1) because the tests are often wrong, and that permitting test regeneration recovers functionality (0.852) and raises Func-Sec@1 to 0.755. The security side is carried by a manually written set of concrete, safe-by-construction guidelines tied to CWEs, filtered by an LLM prediction of relevant CWEs and applied one guideline at a time to avoid overwhelming the model.

What would settle it

For any candidate base LLM, run SCGAgent's enforce-functionality loop on CWEval's C tasks and measure the fraction of samples that still fail the LLM's own unit tests when the loop terminates (the sum of rows 2 and 4 in Table 2). The paper's account predicts that SCGAgent beats that model's security-reminder baseline when this residual failure fraction is at or below Sonnet-3.7's 17%, and underperforms it when the fraction reaches 22–25% as it does for GPT-4o and o4-mini; a model with a low residual failure rate that still fails to beat its own baseline would contradict that account.

Watch

Extended reading notes

Core claim

SCGAgent's central claim is that secure and functional code generation can be decomposed into two reinforcing loops that any strong instruction-following LLM can run: a security loop that retrieves a small set of expert-written, safe-by-construction coding guidelines relevant to the predicted CWEs and rewrites the draft code to follow them, and a functionality loop that generates unit tests for the task, runs them, and when they fail has the LLM decide whether the bug lies in the code or in the tests, then revises the code or regenerates the tests accordingly. The paper's evidence is that on CWEval's 31 C tasks, this workflow raises Sonnet-3.7's Func-Sec@1 to 0.755 from 0.606 direct and 0.619 with a security reminder, at a functionality cost of roughly two points (0.871 to 0.852), and that the joint code-and-test revision step is what makes the difference: guidelines alone (A2) reach 0.699 Func-Sec@1 but drop Func@1 to 0.806, and adding code-only revision (A3) makes both worse (0.634) because LLM-generated tests are often wrong, whereas allowing test regeneration (A4) restores functionality and secures the gain. The paper also claims this matches or beats the best reasoning models tested (o4-mini with a security reminder scores 0.748), and that the approach does not generalize to every base LLM: with GPT-4o and o4-mini it underperforms a plain security reminder, which the paper attributes to worse unit tests and weaker code-vs-test judgment.

Load-bearing premise

Everything rests on the base LLM being able to generate unit tests that reflect the task's true requirements and to tell whether a failing test means the code is wrong or the test is wrong; when that judgment is unreliable, as with GPT-4o and o4-mini, SCGAgent produces worse results than a plain security reminder.

Editorial extensions

If this is right

  • A non-reasoning model plus this workflow can match the best reasoning-model baseline for functional-and-secure C code, so the security payoff of reasoning models can be reproduced by test-time orchestration.
  • Because SCGAgent only needs prompting, it can be applied to proprietary frontier models that cannot be fine-tuned, unlike training-based defenses such as SafeCoder.
  • The functionality loss of secure-code generation largely disappears for compatible base models: Func@1 moves from 0.871 (direct) to 0.852 (SCGAgent), versus 0.806 with guidelines alone.
  • Improving the LLM's CWE prediction and unit-test generation directly improves SCGAgent: substituting ground-truth CWEs and tests raises Func-Sec@1 to 0.849 per the paper's Table 4.
  • The workflow is extensible: new security guidelines can be added as new CWEs emerge, and the same modular design can be pointed at other languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical filter follows directly from Table 2: compute the residual test-failure rate after the enforce-functionality loop, and only deploy SCGAgent on base models whose rate is near Sonnet-3.7's 17%; the failing models (GPT-4o, o4-mini) sit at 22–25%.
  • The same two-loop pattern — apply a hard constraint, then re-verify against a self-generated oracle with an explicit path for correcting the oracle — could carry over to other code-quality constraints (performance, style, portability) where the base model's own judgment is the only available checker.
  • The low CWE recall (0.448) with still-positive security gains hints that guidelines may harden code beyond the benchmark's designated CWE; a per-task analysis separating 'correct CWE retrieved' from 'guideline still helped' would test whether retrieval or guideline generality drives the result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SCGAgent, an agentic workflow that augments a base LLM with manually written secure-coding guidelines and LLM-generated unit tests. The workflow first generates code and unit tests, enforces that the code passes those tests by revising either code or tests, predicts relevant CWEs, retrieves matching guidelines, then iteratively applies each relevant guideline and re-enforces functionality. On the CWEval C-language benchmark (31 tasks, 5 samples per task), the authors report that Sonnet-3.7 with SCGAgent reaches Func-Sec@1 = 0.755, up from 0.606 with direct prompting and 0.619 with a security reminder, while Func@1 drops only from 0.871 to 0.852. They further report that this matches or exceeds the best reasoning-model baseline (o4-mini with security reminder, 0.748). Ablations show that security guidelines alone improve security but hurt functionality, and that the joint code/test revision step is needed to restore functionality. The paper also reports that the approach does not generalize well to GPT-4o, o1, or o4-mini as base models, and that LLM-generated unit tests and CWE prediction are imperfect.

Significance. If the reported effect is real, the paper makes a useful contribution: it shows that an inference-time agentic workflow can close much of the gap between a non-reasoning frontier model and larger reasoning models on a security-sensitive code-generation benchmark, without fine-tuning. The evaluation uses CWEval's expert-written outcome tests for both functionality and security, so the headline metrics are not directly circular with the LLM-generated tests used inside SCGAgent. The ablation structure (Table 3) is clean and the case study in Section 5.4 is illustrative. The authors are also candid about the approach's failure with GPT-4o, o1, and o4-mini, and about the absence of adversarial-prompt evaluation. The main weaknesses are that the central quantitative claim rests on a single point estimate on 31 tasks with no uncertainty quantification, the full guideline set is not included, and the functionality-enforcement mechanism is shown to be imperfect even for the successful Sonnet-3.7 configuration. These issues do not invalidate the approach, but they currently make the strength of the headline claim difficult to assess.

major comments (3)
  1. [Section 5.1 / Table 5] The headline result—Sonnet-3.7 with SCGAgent raising Func-Sec@1 from 0.606 to 0.755 and matching o4-mini with a security reminder at 0.748—is reported as a point estimate with no confidence intervals, no per-task breakdown, and no correction for the fact that the five samples per task are not independent. A naive two-proportion standard error is roughly 5.2 percentage points, and within-task correlation in code-generation outcomes will reduce the effective sample size well below 155; under a modest intra-task correlation the 14.9-point gain is no longer clearly significant, and the 0.7-point gap versus o4-mini is far inside any plausible noise floor. Because the central claim of the paper is a comparative one, please add task-level bootstrapped confidence intervals or a mixed-effects analysis for both Func@1 and Func-Sec@1, and report the per-task pass/fail pattern so the reader can see how the aggregate result is distributed.
  2. [Section 4.1 / Table 1] The secure-coding guideline set is the central artifact of the method, but the paper shows only four example guidelines and states that the full set will be released only 'before publication.' Without the complete guideline list and the CWE-to-guideline mapping, the experiments cannot be independently replicated, and it is impossible for a reader to assess whether the hand-written guidelines were tailored to the 27 CWEs represented in CWEval. Please include the complete guideline set and the mapping in an appendix or provide a permanent reference to a released artifact in the submission.
  3. [Section 5.3 / Table 2 / Algorithm 2] Table 2 shows that even for the successful Sonnet-3.7 configuration, after the enforce-functionality step 17% of samples are not functional (rows 'True False' plus 'False False') and 20% of functional samples fail the LLM-generated tests (row 'False True'). The abstract's claim that SCGAgent 'preserves nearly 98% of the functionality' is just the ratio of two point estimates (0.852/0.871) and does not quantify the uncertainty around either number. More importantly, the paper does not measure the accuracy of the Algorithm 2 line-9 decision (revise code versus regenerate unit tests), which is the mechanism that is supposed to restore functionality. Since the functionality-preservation claim depends on this judgment, please report the per-decision accuracy or otherwise provide evidence that this decision is reliable enough to support the claim.
minor comments (6)
  1. [Section 5.2] The text says o4-mini improves by 7.4 percentage points from the security reminder, but Table 5 gives 0.748 - 0.671 = 7.7 points; please reconcile the numbers.
  2. [Appendix A / Table 3] The appendix prompt labels describe the CWE-description ablation as 'Ablation Experiment A2 in Table 3', but in Table 3 the CWE-description condition is A1 and the guideline condition is A2; the labels are inconsistent.
  3. [Table 5] In the o4-mini SCGAgent row at Pass@5, Func = 0.839 and Func-Sec = 0.839, yet the reported Func-Sec/Func is 0.999 instead of 1.000; please clarify whether this is a rounding artifact or an error.
  4. [Table 2] The table header and row labels are easy to misread; please restructure the table so that the conditioning variable and the values for each model are unambiguous.
  5. [Section 4.1] The sentence 'written to be as concrete as follow' appears to be missing a word; it should probably read 'as concrete as possible' or 'as follows.'
  6. [Section 2.3] The word 'procluding' appears in the discussion of training-based methods; the intended word is likely 'precluding.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SCGAgent's headline Func-Sec@1 results are measured against CWEval's independent expert-written unit tests, not against the LLM-generated tests used internally, and the paper contains no load-bearing self-citations.

full rationale

The paper's central claim — that SCGAgent with Sonnet-3.7 raises Func-Sec@1 from 0.606 (direct) and 0.619 (security reminder) to 0.755 — is an empirical measurement against an external benchmark. Section 5.1 states that 'CWEval comes with a suite of expert-written functionality and security unit tests for each task, which we use to assess SCGAgent's performance.' The reported metrics therefore do not reduce to SCGAgent's own outputs. The chief circularity-adjacent risk would be the LLM-generated unit tests used inside the enforce-functionality module: if the paper scored its code with those same generated tests, improvement could be partially self-fulfilling. But the paper instead treats those tests as an internal proxy whose fidelity it measures against ground truth (Table 2 reports 17-38% disagreement between LLM-generated test verdicts and true functionality), and the headline metrics come from CWEval's ground-truth tests. The security guidelines are a manual engineering input ('we manually develop a list of secure programming guidelines influenced by CERT standards and our own experiences', Section 4.1), not a parameter fitted to CWEval outcomes, so there is no fitted-input-called-prediction pattern. The ablations (Tables 3-4) isolate components and use oracle replacements as an upper-bound analysis, which is transparent rather than circular. The paper's comparisons with reasoning models are against external systems on the same external benchmark. Finally, the reference list contains no works by the paper's own authors, so no self-citation chain carries any load-bearing argument. Concerns such as possible benchmark-shaping of the manually written guidelines or the absence of confidence intervals in Table 5 are experimental-design and statistical-robustness issues, not cases where a claimed prediction is equivalent by construction to its inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new mathematical entities or fitted physical constants. Its load-bearing ingredients are hand-written inputs and model-behavior assumptions. The most fragile is the auxiliary reasoning capability of the base LLM, which the paper itself quantifies as CWE recall 0.448 and imperfect unit-test alignment (Table 2).

free parameters (1)
  • max_att (enforce-functionality retry limit) = 3
    Section 5.1: "We set the max_att parameter in SCGAgent's enforce-functionality module to 3." No sensitivity analysis is reported; the headline improvement could depend on this choice.
assumptions (4)
  • domain assumption Expert-written CWEval functionality and security unit tests accurately determine whether generated code is functional and secure.
    All reported Func and Func-Sec scores are computed against these tests (Section 5.1). If the tests mislabel samples, the headline numbers shift.
  • domain assumption The manually written secure coding guidelines, when followed, prevent or mitigate the target CWEs without breaking functionality.
    Section 4.1 states guidelines are "safe-by-construction" and "sufficient for security." The guidelines were curated by the authors, and the full list is not published, so external validation is not possible from the paper.
  • domain assumption The base LLM can reliably perform auxiliary reasoning: CWE prediction, guideline relevance checking, and deciding whether a failing test indicates a code bug or a test bug.
    Algorithm 2 line 9 and Algorithm 1 lines 5-8 rely on these judgments; Section 5.3 reports CWE recall 0.448 and ablations show poor auxiliary performance can hurt (GPT-4o, o4-mini).
  • domain assumption LLM-generated unit tests are a serviceable proxy for functional correctness during generation.
    The enforce-functionality module trusts these tests; Table 2 shows misalignment with ground truth in roughly a quarter of samples, and ablation A3 shows naive use of unit tests decreased functionality and security.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows." pith.science (2026). https://pith.science/paper/2EYGPFYW

@misc{pith2026250607313,
  author       = {Pith},
  title        = {Pith review of: SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2EYGPFYW}},
  note         = {Machine review of arXiv:2506.07313}
}
read the original abstract

Large language models (LLMs) have seen widespread success in code generation tasks for different scenarios, both everyday and professional. However current LLMs, despite producing functional code, do not prioritize security and may generate code with exploitable vulnerabilities. In this work, we propose techniques for generating code that is more likely to be secure and introduce SCGAgent, a proactive secure coding agent that implements our techniques. We use security coding guidelines that articulate safe programming practices, combined with LLM-generated unit tests to preserve functional correctness. In our evaluation, we find that SCGAgent is able to preserve nearly 98% of the functionality of the base Sonnet-3.7 LLM while achieving an approximately 25% improvement in security. Moreover, SCGAgent is able to match or best the performance of sophisticated reasoning LLMs using a non-reasoning model and an agentic workflow.

Figures

Figures reproduced from arXiv: 2506.07313 by the authors.

Figure 1
Figure 1. By pairing an agentic workflow (SCGAgent) and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A high-level overview of SCGAgent. The task [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Standard prompt from the CWEval dataset. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Functionality@1 scores for different LLMs [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Sonnet-3.7 reasoning about unit test correctness. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Case study of SCGAgent with a prompt susceptible to CWE-918, Server-Side Request Forgery. Blue frames [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Insecure code generated by Sonnet-3.7, directly prompted. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Secure code generated by Sonnet-3.7 with SCGA [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 22 canonical work pages

  1. [1]

    Openai o1 system card,

    OpenAI, “Openai o1 system card,” 2024. [Online]. Available: https://arxiv.org/abs/2412.16720

    arXiv 2024

  2. [2]

    Claude 3.7 sonnet and claude code,

    Anthropic, “Claude 3.7 sonnet and claude code,” https://www.anthropic.com/news/claude-3-7-sonnet, 2025, accessed: 2025-04-23

    work page 2025

  3. [3]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12948

    arXiv 2025

  4. [4]

    Q3 earnings call: CEO’s remarks,

    S. Pichai, “Q3 earnings call: CEO’s remarks,” https://blog.google/inside-google/message-ceo/alphabet-earnings- q3-2024/

    work page 2024

  5. [5]

    Assessing the Security of GitHub Copilot’s Generated Code - A Targeted Replication Study ,

    V . Majdinasab, M. J. Bishop, S. Rasheed, A. Moradidakhel, A. Tahir, and F. Khomh, “ Assessing the Security of GitHub Copilot’s Generated Code - A Targeted Replication Study ,” in2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). Los Alamitos, CA, USA: IEEE Computer Society, Mar. 2024, pp. 435–444. [Online]. Avai...

    arXiv 2024

  6. [6]

    Purple llama cyberseceval: A secure coding benchmark for language models,

    M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y . Kozyrakis, D. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V . V ontimitta, S. Whitman, and J. Saxe, “Purple llama cyberseceval: A secure coding benchmark for language models,” 2023. [Online]. Availa...

    arXiv 2023

  7. [7]

    The formai dataset: Generative ai in software security through the lens of formal verification,

    N. Tihanyi, T. Bisztray, R. Jain, M. A. Ferrag, L. C. Cordeiro, and V . Mavroeidis, “The formai dataset: Generative ai in software security through the lens of formal verification,” inProceedings of the 19th International Conference on Predictive Models and Data Analytics in Software Engineering, ser. PROMISE 2023. New York, NY , USA: Association for Comp...

    work page arXiv 2023

  8. [8]

    Cweval: Outcome-driven evaluation on functionality and security of llm code generation,

    J. Peng, L. Cui, K. Huang, J. Yang, and B. Ray, “Cweval: Outcome-driven evaluation on functionality and security of llm code generation,” 2025. [Online]. Available: https://arxiv.org/abs/2501.08200

    arXiv 2025

Show all 54 references
  1. [9]

    Seccodeplt: A unified platform for evaluating the security of code genai,

    Y . Yang, Y . Nie, Z. Wang, Y . Tang, W. Guo, B. Li, and D. Song, “Seccodeplt: A unified platform for evaluating the security of code genai,” 2024. [Online]. Available: https://arxiv.org/abs/2410.11096

    2024

  2. [10]

    Magic Quad- rant for AI Code Assistants

    A. Batchu, P. Walsh, M. Brasier, and H. Khandabattu, “Magic Quad- rant for AI Code Assistants.”

  3. [11]

    Prompting techniques for secure code generation: A systematic investigation,

    C. Tony, N. E. D. Ferreyra, M. Mutas, S. Dhiff, and R. Scandariato, “Prompting techniques for secure code generation: A systematic investigation,”ArXiv, vol. abs/2407.07064, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:271064536

    2024 arXiv

  4. [12]

    Instruction tuning for secure code generation,

    J. He, M. Vero, G. Krasnopolska, and M. Vechev, “Instruction tuning for secure code generation,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    2024

  5. [13]

    (2024) Aider llm leaderboards

    Aider. (2024) Aider llm leaderboards. [Online]. Available: https://aider.chat/docs/leaderboards/

    2024

  6. [14]

    Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings,

    S. Quan, J. Yang, B. Yu, B. Zheng, D. Liu, A. Yang, X. Ren, B. Gao, Y . Miao, Y . Fenget al., “Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings,”arXiv preprint arXiv:2501.01257, 2025

    2025 arXiv

  7. [15]

    SWE-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=...

    2024

  8. [16]

    Constrained decoding for secure code generation,

    Y . Fu, E. Baker, Y . Ding, and Y . Chen, “Constrained decoding for secure code generation,”arXiv preprint arXiv:2405.00218, 2024

    2024 arXiv

  9. [17]

    Github copilot features,

    GitHub, “Github copilot features,” https://github.com/features/copilot, 2025, accessed: 2025-04-23

    2025

  10. [18]

    Quantifying github copilot’s impact on developer productivity and happiness,

    E. Kalliamvakou, “Quantifying github copilot’s impact on developer productivity and happiness,” https://github.blog/news- insights/research/research-quantifying-github-copilots-impact-on- developer-productivity-and-happiness/, 2022, accessed: 2025-04-23

    2022

  11. [19]

    The llama 3 herd of models,

    A. . M. Llama Team, “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    2024 arXiv

  12. [20]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Ka- plan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavar- ian, C. Win...

    2021

  13. [21]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y . Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.-H. Yee, L. K. Umapat...

    2023 arXiv

  14. [22]

    Code llama: Open foundation models for code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D ´efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier...

    2024 arXiv

  15. [23]

    Magicoder: Empowering code generation with OSS-instruct,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Empowering code generation with OSS-instruct,” inProceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 235. PMLR, 21–27 Jul 2024, pp. 52 632–52 657...

    2024

  16. [24]

    Wizardcoder: Empowering code large language models with evol-instruct,

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,”arXiv preprint arXiv:2306.08568, 2023

    2023 arXiv

  17. [25]

    Code alpaca: An instruction-following llama model for code generation,

    S. Chaudhary, “Code alpaca: An instruction-following llama model for code generation,” https://github.com/sahil280114/codealpaca, 2023

    2023

  18. [26]

    RLCoder: Reinforcement Learning for Repository-Level Code Completion ,

    Y . Wang, Y . Wang, D. Guo, J. Chen, R. Zhang, Y . Ma, and Z. Zheng, “ RLCoder: Reinforcement Learning for Repository-Level Code Completion ,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). Los Alamitos, CA, USA: IEEE Computer Society, May 2025, ...

    2025

  19. [27]

    R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models,

    K. Deng, J. Liu, H. Zhu, C. Liu, J. Li, J. Wang, P. Zhao, C. Zhang, Y . Wu, X. Yinet al., “R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models,”arXiv preprint arXiv:2406.01359, 2024

    2024 arXiv

  20. [28]

    The stack: 3 tb of permissively licensed source code,

    D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y . Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries, “The stack: 3 tb of permissively licensed source code,” 2022. [Online]. Available: https://arxiv.org/abs/2211.15533

    2022 arXiv

  21. [29]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  22. [30]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al., “Training compute-optimal large language models,”arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  23. [31]

    Trading inference-time compute for adversarial robustness,

    W. Zaremba, E. Nitishinskaya, B. Barak, S. Lin, S. Toyer, Y . Yu, R. Dias, E. Wallace, K. Xiao, J. Heideckeet al., “Trading inference-time compute for adversarial robustness,”arXiv preprint arXiv:2501.18841, 2025

    2025 arXiv

  24. [32]

    s1: Simple test-time scaling,

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Cand `es, and T. Hashimoto, “s1: Simple test-time scaling,”arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  25. [33]

    Asleep at the keyboard? assessing the security of github copilot’s code contributions,

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in2022 IEEE Symposium on Security and Privacy (SP), 2022, pp. 754–768

    2022

  26. [34]

    Large language models for code: Security hardening and adversarial testing,

    J. He and M. Vechev, “Large language models for code: Security hardening and adversarial testing,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 1865–1879...

    2023

  27. [35]

    Enhancing security of ai-based code synthesis with github copilot via cheap and efficient prompt-engineering,

    J. Res, I. Homoliak, M. Pere ˇs´ıni, A. Smr ˇcka, K. Malinka, and P. Hanacek, “Enhancing security of ai-based code synthesis with github copilot via cheap and efficient prompt-engineering,” 2024. [Online]. Available: https://arxiv.org/abs/2403.12671

    2024 arXiv

  28. [36]

    Llm security guard for code,

    A. Kavian, M. M. Pourhashem Kallehbasti, S. Kazemi, E. Firouzi, and M. Ghafari, “Llm security guard for code,” inProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’24. New York, NY , USA: Association for Computing ...

    2024

  29. [37]

    Securityeval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques,

    M. L. Siddiq and J. C. S. Santos, “Securityeval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques,” ser. MSR4P&S 2022. New York, NY , USA: Association for Computing Machinery, 2022, p. 29–33. [Online]. Available: https://doi.o...

    2022

  30. [38]

    Incoder: A generative model for code infilling and synthesis,

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W. tau Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” 2023. [Online]. Available: https://arxiv.org/abs/2204.05999

    2023 arXiv

  31. [39]

    GitHub, “Codeql,” https://codeql.github.com, 2025, accessed: 2025- 04-23

    2025

  32. [40]

    Bandit: A security linter from pycqa,

    PyCQA, “Bandit: A security linter from pycqa,” https://github.com/PyCQA/bandit?tab=readme-ov-file, 2025, accessed: 2025-04-23

    2025

  33. [41]

    Autosafecoder: A multi-agent framework for securing llm code generation through static analysis and fuzz testing,

    A. Nunez, N. T. Islam, S. K. Jha, and P. Najafirad, “Autosafecoder: A multi-agent framework for securing llm code generation through static analysis and fuzz testing,” 2024. [Online]. Available: https://arxiv.org/abs/2409.10737

    2024 arXiv

  34. [42]

    Llms cannot reliably identify and reason about security vulnera- bilities (yet?): A comprehensive evaluation, framework, and bench- marks,

    S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini, “Llms cannot reliably identify and reason about security vulnera- bilities (yet?): A comprehensive evaluation, framework, and bench- marks,” in2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024, pp. 862–880

    2024

  35. [43]

    Fuzzing: a survey,

    J. Li, B. Zhao, and C. Zhang, “Fuzzing: a survey,”Cybersecurity, vol. 1, pp. 1–13, 2018

    2018

  36. [44]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” inAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tom...

    2024

  37. [45]

    Toolformer: language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess ´ı, R. Raileanu, M. Lomeli, E. Ham- bro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: language models can teach themselves to use tools,” inProceedings of the 37th International Conference on Neural Information Processing Systems,...

    2023

  38. [46]

    Building effective agents,

    Anthropic, “Building effective agents,” https://www.anthropic.com/engineering/building-effective-agents

  39. [47]

    Sei CERT c coding standard,

    SEI CERT, “Sei CERT c coding standard,” https://wiki.sei.cmu.edu/confluence/display/c/SEI+CERT+C+Coding+Standard, Pittsburgh, PA, 2016, version 3.1 (Latest stable release)

    2016

  40. [48]

    Common weakness enumeration (cwe),

    MITRE Corporation, “Common weakness enumeration (cwe),” Online, 2025, accessed: 26 April, 2025. [Online]. Available: https://cwe.mitre.org/

    2025

  41. [49]

    Multi-language unit test generation using llms,

    R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha, “Multi-language unit test generation using llms,”arXiv preprint arXiv:2409.03093, 2024

    2024 arXiv

  42. [50]

    No more manual tests? evaluating and improving chatgpt for unit test generation,

    Z. Yuan, Y . Lou, M. Liu, S. Ding, K. Wang, Y . Chen, and X. Peng, “No more manual tests? evaluating and improving chatgpt for unit test generation,” 2024. [Online]. Available: https://arxiv.org/abs/2305.04207

    2024 arXiv

  43. [51]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the F oundations of Software Engineering, ser. FSE 2024. New York, NY , USA: Association for Comp...

    2024

  44. [52]

    Generate code (Algorithm 1 Line 2):“I have a task below that is given by a oblivious user

    Prompts The prompts for the various stages of SCGAgent are as follows. Generate code (Algorithm 1 Line 2):“I have a task below that is given by a oblivious user. Please generate the code corresponding to the description{task description}. The code should be a single block wrap...

  45. [53]

    Full Results Our full results are available in Table 5

  46. [54]

    Welcome, <username>!

    Case Study Code Samples Figures 7 and 8 show the full code samples for the CWE-79 case study discussed in Section 5.4. Figure 7 show the insecure code generated by Sonnet-3.7 when directly prompted, and Figure 8 depicts the secure version of the same program generated by Sonne...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.