Pith. sign in

REVIEW 4 major objections 5 minor 70 references

Understanding and Improving Model Editing for Secure Code Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that model editing—surgically updating a small parameter subset—can harden code LLMs against known vulnerability types more effectively than inference-time steering like CoSec, and that a post-edit refinement stage…

desk verdict A solid first systematic comparison of model editing vs. inference-time hardening for secure code generation, with a useful SafeEdit refinement, but the headline functional-recovery numbers rest on an unverified decontamination assumption. read the letter →

arxiv 2608.06848 v1 pith:A32QY2W6 submitted 2026-08-07 cs.CR cs.SE

Weifeng Sun , Quanjun Zhang , Yuchen Chen , Chengran Yang , Gou Tan , David Lo This is my paper · ORCID
classification cs.CRcs.SE
keywords modeleditingsecurecodegenerationsecurityhardeningfunctionalcorrectnessgeneralizationSafeEditLLMvulnerabilitymitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that model editing, which surgically updates a small subset of an LLM's parameters, can serve as a practical model-level hardening mechanism for secure code generation, and that it beats the standard inference-time approach CoSec on known vulnerability types while adding no per-query runtime cost. Across six to eight open-weight LLMs, the evaluated editing methods raised security ratios by 15–25% over vanilla models on seen CWE categories, and the gains survived prompt perturbations, whereas CoSec often delivered smaller or even negative gains. The paper also finds a trade-off: editing transfers unreliably to unseen vulnerability types and degrades functional correctness. To address that, it proposes SafeEdit, a post-edit refinement that fine-tunes the edited model on normal coding tasks while applying an L2 penalty that keeps the edited parameters pinned to their post-edit values, recovering 11.73/13.70/15.50 Pass@1 points over UltraEdit at the three tested temperatures while largely preserving security. If correct, this gives developers a one-time hardening option with no auxiliary-model overhead and concrete guidance on where edits should be placed.

What carries the argument

The central objects are the security editing instance $M_i = (X_i, \hat{Y}_i, Y_i, \text{CWE}_i)$ and the edited model $f_e = \mathcal{A}(f, \mathcal{E})$ produced by an editing algorithm. The load-bearing mechanism is SafeEdit's edit-aware regularization, expressed as $\mathcal{L} = \mathcal{L}_{\text{std}} + \mu \|\theta_{\text{edit}} - \theta^e_{\text{edit}}\|_2^2$, which anchors the exact parameter subset modified by UltraEdit to its post-edit values while fine-tuning on functional code data. This is what lets functional recovery proceed without erasing injected security knowledge. The three evaluated editing methods differ in where they write: UltraEdit computes a closed-form parameter shift at an editable layer, DINM updates gradient-attributed neurons, and DEFER* routes per-token between base and adapter predictions.

What would settle it

Search the Code Evol-Instruct corpus for near-duplicates of the 164 HumanEval prompts; finding any overlap would mean the reported Pass@1 recoveries are inflated. Separately, rerun SafeEdit with $\mu=0$ to see whether removing the edit-anchoring term makes the security gains collapse; if security survives without the regularization term, the proposed mechanism is not the reason edits are preserved.

Watch

Extended reading notes

Core claim

The paper's central claim is that localized parameter edits can inject security knowledge more effectively than inference-time token steering, but the injection is not free: it trades functional correctness for security, and the balance depends on where and how the edits are applied. On seen CWEs, the best editing methods outperformed CoSec—DINM achieved the largest average security-ratio gain of +11.27% versus CoSec's +1.46%—and most editing gains persisted under prompt perturbations, while CoSec often became brittle. On 17 unseen CWE categories, transfer was uneven: UltraEdit's sparse up-projection edits sometimes bound the injected behavior to edited contexts, while DINM's neuron-level updates transferred more consistently. Functional correctness dropped for both paradigms, more so for model editing, especially under cumulative edits. To mitigate this, SafeEdit combines standard functional tuning on instruction-style code data with edit-aware regularization that anchors exactly the parameter subset UltraEdit changed, preventing functional recovery from overwriting the injected security behavior. Across eight LLMs, SafeEdit improved Pass@1 over UltraEdit by 11.73/13.70/15.50 percentage points at T=0.1/0.4/0.8 while keeping security within a few points, and it beat CoSec on both security and correctness; CodeGuard+ confirmed the joint secure-and-correct gains.

Load-bearing premise

The coding-examples data SafeEdit is fine-tuned on is assumed not to overlap with the HumanEval benchmark used to measure functional recovery; the paper checks it against the security editing set but not against HumanEval itself.

Editorial extensions

If this is right

  • Model editing gives open-weight model owners a one-time hardening path that avoids CoSec's auxiliary model and its 1.4–3.2 times inference-latency overhead, with editing costs as low as 50–100 seconds for methods like UltraEdit.
  • SafeEdit lifts Pass@1 over UltraEdit by 11.73/13.70/15.50 percentage points at T=0.1/0.4/0.8 while holding security within a few points, so most of the functional cost of security editing is recoverable.
  • Stacking CoSec on top of SafeEdit improves security further under stochastic decoding without sacrificing correctness, so parameter-level editing and inference-time steering are complementary rather than mutually exclusive.
  • Editing depth, parameter location, suffix length, and dataset size materially change the security–correctness balance, so a fixed editing recipe cannot be assumed safe across models or tasks.
  • Security gains on seen vulnerabilities do not reliably transfer to unseen CWE categories, which caps the practical value of editing as a standalone defense against novel vulnerability classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper tests the SafeEdit–CoSec combination on only two backbones; a natural extension not explored in the paper is to run the combination across all eight models and on unseen CWEs, where the two mechanisms might also patch each other's generalization gaps.
  • The finding that DINM transfers better to unseen CWEs while UltraEdit binds more tightly to seen contexts suggests a design heuristic for future editing methods: choose parameterizations that encode broader security semantics when the threat model includes novel vulnerability types.
  • Because SafeEdit's regularization term is agnostic to which parameters the base editor changed, the same recipe could be applied after more aggressive editors like DINM; if stronger anchoring suffices, the trade-off frontier may shift further toward security.
  • The security ratios are static-analysis proxies rather than proofs, so a deployment-grade pipeline would still need a second oracle or runtime checks for vulnerability classes not covered by the editing data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a systematic empirical comparison of model editing (UltraEdit, DINM, DEFER*) with inference-time hardening (CoSec) for secure code generation. It evaluates security effectiveness on seen CWEs, robustness to prompt perturbations, generalization to unseen CWEs, and functional correctness on HumanEval, using six to eight open-weight LLMs. The authors report that model editing provides larger security gains than CoSec on seen vulnerability types but with functional regressions and unreliable transfer to unseen CWEs. They then propose SafeEdit, a post-edit refinement combining functional tuning on Code Evol-Instruct with an edit-aware L2 regularizer that anchors the edited parameters, and report that it improves Pass@1 over UltraEdit by 11.73/13.70/15.50 percentage points while largely preserving security. The paper also includes ablations of editing depth, suffix length, injection location, dataset size, and an efficiency comparison.

Significance. If the central results hold, this is a practically useful study: it provides the first systematic comparison of model editing versus inference-time steering for code security, identifies a clear security–correctness trade-off, and proposes a refinement method with a plausible mechanism (anchor the edited parameter subset during functional tuning). The study is strengthened by the use of multiple LLM families, standard benchmarks (HumanEval, CodeGuard+, MultiPL-E), statistical tests over eight models, manual validation of a sample of CodeQL outputs, and a promised replication package. The main caveats are that the SafeEdit functional-correctness claim rests on an unverified decontamination assumption about the Code Evol-Instruct tuning data, the headline improvement over UltraEdit is not directly visible in the main table, and the abstract's 15%–25% security-gain range is above the reported average gains. These issues are addressable and should be fixed before publication.

major comments (4)
  1. [Section 3.4, Eq. (5)] SafeEdit's headline Pass@1 gains over UltraEdit are measured on HumanEval after fine-tuning on Code Evol-Instruct, but the text only states that Code Evol-Instruct has 'no overlap with E' (the security editing set). This does not establish disjointness from HumanEval or from the MultiPL-E translations used in Section 4.3. WizardCoder's Evol-Instruct is a general instruction-tuning collection, so without a decontamination analysis the reported +11.73/+13.70/+15.50 percentage-point improvements may be inflated by training on HumanEval-like tasks. Please add an explicit overlap check (exact and near-duplicate matching) between the SafeEdit tuning set and the functional-correctness benchmarks, or otherwise demonstrate that the tuning data cannot bias the Pass@1 measurement.
  2. [Section 3.4.1, Table 6] Table 6 reports only Vanilla, CoSec, and SafeEdit, yet the paper's central SafeEdit claims are relative to UltraEdit (e.g., Pass@1 improvements of +11.73/+13.70/+15.50 and SR changes of +7.44/-0.28/-3.48 percentage points). UltraEdit is absent from this table, so the claimed deltas cannot be verified from the presented data for any of the eight models. Please add UltraEdit rows (and ideally the other editing baselines) for the same models and temperatures, or provide an aligned appendix table.
  3. [Abstract and Section 3.1.1, Table 2] The abstract and RQ1 answer state that model editing improves security ratios by 15%–25% over vanilla models, but the averages reported in Table 2 and Section 3.1.1 are +11.27% for DINM and +6.54% for UltraEdit. The 15%–25% range appears to reflect selected per-configuration cells rather than the overall result. Please reword the abstract and RQ1 summary to report the average gains and the range across configurations, or explicitly justify the range as a per-configuration best case.
  4. [Section 2.5.2, RQ1.2] The perturbation robustness evaluation is built from perturbations of a single CWE-089 0-Py scenario, yet the abstract and RQ1.2 answer generalize this to a claim that security gains 'remain stable under prompt perturbations.' As presented, this robustness result is established for one vulnerability context only. Either expand the perturbation evaluation to additional CWEs or soften the claim to reflect the single-scenario basis of the robustness analysis.
minor comments (5)
  1. [Figure 1 vs. Section 2.5.2] Figure 1 labels the benchmark as 'HumanEval (163 Tasks)', while Section 2.5.2 states that HumanEval contains 164 Python tasks; please correct the inconsistency.
  2. [Table 6] The 95% confidence interval for StarCoder-7B at T=0.1 has an upper bound of 102.4, which is impossible for a security ratio; clip intervals to [0,100] or explain the interval computation.
  3. [Table 2 caption] The 'Avg.' columns in Table 2 would be clearer if they included the relative-change values in the same format as the per-temperature cells, and if the caption stated whether the average is computed over per-temperature relative gains or over the aggregated security ratios.
  4. [Section 4.6, Table 14] SafeEdit's offline cost on DeepSeek-6.7B is 176,055 seconds (roughly 49 GPU-hours), which is a substantial one-time expense; the discussion in Section 4.6 should mention this magnitude explicitly rather than only saying the cost is paid once before deployment.
  5. [Section 4.2] The unseen-CWE evaluation of SafeEdit is reported for only two of the eight models (LLaMA3.1-8B and LLaMA3.2-3B); please state this limitation in the text so readers do not infer that unseen-CWE generalization was measured across the full model set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's empirical claims rest on external benchmarks and the proposed method's loss does not encode the target evaluation metrics.

full rationale

I walked the paper's claimed derivation chain. The central empirical comparisons (model editing vs. CoSec on security, generalization, and functional correctness) are evaluated on external benchmarks: D_sec, D_per, unseen CWE scenarios, HumanEval, MultiPL-E, and CodeGuard+. These benchmarks are not used to fit SafeEdit's constants; the SafeEdit loss (L = L_std + mu * L_reg) anchors the UltraEdit-modified parameters to their post-edit values, which is a construction, not a prediction that reduces to an evaluation metric. The reported Pass@1 gains are measured on HumanEval and MultiPL-E, and the security ratios are measured by CodeQL/CodeGuard+, none of which enter the SafeEdit objective. The only self-citation signal is that CoSec [27,28] is co-authored by David Lo, a co-author of this paper; however, CoSec is used as an external baseline and is not invoked to justify any uniqueness theorem or to forbid alternative methods, so it is not load-bearing. The unverified assumption that Code Evol-Instruct tuning data does not overlap HumanEval is a data-contamination threat to validity, not circularity: the paper does not define SafeEdit's functional tuning success in terms of HumanEval scores, and no equation in the paper makes the claimed improvement true by construction. Therefore, no circular step meets the evidentiary bar, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The core contribution is empirical; the ledger contains one tuned hyperparameter and several benchmark-level assumptions rather than new formal entities. The most fragile assumption is the unverified non-overlap of SafeEdit's tuning data with HumanEval.

free parameters (2)
  • Regularization strength mu = 1e-3 (default; 1e-4 in sensitivity test)
    SafeEdit's final loss is L = L_std + mu * L_reg (Eq. 5); mu is a hand-chosen hyperparameter with sensitivity checked on only two models.
  • Target suffix length L = 50 tokens (default)
    Editing instances are constructed with a 50-token secure suffix; Section 4.8 ablates L=10/50/100 and shows security and correctness depend on it.
assumptions (4)
  • domain assumption CodeQL static analysis is an adequate oracle for classifying generated code as secure or vulnerable.
    Section 4.11 and Section 5 note CodeQL can produce false positives and negatives; the paper manually validated 40 outputs with 37 agreement, so security ratios are approximations.
  • domain assumption The Code Evol-Instruct tuning data does not overlap with the HumanEval tasks used to measure Pass@1.
    Section 3.4 states the data has no overlap with the security editing set E, but no HumanEval overlap check is reported; if overlap exists, Pass@1 gains are partly memorization.
  • domain assumption The seen/unseen CWE split is representative enough to support generalization conclusions.
    Section 2.5.1 reports disjoint CWE types and CodeQL rules with low embedding similarity, but only 29 unseen prompts support the transfer claim.
  • domain assumption Editing a small parameter subset does not affect capabilities outside the measured HumanEval, MultiPL-E, and CodeGuard+ tasks.
    The functional-correctness conclusions extrapolate from three coding benchmarks; other model capabilities are not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding and Improving Model Editing for Secure Code Generation." pith.science (2026). https://pith.science/paper/A32QY2W6

@misc{pith2026260806848,
  author       = {Pith},
  title        = {Pith review of: Understanding and Improving Model Editing for Secure Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A32QY2W6}},
  note         = {Machine review of arXiv:2608.06848}
}
read the original abstract

Large language models (LLMs) are widely used for code generation, yet they can reproduce vulnerable implementations learned from insecure training patterns. Prior work has mainly explored inference-time hardening, which reduces insecure generations without modifying the target model but relies on auxiliary components and adds runtime overhead. We conduct the first systematic study of model editing as a model-level hardening mechanism for secure code generation. We evaluate 3 state-of-the-art editing methods across diverse LLM families and compare them with CoSec, a representative inference-time approach, focusing on security, robustness, generalization, and functional correctness. Model editing yields larger security gains than CoSec on seen vulnerability types, improving security ratios by 15%-25% over vanilla models, with gains remaining stable under prompt perturbations. However, these improvements transfer unreliably to unseen vulnerabilities and can reduce functional correctness. To mitigate this trade-off, we propose SafeEdit, a post-edit refinement method combining functional tuning with edit-aware regularization. Across eight target LLMs, SafeEdit improves Pass@1 over UltraEdit by 11.73/13.70/15.50 percentage points at T=0.1/0.4/0.8 while largely preserving security. Compared with CoSec, it achieves relative security-ratio gains of 7.54%-12.04%. Additional evaluation on CodeGuard+ confirms improved joint secure-and-correct generation. SafeEdit and CoSec are also complementary, and their combination can further improve security while maintaining strong functional correctness. Overall, our results provide evidence-backed guidance for applying model editing to secure code generation.

Figures

Figures reproduced from arXiv: 2608.06848 by the authors.

Figure 1
Figure 1. Overview of our experimental pipeline for secure code generation hardening, comparing model editing [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Impact of editing depth on security effectiveness and functional correctness for UltraEdit and DINM. [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Effect of target suffix length 𝐿 on security effectiveness and functional correctness [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Impact of parameter injection location on security effectiveness and Pass@1. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Simplified examples illustrating CodeQL false-positive and false-negative cases. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 31 canonical work pages

  1. [1]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  2. [2]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation.IEEE Transactions on Software Engineering49, 7 (2023), 3675–3691

    work page 2023

  3. [3]

    Junkai Chen, Li Zhenhao, Hu Xing, and Xia Xin. 2024. Nlperturbator: Studying the robustness of code llms to natural language variations.ACM Transactions on Software Engineering and Methodology(2024)

    2024

  4. [4]

    Jiachi Chen, Qingyuan Zhong, Yanlin Wang, Kaiwen Ning, Yongkun Liu, Zenan Xu, Zhe Zhao, Ting Chen, and Zibin Zheng. 2024. Rmcbench: Benchmarking large language models’ resistance to malicious code. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 995–1006

    work page 2024

  5. [5]

    Mark Chen. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    arXiv 2021

  6. [6]

    Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions.Psychological bulletin114, 3 (1993), 494

    1993

  7. [7]

    CodeQL Team. 2026. CodeQL: The Libraries and Queries That Power Security Researchers Around the World. https://github.com/github/codeql. accessed 2026-07

    work page 2026

  8. [8]

    Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, et al. 2024. What’s wrong with your code generated by large language models? an extensive study.arXiv preprint arXiv:2407.06153(2024)

    arXiv 2024

Show all 70 references
  1. [9]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. InProceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long ...

    2022

  2. [10]

    Lishui Fan, Jiakun Liu, Zhongxin Liu, David Lo, Xin Xia, and Shanping Li. 2025. Exploring the capabilities of llms for code-change-related tasks.ACM Transactions on Software Engineering and Methodology34, 6 (2025), 1–36

    2025

  3. [11]

    Yanjun Fu, Ethan Baker, Yu Ding, and Yizheng Chen. 2024. Constrained decoding for secure code generation.arXiv preprint arXiv:2405.00218(2024)

    2024 arXiv

  4. [12]

    2025.The world’s most widely adopted AI developer tool

    GitHub. 2025.The world’s most widely adopted AI developer tool. Accessed: 2025. https://github.com/features/copilot

    2025

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al . 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    2024 arXiv

  6. [14]

    Jian Gu, Aldeida Aleti, Chunyang Chen, and Hongyu Zhang. 2023. Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generation.arXiv preprint arXiv:2312.05356(2023)

    2023 arXiv

  7. [15]

    Xiaojie Gu, Ziying Huang, Jia-Chen Gu, and Kai Zhang. 2025. UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models. arXiv:2505.14679 [cs.CL] https://arxiv.org/abs/2505.14679

    2025

  8. [16]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  9. [17]

    Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. 2024. CodeLMSec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models. In2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)...

    2024

  10. [18]

    Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors.Advances in Neural Information Processing Systems36 (2023), 47934–47959

    2023

  11. [19]

    Jingxuan He and Martin Vechev. 2023. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. 1865–1879

    2023

  12. [20]

    Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. Instruction tuning for secure code generation. arXiv preprint arXiv:2402.09497(2024)

    2024 arXiv

  13. [21]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. InInternational conference on machine learning. PMLR, 2790–2799

    2019

  14. [22]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    2024 arXiv

  15. [23]

    Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. 2024. Language models for code completion: A practical evaluation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. Proc. ACM Softw. En...

    2024

  16. [24]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.ACM computing surveys55, 12 (2023), 1–38

    2023

  17. [25]

    Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How secure is code generated by chatgpt?. In2023 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, 2445–2451

    2023

  18. [26]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of ...

    2017

  19. [27]

    Dong Li, Shanfu Shu, Meng Yan, Zhongxin Liu, Chao Liu, Xiaohong Zhang, and David Lo. 2025. Improving co-decoding based security hardening of code llms leveraging knowledge distillation.IEEE Transactions on Software Engineering (2025)

    2025

  20. [28]

    Dong Li, Meng Yan, Yaosheng Zhang, Zhongxin Liu, Chao Liu, Xiaohong Zhang, Ting Chen, and David Lo. 2024. CoSec: On-the-Fly security hardening of code LLMs via supervised co-decoding. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysi...

    2024

  21. [29]

    Hongyan Li, Weifeng Sun, and Meng Yan. 2026. Retrieval-Augmented Unit Test Suggestion Generation.ACM Transactions on Software Engineering and Methodology(2026)

    2026

  22. [30]

    Hongyan Li, Weifeng Sun, Meng Yan, Ling Xu, Qiang Li, Xiaohong Zhang, and Hongyu Zhang. 2025. Retrieval- Augmented Fine-Tuning for Improving Retrieve-and-Edit Based Assertion Generation.IEEE Transactions on Software Engineering(2025)

    2025

  23. [31]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  24. [32]

    Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2024. Pmet: Precise model editing in a transformer. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18564–18572

    2024

  25. [33]

    Xiaopeng Li, Shangwen Wang, Shasha Li, Jun Ma, Jie Yu, Xiaodong Liu, Jing Wang, Bin Ji, and Weimin Zhang. 2024. Model editing for llms4code: How far are we?arXiv preprint arXiv:2411.06638(2024)

    2024 arXiv

  26. [34]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct.arXiv preprint arXiv:2306.08568(2023)

    2023 arXiv

  27. [35]

    Vahid Majdinasab, Michael Joshua Bishop, Shawn Rasheed, Arghavan Moradidakhel, Amjed Tahir, and Foutse Khomh

  28. [36]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems35 (2022), 17359–17372

    2022

  29. [37]

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022. Mass-editing memory in a transformer.arXiv preprint arXiv:2210.07229(2022)

    2022 arXiv

  30. [38]

    MITRE Corporation. 2024. 2024 CWE Top 25 Most Dangerous Software Weaknesses. https://cwe.mitre.org/top25/. Accessed: October 29, 2025

    2024

  31. [39]

    Nero0113 and contributors. 2025. CoSec: On-the-Fly security hardening of code LLMs via supervised co-decoding. https://github.com/Nero0113/CoSec. Accessed: 2025-06-15

    2025

  32. [40]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  33. [41]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions.Commun. ACM68, 2 (2025), 96–105

    2025

  34. [42]

    Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language models. In2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2339–2356

    2023

  35. [43]

    Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. Cweval: Outcome-driven evaluation on functionality and security of llm code generation. In2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). IEEE, 33–40

    2025

  36. [44]

    Python Software Foundation. 2025. difflib: Helpers for Computing Deltas. https://docs.python.org/3/library/difflib.html

    2025

  37. [45]

    André Silva, Sen Fang, and Martin Monperrus. 2025. Repairllama: Efficient representations and fine-tuned adapters for program repair.IEEE Transactions on Software Engineering(2025)

    2025

  38. [46]

    Weifeng Sun, Naiqi Huang, Meng Yan, Li Huang, Zhongxin Liu, Xiao Liu, and David Lo. 2026. Cost-Effective Adversarial Attacks Against Code LLM with Model Attention.IEEE Transactions on Software Engineering(2026)

    2026

  39. [47]

    Weifeng Sun, Naiqi Huang, Meng Yan, Zhongxin Liu, Hongyan Li, Yan Lei, and David Lo. 2025. On-the-fly Generation- Quality Enhancement of Deep Code Models via Model Collaboration.ACM Transactions on Software Engineering and Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISST...

    2025

  40. [48]

    Weifeng Sun, Hongyan Li, Meng Yan, Yan Lei, and Hongyu Zhang. 2023. Revisiting and improving retrieval-augmented deep assertion generation. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1123–1135

    2023

  41. [49]

    Ali Tehrani, Arijit Bhattacharjee, Le Chen, Nesreen K Ahmed, Amir Yazdanbakhsh, and Ali Jannesari. 2024. Coderosetta: Pushing the boundaries of unsupervised code translation for parallel programming.Advances in Neural Information Processing Systems37 (2024), 100965–100999

    2024

  42. [50]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    2023 arXiv

  43. [51]

    Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jin- dong Wang, and Huajun Chen. 2024. Detoxifying large language models via knowledge editing.arXiv preprint arXiv:2403.14472(2024)

    2024 arXiv

  44. [52]

    Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen

  45. [53]

    Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024. Rlcoder: Reinforce- ment learning for repository-level code completion.arXiv preprint arXiv:2407.19487(2024)

    2024 arXiv

  46. [54]

    Wise: Rethinking the knowledge memory for lifelong model editing of large language models.Advances in Neural Information Processing Systems37 (2024), 53764–53797

    2024

  47. [55]

    Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. InProceedings of the 6th ACM SIGPLAN international symposium on machine programming. 1–10

    2022

  48. [56]

    Frank Wilcoxon. 1992. Individual comparisons by ranking methods. InBreakthroughs in statistics: Methodology and distribution. Springer, 196–202

    1992

  49. [57]

    Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang

  50. [58]

    Zhen Yang, Fang Liu, Zhongxing Yu, Jacky Wai Keung, Jia Li, Shuo Liu, Yifan Hong, Xiaoxue Ma, Zhi Jin, and Ge Li

  51. [59]

    Exploring and unleashing the power of large language models in automated code translation.Proceedings of the ACM on Software Engineering1, FSE (2024), 1585–1608

    2024

  52. [60]

    Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. 2024. A comprehensive study of knowledge editing for large language models.arXiv preprint arXiv:2401.01286(2024)

    2024 arXiv

  53. [61]

    Quanjun Zhang, Chunrong Fang, Siqi Gu, Ye Shang, Zhenyu Chen, and Liang Xiao. 2025. Large language models for unit testing: A systematic literature review.arXiv preprint arXiv:2506.15227(2025)

    2025 arXiv

  54. [62]

    Z. AI. 2025.Powerful AI assistant for developers. Accessed: 2025. https://codegeex.cn/en-US

    2025

  55. [63]

    Pengwei Zhan, Zhen Xu, Qian Tan, Jie Song, and Ru Xie. 2024. Unveiling the lexical sensitivity of LLMs: Combinatorial optimization for prompt enhancement.arXiv preprint arXiv:2405.20701(2024)

    2024 arXiv

  56. [64]

    Quanjun Zhang, Chunrong Fang, Tongke Zhang, Bowen Yu, Weisong Sun, and Zhenyu Chen. 2023. Gamma: Revisiting Template-based Automated Program Repair via Mask Prediction. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering. 535–547

    2023

  57. [65]

    Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models.arXiv preprint arXiv:2012.00363(2020). Received 2026-01-30; accepted 2026-06-25 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Ar...

    2020 arXiv

  58. [66]

    Quanjun Zhang, Chunrong Fang, Yang Xie, Yuxiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2026. A Systematic Literature Review on Large Language Models for Automated Program Repair. (2026). Just Accepted. doi:10.1145/3799693

    2026 doi

  59. [67]

    Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Shengcheng Yu, Weisong Sun, Yun Yang, and Zhenyu Chen

  60. [2023]

    Editing large language models: Problems, methods, and opportunities.arXiv preprint arXiv:2305.13172(2023)

    2023 arXiv

  61. [2024]

    In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)

    Assessing the security of github copilot’s generated code-a targeted replication study. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 435–444

  62. [2026]

    A Survey on Large Language Models for Software Engineering.Science China Information Sciences69, 4 (2026), 141102

    2026

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.