On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code

Md Imran Hossen; Xiali Hei

REVIEW 4 major objections 4 minor 31 references

On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DegradePrompter, a black-box attack that appends short misleading code snippets to coding prompts, cuts functional correctness by 12%–34% on open-source code models and by 3%–24% on commercial models.

desk verdict A simple black-box attack worth knowing about, but the open-vs-closed robustness finding is an artifact of the CDRA normalization. read the letter →

arxiv 2411.19508 v1 pith:3D7SEQW6 submitted 2024-11-29 cs.SE cs.CR

Md Imran Hossen , Xiali Hei This is my paper · ORCID

classification cs.SEcs.CR

keywords adversarialrobustnessinstruction-tunedCodeLLMsDegradePrompterblack-boxattackpass@1CDRAguidedpromptingdefensegeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DegradePrompter is a black-box attack that tests how easily instruction-tuned code models can be misled: it asks a powerful helper model, GPT-4-turbo, to append a one-to-three-line misleading code snippet to a clean coding problem. The paper claims these small additions reduce pass@1 functional correctness by 12%–34% across five open-source models and by 3%–24% across three commercial models on HumanEval and MBPP. The importance is practical: AI coding assistants are being used in real software workflows, and the finding suggests subtle prompt changes can silently degrade the code they produce. The paper also tests a simple inference-time defense, guided prompting, and reports that it restores performance for some models but not others.

What carries the argument

The carrier of the argument is DegradePrompter, a black-box procedure in which an oracle language model $O$ (GPT-4-turbo) receives a coding problem and returns a short adversarial suffix $s_{adv}$ that is appended to the problem to form $x_{adv}$. The suffix is supposed to be contextually relevant, syntactically valid Python, and at most three lines. The paper measures the attack's force with CDRA, the fractional drop in pass@1 from clean to adversarial prompts, and evaluates the defense with ANR, the fraction of the lost performance that guided prompting restores. The same machinery is used across eight models and two benchmarks, so the empirical pattern—open-source models degrade more than commercial ones—is carried by these two metrics applied to the oracle-generated suffixes.

What would settle it

Re-run the HumanEval evaluation keeping only adversarial prompts whose cosine distance to the clean prompt is actually below $\epsilon = 0.1$, discarding any that exceed it; if the filtered CDRA drops to the handcrafted baseline, the reported degradation is driven by unbounded prompt changes rather than the bounded adversarial perturbation the paper claims.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that instruction-tuned Code LLMs are substantially less robust to adversarial prompt perturbations than their clean benchmark scores suggest. Using DegradePrompter, an oracle LLM generates adversarial suffixes $s_{adv} = O(x)$ that are appended to the original prompt, and the correctness of the generated code is compared through pass@1 before and after the perturbation. The result is expressed as CDRA, the fractional drop in pass@1 under attack: open-source models show degradation from about 12% to 34%, while commercial models degrade from about 3% to 24%. The attack transfers across model families and does not require logits or gradients; a guided-prompting defense partially neutralizes the attack on some models, with ANR values above 50% on HumanEval for most models but negative for some models on MBPP.

Load-bearing premise

The method assumes that a single call to GPT-4-turbo reliably produces a small, contextually relevant suffix that stays within the stated perturbation budget, even though the budget is only checked after generation and is not enforced during creation.

Editorial extensions

If this is right

Open-source code assistants can silently fail on roughly a third of otherwise-solvable tasks when the prompt contains a short misleading snippet.
Black-box robustness evaluation is feasible without access to model internals: a single call to a strong helper model generates transferable adversarial suffixes.
Model family matters more than raw size: DeepSeek-Coder-Instruct is repeatedly the most resilient open-source family, while larger CodeLlama versions do not clearly beat the 7B model.
A cheap inference-time instruction emphasizing that distracting code should be ignored restores most lost accuracy for some models, including Phind 34B and Gemini 1.5, but fails or backfires on others.
Commercial models are not robust by default: GPT-4o still loses 13.9% on MBPP, and Claude 3 and Gemini 1.5 lose over 21% on both benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the oracle-attack design suggests an iterative variant: feed the target model's failures back to the oracle to refine suffixes, which could produce stronger transfer attacks than a single forward pass.
Beyond the paper, the unenforced cosine-distance budget implies a cleaner protocol: filter suffixes by $\epsilon$ before evaluation. If random suffixes at the same cosine distance cause similar CDRA, the reported degradation is a sensitivity effect rather than a tightly bounded adversarial effect.
Beyond the paper, the same prompt-level evaluation could be extended to natural-language-only perturbations and to non-Python languages, where instruction-tuned models may show different failure modes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

DegradePrompter is a simple and useful black-box attack generator for instruction-tuned code LLMs: use GPT-4 as an oracle to append 1-3 lines of misleading but contextually relevant code to a coding prompt. That idea is a reasonable extension of DeceptPrompt and ReCode, and the paper executes it competently across eight models, two benchmarks, plus a handcrafted baseline and a mitigation experiment. The tables give a directional picture that DegradePrompter degrades pass@1 for most models; I think that central observation is defensible.

The load-bearing problem is the headline claim. The abstract says open-source models degrade 12-34% while commercial models degrade 3-24%, but that comparison is built on CDRA, which normalizes the pass@1 loss by the clean pass@1. Commercial models have much higher clean scores, so their CDRA is compressed even when the absolute drop is equal or larger. Recomputing the numbers from Tables 2 and 3 as absolute pass@1 drops: on HumanEval the open-source mean drop is 13.1 points and the commercial mean drop is 13.6 (Claude 19.7, Gemini 18.6, GPT-4 2.5). On MBPP, open-source drops 12.8 points against commercial 14.5. So Claude 3 and Gemini 1.5 are not more resilient than open-source models; GPT-4 is the only commercial model that clearly stands out. The abstract's ranges do not establish the stated divide.

There are also two secondary issues. First, the epsilon constraint in Eq. 1 is not enforced during construction; Eq. 6 assumes a single oracle pass yields a suffix satisfying the bound, but the cosine distance is measured only after the fact. The reported average distances (0.036 and 0.04) are below the thresholds, so this is not fatal, but the attack is not truly bounded. Second, pass@1 and CDRA are point estimates with no error bars, confidence intervals, seeds, or repeated runs; the differences between some models could easily be noise. The reproducibility section promises code and data, but nothing is public yet.

This paper deserves a serious referee, not a desk reject. The attack is cheap, the empirical body is substantial, and the flaws are fixable. A revision should report absolute pass@1 drops alongside or instead of CDRA, add uncertainty measures, and either enforce the epsilon constraint during generation or drop the pretense of a bounded attack. With those changes it could be a solid workshop or short conference paper. The right audience is practitioners evaluating code assistants and researchers working on prompt robustness.

Send it to peer review, but expect heavy revision.

Referee Report

4 major / 4 minor

Summary. This paper introduces DegradePrompter, a black-box method that uses GPT-4-turbo as an oracle to generate misleading suffixes appended to coding prompts, and evaluates eight instruction-tuned Code LLMs on HumanEval and MBPP using pass@1 and the CDRA metric. It reports that open-source models degrade by 12% to 34% while commercial models degrade by 3% to 24%, and it proposes a guided-prompting defense measured by ANR. The paper's central claim is that open-source Code LLMs are more susceptible to adversarial prompt perturbations than commercial models.

Significance. If the claims held, DegradePrompter would be a useful lightweight robustness benchmark for code LLMs: it is black-box, needs only a single forward pass through the oracle, is tested across several open and closed models, and includes both a handcrafted baseline and a defense. The tables do support the directional claim that DegradePrompter lowers pass@1 for most models, and the defense results are informative. However, the headline open-source-versus-commercial conclusion is not supported by the absolute performance drops in the paper's own tables; the study is better viewed as a per-model robustness measurement than as evidence of a systematic divide.

major comments (4)

[Section 4, Eq. (7), Tables 2-3] The central open-source-versus-commercial claim is an artifact of the CDRA normalization. Recomputing absolute pass@1 drops (pass@1(C) - pass@1(A)) under DegradePrompter from Tables 2 and 3 gives HumanEval mean drops of 13.1 percentage points for the five open-source models (10.2, 11.4, 12.2, 12.7, 19.0) versus 13.6 points for the three commercial models (19.7, 18.6, 2.5). On MBPP the means are 12.8 versus 14.5. Claude 3 and Gemini 1.5 therefore lose more absolute correctness than most open-source models, and the claimed open-vs-closed divide is driven by normalization by clean pass@1 combined with GPT-4's unusually small drop. The paper must either present absolute deltas as the primary evidence or justify explicitly why relative degradation is the appropriate measure for the resilience claim made in the abstract and conclusion.
[Section 3.2, Eq. (6), and Section 4 (d and epsilon)] The perturbation bound in the problem formulation is not enforced during attack generation. Equation (6) sets sadv = O(x) via a single forward pass through the oracle, and Section 4 reports only post-hoc average cosine distances (0.036 on HumanEval, 0.04 on MBPP) rather than using the distance constraint to filter or regenerate suffixes. The paper should state whether every adversarial prompt satisfies d(xadv, x) <= epsilon, report the fraction that do, and clarify how the epsilon thresholds affected construction. Without this, CDRA measures sensitivity to arbitrary appended suffixes rather than to perturbations verified to lie within the stated budget.
[Section 3.2 and Table 3] GPT-4 serves as both the suffix-generation oracle (gpt-4-turbo) and one of the three commercial evaluation targets (gpt-4o). GPT-4 is also the only commercial model with a small absolute drop (2.5 points on HumanEval), and removing it makes the commercial mean absolute HumanEval drop 19.15 points, larger than the open-source mean of 13.1. The authors should either exclude GPT-4 from the target set, use a different oracle, or provide evidence that oracle-target overlap does not inflate the measured robustness of GPT-4; as it stands, the commercial-resilience conclusion rests substantially on this confounded data point.
[Section 5, Tables 2-4] All pass@1, CDRA, and ANR values are reported as point estimates with no error bars, confidence intervals, or information about repeated runs or seeds. Because the open-versus-closed comparison hinges on mean absolute drops that are nearly equal (13.1 vs 13.6 on HumanEval), the absence of uncertainty quantification makes the comparative claim unverifiable. The paper should report standard errors, bootstrap confidence intervals, or at least the variance across n=10 samples per problem.

minor comments (4)

[Section 5.1] The text refers to 'Phind 33B' while Table 2 and Figure 2 list Phind 34B; the model size should be consistent.
[Section 4 (Decoding Parameters)] The paper says n=10 samples are generated to estimate pass@1 but does not specify the estimator used (e.g., the unbiased pass@1 estimator from Chen et al. or the pass rate of the first sample). This should be clarified for reproducibility.
[Abstract and Section 5.2] The abstract's commercial degradation range of '3% to 24%' does not match the table values exactly: GPT-4 has CDRA 2.7% on HumanEval and Gemini 1.5 has 24.9% on HumanEval, so the stated bounds should be updated.
[Section 3.2 (Prompt for generating adversarial coding prompts)] The oracle prompt instructs a limit of 1-3 inserted lines, but there is no validation that the oracle's output always complies; a sentence describing any post-processing or acceptance check would remove ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DegradePrompter is an empirical black-box attack evaluation with no fitted parameters or self-citation load-bearing steps.

full rationale

The paper's derivation chain is straightforward: Equation (1) defines an adversarial suffix objective; Equation (6) approximates that objective by having an oracle LLM generate a suffix in a single forward pass; the attack is then measured by pass@1 on clean versus adversarial prompts; and Equation (7) aggregates the drop as CDRA. None of these steps defines the output in terms of the input or fits a parameter and then predicts the same quantity. The oracle substitution sadv = O(x) is a heuristic approximation, not an optimization that enforces Equation (1), and its success is judged only by the downstream pass@1 measurements, so the measured degradation is not forced by construction. CDRA is a relative-degradation metric, but it is a post-hoc aggregation of two independently measured pass@1 values; it is not used as a fitting target. The only notable overlap is that GPT-4-turbo is used as the suffix-generation oracle while GPT-4o is one of the evaluated targets; this may affect transferability or independence of the attack, but it does not make the measurement equal to its input by construction. The one self-citation, reference [14], supports a general related-work statement about data poisoning and backdoor attacks and is not load-bearing for the method or the central empirical claim. Concerns about epsilon thresholds not being enforced during generation are methodological validity issues, not circularity. The paper is self-contained against external benchmarks (HumanEval, MBPP) and against external pass@1 evaluation, so the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical benchmark, so the ledger contains experimental protocol assumptions rather than fitted physical parameters. No new physical or conceptual entities are introduced; DegradePrompter and guided prompting are procedural methods, not entities. The main load-bearing premises concern the oracle LLM, the benchmarks, and the distance metric.

free parameters (4)

epsilon distance thresholds = 0.1 (HumanEval), 0.2 (MBPP)
Hand-chosen bounds on cosine distance between clean and adversarial prompts; they appear in the problem formulation (Equation 1) but are not enforced during suffix generation, only measured afterward.
suffix length limit = 1 to 3 lines
Hand-chosen constraint on inserted code snippets in the DegradePrompter prompt; it shapes attack subtlety and strength.
decoding temperature and top_p = 0.4 and 1.0
Chosen by the authors for all samples; pass@1 estimates and CDRA values depend on this sampling configuration.
number of samples n = 10
Used to estimate pass@1 per problem; no variance, seeds, or repeated trials are reported.

assumptions (4)

domain assumption HumanEval and MBPP unit tests correctly define functional correctness
The correctness function F in Equation 2 is operationalized as pass@1 against these tests; if tests are incomplete or buggy, CDRA misstates robustness.
ad hoc to paper GPT-4-turbo generates effective adversarial suffixes in a single forward pass
Equation 6 assumes the oracle output satisfies Equation 1 without search or verification; the paper reports no oracle success rate or filtering procedure.
domain assumption Cosine similarity from SentenceTransformer approximates prompt distance
Used to quantify d in Equation 1; embedding similarity may not reflect the semantic or functional distance relevant to code correctness.
domain assumption Chat and instruction templates are applied correctly for each model
Formatting a prompt according to each model's chat template is necessary for a valid comparison; template mistakes would confound CDRA.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code." pith.science (2026). https://pith.science/paper/3D7SEQW6

@misc{pith2026241119508,
  author       = {Pith},
  title        = {Pith review of: On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3D7SEQW6}},
  note         = {Machine review of arXiv:2411.19508}
}

read the original abstract

The advent of instruction-tuned Large Language Models designed for coding tasks (Code LLMs) has transformed software engineering practices. However, their robustness against various input challenges remains a critical concern. This study introduces DegradePrompter, a novel method designed to systematically evaluate the robustness of instruction-tuned Code LLMs. We assess the impact of diverse input challenges on the functionality and correctness of generated code using rigorous metrics and established benchmarks. Our comprehensive evaluation includes five state-of-the-art open-source models and three production-grade closed-source models, revealing varying degrees of robustness. Open-source models demonstrate an increased susceptibility to input perturbations, resulting in declines in functional correctness ranging from 12% to 34%. In contrast, commercial models demonstrate relatively greater resilience, with performance degradation ranging from 3% to 24%. To enhance the robustness of the models against these vulnerabilities, we investigate a straightforward yet effective mitigation strategy. Our findings highlight the need for robust defense mechanisms and comprehensive evaluations during both the development and deployment phases to ensure the resilience and reliability of automated code generation systems.

Figures

Figures reproduced from arXiv: 2411.19508 by the authors.

**Figure 1.** Overview of the DegradePrompter attack. 𝐹 (y, x) = ( 1, if y is functionally correct with respect to x 0, otherwise (2) The effectiveness of our approach can be measured by validating the functional correctness of the generated code. Let P represent a set of clean coding prompts, and let Padv denote the corresponding set of adversarial prompts obtained by appending adversarial suffixes. The correctness of functiona… view at source ↗

**Figure 2.** Correctness Degradation Rate under Attack (CDRA) for handcrafted and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 9 canonical work pages

[1]

2024. Phind. https://www.phind.com/blog/code-llama-beats-gpt4

work page 2024
[2]

Report from GitHub Copilot

2024. Report from GitHub Copilot. https://github.com/features/copilot

work page 2024
[3]

Anthropic. 2024. Claude 3. https://www.anthropic.com/news/claude-3-family

work page 2024
[4]

Owura Asare, Meiyappan Nagappan, and Nirmal Asokan. 2022. Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code? Empirical Software Engineering 28 (2022), 1–24. https://api.semanticscholar.org/CorpusID: 248085518

work page 2022
[5]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL]

arXiv 2021
[6]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...
[7]

Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evti- mov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksandar Straumann, Gabriel Synnaeve, Varun Von- timitta, Spencer Whitman, and Joshua Saxe. 2023...

arXiv 2023
[8]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv 2021

Show all 31 references

[9]

Desmarais, Zhen Ming, and Jiang

Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Zhen Ming, and Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? arXiv:2206.15331 [cs.SE]

2023 arXiv
[10]

Thomas Dohmke. 2023. GitHub Copilot for Business is now available. https: //github.blog/2023-02-14-github-copilot-for-business-is-now-available/

2023
[11]

Hugging Face. 2024. Hugging Face Hub. https://huggingface.co/

2024
[12]

Google. 2024. Our next-generation model: Gemini 1.5. https://blog.google/ technology/ai/google-gemini-next-generation-model-february-2024/

2024
[13]

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guant- ing Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang
[14]

Md Imran Hossen, Jianyi Zhang, Yinzhi Cao, and Xiali Hei. 2024. Assessing Cybersecurity Vulnerabilities in Code Large Language Models. arXiv:2404.18567

2024 arXiv
[15]

Akshita Jha and Chandan K Reddy. 2023. Codeattack: Code-based adversarial attacks for pre-trained programming language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 14892–14900

2023
[16]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAtten- tion. In Proceedings of the ACM SIGOPS 29th Symposium on Operating...

2023
[17]

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. arXiv preprint arXiv:2306.08568 (2023)

2023 arXiv
[18]

Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023. OctoPack: Instruction Tuning Code Large Language Models. arXiv:2308.07124 [cs.CL]

2023 arXiv
[19]

Vijayaraghavan Murali, Chandra Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, and Nachiappan Nagappan. 2023. CodeCompose: A Large-Scale Industrial Deployment of AI-assisted Code Author- ing. arXiv preprint arXiv:2305.12050 (2023)

2023 arXiv
[20]

OpenAI. 2023. GPT-4 Technical Report. Technical Report. OpenAI. https: //arxiv.org/abs/2303.08774 arXiv:2303.08774

2023 arXiv
[21]

Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 754–768

2022
[22]

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xi- aoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Fer- rer, Aaron Grattafiori, Wenhan Xiong, Alexandre Dé...

2023 arXiv
[23]

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. arXiv preprint arXiv:1908.07125 (2019)

2019 arXiv
[24]

Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. 2022. ReCode: Robustness Evaluation of Code Generation Models. arXiv:...

2022 arXiv
[25]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

2020 arXiv
[26]

Fangzhou Wu, Xiaogeng Liu, and Chaowei Xiao. 2023. DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions. arXiv:2312.04730 [cs.CR]

2023 arXiv
[27]

Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2023. Virtual Prompt Injection for Instruction-Tuned Large Language Models. arXiv preprint arXiv:2307.16888 (2023)

2023 arXiv
[28]

Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022. Natural attack for pre-trained models of code. InProceedings of the 44th International Conference on Software Engineering. 1482–1493

2022
[29]

Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023)

2023 arXiv
[2022]

arXiv:2204.05862 [cs.CL]

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862 [cs.CL]

arXiv
[2024]

arXiv:2401.14196 [cs.SE]

DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. arXiv:2401.14196 [cs.SE]

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

2024. Phind. https://www.phind.com/blog/code-llama-beats-gpt4

work page 2024

[2] [2]

Report from GitHub Copilot

2024. Report from GitHub Copilot. https://github.com/features/copilot

work page 2024

[3] [3]

Anthropic. 2024. Claude 3. https://www.anthropic.com/news/claude-3-family

work page 2024

[4] [4]

Owura Asare, Meiyappan Nagappan, and Nirmal Asokan. 2022. Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code? Empirical Software Engineering 28 (2022), 1–24. https://api.semanticscholar.org/CorpusID: 248085518

work page 2022

[5] [5]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL]

arXiv 2021

[6] [6]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

[7] [7]

Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evti- mov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksandar Straumann, Gabriel Synnaeve, Varun Von- timitta, Spencer Whitman, and Joshua Saxe. 2023...

arXiv 2023

[8] [8]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv 2021