Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Code LLMs stay vulnerable when one malicious request is split into benign-looking subtasks across turns; the new MOCHA benchmark shows rejection rates drop by up to 54.1 points, and fine-tuning on it restores safety without hurting coding…

desk verdict MOCHA is a genuinely useful benchmark with a new multi-turn decomposition attack, but the headline rejection numbers rest on an unvalidated LLM judge and should not be quoted until human agreement is reported. read the letter →

arxiv 2507.19598 v1 pith:7QGH57FE submitted 2025-07-25 cs.CL cs.AIcs.CRcs.LG

Muntasir Wahed , Xiaona Zhou , Kiet A. Nguyen , Tianjiao Yu , Nirav Diwan , Gang Wang , Dilek Hakkani-Tür , Ismini Lourentzou This is my paper
classification cs.CLcs.AIcs.CRcs.LG
keywords codelanguagemodelsmulti-turnjailbreakmaliciousgenerationdecompositionattacksafetyalignmentrejectionrateMOCHAbenchmarkLoRAfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that code-generation LLMs, including the strongest closed-source models, are not robust against multi-turn adversarial prompts, and that this vulnerability can be measured and partially closed. To do this it introduces MOCHA, a benchmark of about 10.5K malicious coding prompts spanning 13 threat categories, and a new 'code decomposition attack' that fragments a harmful request into benign-looking subtasks spread over several conversation turns. Across 19 models, rejection rates fall substantially from single-turn to multi-turn prompts, with drops such as -45.0% for GPT-4o and -54.1% for Qwen2.5-Coder-14B, while average rejection rates stay low for both open- and closed-source models. The paper further claims that LoRA fine-tuning on MOCHA raises rejection rates, preserves coding benchmark performance, and transfers to external adversarial datasets without extra supervision. A sympathetic reader would care because the results imply that current safety evaluations that only use single-turn prompts understate real-world misuse risk.

What carries the argument

The load-bearing object is the Code Decomposition Attack: an LLM-guided generator divides a malicious seed prompt into two to five subtasks, applies randomly sampled jailbreak strategies to each, and labels every turn with a cumulative maliciousness level (benign, slightly suspicious, highly suspicious, malicious). This attack is packaged in MOCHA, a 10,084-example training set plus two 200-sample test sets (a general in-distribution split and a held-out 'Logic_Bomb' split for zero-shot evaluation) built from 387 seed phrases across 13 malware categories and 17 jailbreak strategies. The other load-bearing mechanism is the evaluation pipeline: model responses are classified as rejections or non-rejections by a combination of rejection string matching, code search, and an LLM judge prompted to flag either malicious code or malicious explanation, and that binary decision feeds every rejection-rate number in the paper. The machinery works by converting contextual intent, not just individual prompt wording, into a measurable robustness score.

What would settle it

Re-annotate a random sample of the model responses from MOCHA's multi-turn split with human red-teamers who do not see the automated judge's labels and measure agreement on the binary rejection/assist decision; if agreement is substantially below near-total, the reported rejection rates and the single-to-multi-turn drops would need to be recomputed.

Watch

Extended reading notes

Core claim

The central claim is that code decomposition attacks work: a malicious coding task, when broken into a sequence of individually benign subtasks across multiple conversational turns, reliably bypasses the safety filters of open- and closed-source code LLMs, and that fine-tuning on MOCHA makes models measurably safer without degrading general-purpose coding. Empirically, in MOCHA's single-turn versus multi-turn comparison every tested model's rejection rate drops in multi-turn settings; the largest drops are observed for GPT-4o (-45.0%), Claude 3.5 Haiku (-37.5%), and Qwen2.5-Coder-14B (-54.1%). On average, closed-source models reject 13.0-54.5% of prompts and open-source models 2.5-49.0%. After one-epoch LoRA fine-tuning on MOCHA's 10,084 training examples, rejection rates improve across model families and scales (up to 36%, averaging 21.8%), utility benchmarks roughly hold, and rejection rates on four external adversarial datasets increase by up to 32.4% (MALICIOUSGPT for Qwen2.5-Coder-1.5B). The paper treats this not as a solved-defense result but as evidence that multi-turn context is the missing dimension in current red-teaming and safety evaluation for code models.

Load-bearing premise

The central numbers all depend on an automated language-model judge that decides which model responses count as rejections, and the paper gives no human validation, no inter-annotator agreement, and no exact judge prompt for that classifier.

Editorial extensions

If this is right

  • Multi-turn decomposition attacks lower rejection rates for every model tested, so safety evaluations that only use single-turn prompts will systematically overestimate the real-world robustness of code LLMs.
  • Parameter-efficient fine-tuning on a diverse adversarial coding benchmark is a practical defense: it raises rejection rates while leaving HumanEval+ and MBPP+ Pass@1 roughly unchanged.
  • Rejection gains transfer to unseen adversarial benchmarks, with fine-tuned models improving by up to 32.4% on external datasets such as MALICIOUSGPT without additional supervision.
  • Models struggle most on rare threat categories and on the held-out Logic_Bomb split, so novel or rare categories remain an open weakness even after fine-tuning.
  • Safety-focused system prompts are only modestly helpful, and mainly when the base model already has some safety alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the automated judge is replaced by human raters or a different judge model, the absolute rejection rates may shift; since the judge prompt and inter-annotator agreement are not reported, the single-to-multi-turn gap should be read as an upper-confidence bound until re-validated.
  • Because prompt generation is preemptive rather than adaptive, real adversaries who probe model behavior and adjust their next turn could be even more effective than MOCHA's fixed sequences, making the reported multi-turn drops a lower bound on real-world risk.
  • The decomposition idea should transfer to other LLM agents that take multi-step actions such as web automation, shell use, or database access, where each step looks benign; extending MOCHA to those settings would test whether the vulnerability is code-specific or general to long-horizon instruction following.
  • Fine-tuned models may overfit the distribution of LLM-generated jailbreaks, so refreshing the benchmark with newly discovered attack strategies will be necessary to keep the defense generalization claim valid over time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MOCHA, a benchmark of adversarial coding prompts spanning 13 malicious categories, with single-turn jailbreak variants and new multi-turn 'code decomposition attacks' that break a malicious request into progressively escalating subtasks. The authors evaluate 20 open- and closed-source code LLMs on two 200-sample test splits, reporting rejection rates and utility (Pass@1). They further fine-tune several open models with LoRA on MOCHA, reporting improved rejection rates (up to 36% increase on MOCHA) that generalize to four public adversarial benchmarks (up to 32.4% increase). The central empirical claims are that multi-turn prompts markedly lower rejection rates relative to single-turn prompts, and that fine-tuning on MOCHA improves robustness without sacrificing coding ability.

Significance. If the reported numbers are trustworthy, the paper addresses a genuinely underexplored and practically important gap: multi-turn, decomposition-based jailbreaks of code LLMs. The benchmark is large (10.5K prompts), covers diverse threat categories and 17 jailbreak strategies, includes a held-out category for zero-shot evaluation, and is publicly released. The breadth of models (20, including 7 proprietary APIs) and the demonstrated fine-tuning gains on external datasets are valuable contributions. However, the entire quantitative core rests on an automated LLM-based rejection classifier that is not validated against human judgment, and the scoring unit for multi-turn conversations is ambiguous. These issues must be resolved before the central safety comparisons can be accepted.

major comments (3)
  1. [Appendix D] All rejection rates in Figures 3, 4, and Tables 2 and 3 are produced by the multi-step pipeline combining rejection string matching, code search, and a structured prompt to an unnamed LLM evaluator. The paper reports no human validation of these classifications, no inter-annotator agreement, and does not release the judge prompt or the judge model. Because every central claim is a comparison of these rates, an evaluator that over-flags responses containing any code block would inflate rejection rates and shrink the reported multi-turn vulnerability gap, while a judge that under-flags refusals with disclaimers would do the opposite. The authors need to report agreement against human labels on a representative sample, release the full judge prompt, and state the judge model and version.
  2. [Section 4, Figure 4] It is not specified whether multi-turn rejection rates are computed per assistant turn or per full conversation, nor whether a refusal issued after earlier turns produced benign-looking code counts as a rejection. The large multi-turn gaps shown in Figure 4 (e.g., -45.0% for GPT-4o, -54.1% for Qwen2.5-Coder-14B) could be inflated if a single refusal anywhere in the conversation is counted as rejection of the entire exchange, or deflated if each turn is scored in isolation. The paper should explicitly define the scoring unit and report per-turn rejection rates so that the multi-turn vulnerability claim is unambiguous.
  3. [Table 3] The generalization claim, including the headline '32.4% increase' on MALICIOUSGPT, is computed with the same unvalidated evaluator applied to external datasets whose native labeling schemes differ (e.g., RMCBENCH, HARMBENCH, MALICIOUSGPT). If the MOCHA evaluator applies different criteria than those used to construct these datasets, the cross-dataset improvement may reflect the judge's decision rule rather than genuine robustness transfer. The authors should validate the evaluator separately on each external dataset or report results using the official evaluation protocol of each benchmark.
minor comments (5)
  1. [Section 3.4] The dataset size is reported inconsistently: the abstract says '10K', Table 1 says '10.5K', and Section 3.4 says the training split has 10,084 examples. Please harmonize these numbers.
  2. [Section 3.1] There is a typo: 'spanning all thread categories' should be 'spanning all threat categories.'
  3. [Figure 1] The caption states the ideal models reside in the top-left quadrant, but with utility on the x-axis and rejection rate on the y-axis, the ideal quadrant would be top-right (high utility, high rejection). Please correct.
  4. [Section 3.3] The text says the four risk categories are 'uniformly distributed,' but the generation template in Figure 7 only requires at least two distinct labels and does not enforce uniform distribution; clarify how uniformity is achieved.
  5. [Appendix D] The limitations section (Appendix I) acknowledges that generated code is not executed and that analysis is textual, but it does not mention the absence of human validation for the rejection classifier, which is a more immediate threat to the reported numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MOCHA is an empirical benchmark whose fine-tuning generalization is measured on held-out and external public datasets, with no self-citation chain or fitted quantity renamed as a prediction.

full rationale

MOCHA is an empirical benchmark paper, not a derivation; the central claims are measurements of rejection rates before and after LoRA fine-tuning. The fine-tuning results are evaluated on held-out MOCHA test splits and on four external public datasets (RMCBENCH, MALICIOUSGPT, ADVBENCH, HARMBENCH), so the reported generalization is not fitted from the data used to make the claim. The attack prompts are synthesized from 387 seed phrases and external jailbreak-strategy taxonomies, and the held-out Logic_Bomb category is excluded from training, giving the zero-shot evaluation independent content. The only notable weakness is the automated LLM-judge evaluation pipeline described in Appendix D: the judge prompt and model are not fully disclosed, and Appendix I states that generated outputs are not executed to verify functional correctness. These are validation and reproducibility risks, not circularity, because the rejection-rate quantity is not defined in terms of itself, no fitted parameter is renamed as a prediction, and no load-bearing self-citation is used to justify the benchmark's construction or conclusions. Therefore no circular step meeting the quote-and-reduction standard is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No physical constants or fitted numerical parameters are used. The paper relies on hand-chosen dataset construction hyperparameters and on assumptions about the validity of LLM-generated prompts and the LLM judge. These assumptions are not circular, but they are load-bearing for the empirical claims.

free parameters (4)
  • Number of subtasks per decomposition = 2 to 5
    Chosen by hand in the meta-prompt (Figure 7). This design choice affects how fragmented the multi-turn attacks are.
  • Jailbreak strategies sampled per prompt = 5
    A random subset of five strategies is presented to the generator during single-turn jailbreak and multi-turn creation (Sections 3.2, 3.3). This is an arbitrary selection that influences diversity.
  • Risk label distribution = uniform across 4 categories
    Section 3.3 states the four maliciousness labels are uniformly distributed across subtasks. This is a hand-set design choice.
  • Seed phrases per category = 387 total
    Hand-curated seed phrase count (Section 3.1). It anchors prompt synthesis and coverage.
assumptions (4)
  • domain assumption The 13 malware categories from Vasani et al. (2023) provide comprehensive coverage of malicious coding intents.
    Section 3.1 builds the taxonomy on this reference. If these categories are not representative, the benchmark's coverage is limited.
  • domain assumption DeepSeek-R1 generated prompts are high-fidelity malicious prompts; only one sample was removed by the filtering step.
    Section 3.1 reports minimal filtering, which implies the generator is reliable. The paper does not quantify generation failure rates beyond this.
  • domain assumption The LLM-based evaluator accurately classifies model responses as malicious or rejected.
    Appendix D uses a structured evaluation prompt but provides no human validation of response classifications or inter-annotator agreement.
  • domain assumption Logic_Bomb is a semantically distinct held-out category, selected via unsupervised anomaly detection on embeddings.
    Section 3.4 relies on this selection to claim zero-shot generalization in MOCHA2. If the category is not truly out-of-distribution relative to training, the claim is weakened.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?." pith.science (2026). https://pith.science/paper/7QGH57FE

@misc{pith2026250719598,
  author       = {Pith},
  title        = {Pith review of: MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QGH57FE}},
  note         = {Machine review of arXiv:2507.19598}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) have significantly enhanced their code generation capabilities. However, their robustness against adversarial misuse, particularly through multi-turn malicious coding prompts, remains underexplored. In this work, we introduce code decomposition attacks, where a malicious coding task is broken down into a series of seemingly benign subtasks across multiple conversational turns to evade safety filters. To facilitate systematic evaluation, we introduce \benchmarkname{}, a large-scale benchmark designed to evaluate the robustness of code LLMs against both single-turn and multi-turn malicious prompts. Empirical results across open- and closed-source models reveal persistent vulnerabilities, especially under multi-turn scenarios. Fine-tuning on MOCHA improves rejection rates while preserving coding ability, and importantly, enhances robustness on external adversarial datasets with up to 32.4% increase in rejection rates without any additional supervision.

Figures

Figures reproduced from arXiv: 2507.19598 by the authors.

Figure 1
Figure 1. Safety vs. Utility of Code Models. Here, x-axis presents utility, measured as the average Pass@1 across HumanEval+ and MBPP+ benchmarks, while y-axis presents the average Rejection Rate (RR) in MOCHA 1 test and MOCHA 2 test.The ideal models reside in the top-left quadrant, achieving both high utility and high safety. Raw numbers in Appendix A. prompts, spanning a spectrum of difficulty from explicit single-turn prom… view at source ↗
Figure 2
Figure 2. Example from a representative category (TROJAN) and an associated seed phrase (disguised system backdoors). The category and the seed phrase are used to generate diverse seed prompts, which then create single-turn and multi-turn jailbreak prompts. The multi-turn prompts are constructed using a novel Code Decomposition Attack, where the same malicious functionality is elicited through a series of benign-looking promp… view at source ↗
Figure 3
Figure 3. Robustness of various LLMs across 13 adversarial MOCHA categories. Lower Rejection Rate (lighter color) is better. Average column presents overall RR across all models for each category. to unseen adversarial scenarios. We manually ver￾ify the malicious intent of all the samples in both test sets. In addition, we construct MOCHAval, a validation set of randomly sampled prompts from the training categories that serve… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Comparison of model performance on single-turn and multi-turn prompts in MOCHA. The large performance gaps (indicated by arrows) highlight the challenge multi-turn prompts [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Prompt template for malicious seed prompt generation. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Prompt template for malicious single-turn jailbreak generation. For jailbreaking strategies, we provide the LLM with the name and definition of 5 randomly sampled strategy for each seed prompt. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for malicious multi-turn code decomposition attack. We synthesize an equal proportion of prompts that incorporate jailbreaking strategies and those that do not. For jailbreaking strategies, we provide the LLM with the name and definition of 5 randomly s…
Figure 8
Figure 8. Figure 8: Contrasting model behavior: (a) the assistant responds with potentially harmful code, and (b) the assistant appropriately rejects the request. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Response from the base Qwen2.5-Coder-14B that provides code for the malicious task. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Response from the LoRA-adapted Qwen2.5-Coder-14B that appropriately rejects the prompt. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Response from the base Qwen2.5-Coder-14B that provides code for the malicious task. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Response from the LoRA-adapted Qwen2.5-Coder-14B that appropriately rejects the prompt. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Rejection Rate (RR) comparison between different prompt types. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Validated Prompt Bank for Malicious Code Generation: Separating Executable Weapons from Security Knowledge in 1,554 Consensus-Labeled Prompts

    cs.CR 2026-05 accept novelty 5.0 of 10

    The paper releases a 1,554-prompt consensus-labeled bank separating executable malicious code requests from security knowledge requests, validated by five-model majority labeling with Fleiss' kappa of 0.876.

Reference graph

Works this paper leans on

66 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AmazonAGI. 2024. The amazon nova family of models: Technical report and model card

    work page 2024

  4. [4]

    Anthropic. 2024. Introducing claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. Accessed May 19, 2025

    work page 2024

  5. [5]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  6. [6]

    Suriya Ganesh Ayyamperumal and Limin Ge. 2024. Current state of llm risks and ai guardrails. arXiv preprint arXiv:2406.12934

    arXiv 2024

  7. [7]

    Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In NeurIPS Datasets and Benchmarks Track

    work page 2024

  8. [8]

    Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca

    2023

Show all 66 references
  1. [9]

    Jiachi Chen, Qingyuan Zhong, Yanlin Wang, Kaiwen Ning, Yongkun Liu, Zenan Xu, Zhe Zhao, Ting Chen, and Zibin Zheng. 2024 a . Rmcbench: Benchmarking large language models' resistance to malicious code. In Proceedings of the 39th IEEE/ACM International Conference on Automated So...

    2024

  2. [10]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  3. [11]

    Xuan Chen, Yuzhou Nie, Wenbo Guo, and Xiangyu Zhang. 2024 b . https://openreview.net/forum?id=FfFcDNDNol When LLM meets DRL : Advancing jailbreaking efficiency via DRL -guided search . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  4. [12]

    GeminiTeam, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  5. [13]

    Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. 2024 a . Redcode: Risky code execution and generation benchmark for code agents

    2024

  6. [14]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  7. [15]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024 b . Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  8. [16]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3

    2022

  9. [17]

    Kai Hu, Weichen Yu, Yining Li, Tianjun Yao, Xiang Li, Wenhe Liu, Lijun Yu, Zhiqiang Shen, Kai Chen, and Matt Fredrikson. 2024. Efficient llm jailbreak via adaptive dense-to-sparse constrained optimization. Advances in Neural Information Processing Systems, 37:23224--23245

    2024

  10. [18]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  11. [19]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  12. [20]

    Cristina Improta. 2023. Poisoning programs by un-repairing code: security concerns of ai-generated code. In 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW), pages 128--131. IEEE

    2023

  13. [21]

    Slobodan Jenko, Jingxuan He, Niels M \"u ndler, Mark Vero, and Martin Vechev. 2024. Practical attacks against black-box code completion engines. arXiv preprint arXiv:2408.02509

    2024 arXiv

  14. [22]

    Akshita Jha and Chandan K Reddy. 2023. Codeattack: Code-based adversarial attacks for pre-trained programming language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14892--14900

    2023

  15. [23]

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and 1 others. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural In...

    2024

  16. [24]

    Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660

    2020 arXiv

  17. [25]

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023 a . Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197

    2023 arXiv

  18. [26]

    Jia Li, Zhuo Li, Huangzhao Zhang, Ge Li, Zhi Jin, Xing Hu, and Xin Xia. 2022 a . Poison attack and defense on deep source code processing models. arXiv preprint arXiv:2210.17029

    2022 arXiv

  19. [27]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190

    2021 arXiv

  20. [28]

    Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Han Fang, Aishan Liu, and Ee-Chien Chang. 2024. Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms. arXiv preprint arXiv:2402.14872

    2024 arXiv

  21. [29]

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023 b . Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191

    2023 arXiv

  22. [30]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022 b . Competition-level code generation with alphacode. Science, 378(6624):1092--1097

    2022

  23. [31]

    Zilong Lin, Jian Cui, Xiaojing Liao, and XiaoFeng Wang. 2024. Malla: Demystifying real-world large language model integrated malicious services. In 33rd USENIX Security Symposium (USENIX Security 24). USENIX Association

    2024

  24. [32]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558--21572

    2023

  25. [33]

    Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, and Kai Chen. 2024 a . Making them ask and answer: Jailbreaking large language models in few queries via disguise and reconstruction. arXiv preprint arXiv:2402.18104

    2024 arXiv

  26. [34]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024 b . https://openreview.net/forum?id=7Jwpw4qKkb Auto DAN : Generating stealthy jailbreak prompts on aligned large language models . In The Twelfth International Conference on Learning Representations

    2024

  27. [35]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:...

    2024 arXiv

  28. [36]

    Mistral AI Team . 2024 a . Codestral: Empowering developers and democratising coding with mistral ai. https://mistral.ai/news/codestral. Accessed May 19, 2025

    2024

  29. [37]

    Mistral AI Team . 2024 b . Large enough: Announcing mistral large 2. https://mistral.ai/news/mistral-large-2407. Accessed: 2025-05-12

    2024

  30. [38]

    Thanh-Dat Nguyen, Yang Zhou, Xuan Bach D Le, David Lo, and 1 others. 2023. Adversarial attacks on code models with discriminative graph patterns. arXiv preprint arXiv:2308.11161

    2023 arXiv

  31. [39]

    Kaiwen Ning, Jiachi Chen, Qingyuan Zhong, Tao Zhang, Yanlin Wang, Wei Li, Yu Zhang, Weizhe Zhang, and Zibin Zheng. 2024. Mcgmark: An encodable and robust online watermark for llm-generated malicious code. arXiv preprint arXiv:2408.01354

    2024 arXiv

  32. [40]

    Sheng Ouyang, Yihao Qin, Bo Lin, Liqian Chen, Xiaoguang Mao, and Shangwen Wang. 2025. Smoke and mirrors: Jailbreaking llm-based code generation via implicit malicious prompts. arXiv preprint arXiv:2503.17953

    2025 arXiv

  33. [41]

    Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, Ashish Datta, Maksym Zhuravinskyi, Dakota Mahan, Marco Bellagente, Carlos Riquelme, and 1 others. 2024. Stable code technical report. arXiv preprint arXiv:2404.01226

    2024 arXiv

  34. [42]

    Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-Fran c ois Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan Boyd-Graber. 2023. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a globa...

    2023 arXiv

  35. [43]

    Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You autocomplete me: Poisoning vulnerabilities in neural code completion. In 30th USENIX Security Symposium (USENIX Security 21), pages 1559--1575

    2021

  36. [44]

    Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, and 1 others. 2023. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348

    2023 arXiv

  37. [45]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825

    2023 arXiv

  38. [46]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In CCS, pages 1671--1685

    2024

  39. [47]

    Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Gust Verbruggen, and Carina Negreanu. 2023. https://www.microsoft.com/en-us/research/publication/codefusion-a-pre-trained-diffusion-model-for-code-generation/ Codefusion: A pre-trained diffusion model for code generation . In...

    2023

  40. [48]

    Vatsal Vasani, Amit Kumar Bairwa, Sandeep Joshi, Anton Pljonkin, Manjit Kaur, and Mohammed Amoon. 2023. Comprehensive analysis of advanced techniques and vital tools for detecting malware intrusion. Electronics, 12(20):4299

    2023

  41. [49]

    Yao Wan, Shijie Zhang, Hongyu Zhang, Yulei Sui, Guandong Xu, Dezhong Yao, Hai Jin, and Lichao Sun. 2022. You see what i want you to see: poisoning vulnerabilities in neural code search. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium...

    2022

  42. [50]

    Bui, Junnan Li, and Steven C

    Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D.Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint

    2023

  43. [51]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. Proceedings of Machine Learning Research, 235:52632--52657

    2024

  44. [52]

    Zeming Wei, Yifei Wang, and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387

    2023 arXiv

  45. [53]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, and 1 others. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  46. [54]

    Fangzhou Wu, Xiaogeng Liu, and Chaowei Xiao. 2023. Deceptprompt: Exploiting llm-driven code generation via adversarial natural language instructions. arXiv preprint arXiv:2312.04730

    2023 arXiv

  47. [55]

    Zeguan Xiao, Yan Yang, Guanhua Chen, and Yun Chen. 2024. Tastle: Distract large language models for automatic jailbreak attack. arXiv preprint arXiv:2403.08424

    2024 arXiv

  48. [56]

    Zhao Xu, Fan Liu, and Hao Liu. 2024. https://openreview.net/forum?id=yg4Tt2QeU7 Bag of tricks: Benchmarking of jailbreak attacks on LLM s . In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track

    2024

  49. [57]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  50. [58]

    Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022. Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering, pages 1482--1493

    2022

  51. [59]

    Dongyu Yao, Jianshu Zhang, Ian G Harris, and Marcel Carlsson. 2024. Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signa...

    2024

  52. [60]

    Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407.04295

    2024 arXiv

  53. [61]

    Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253

    2023 arXiv

  54. [62]

    Huangzhao Zhang, Zhiyi Fu, Ge Li, Lei Ma, Zhehao Zhao, Hua’an Yang, Yizhe Sun, Yang Liu, and Zhi Jin. 2022. Towards robustness of deep program processing models—detection, estimation, and enhancement. ACM Transactions on Software Engineering and Methodology (TOSEM), 31(3):1--40

    2022

  55. [63]

    Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B Tenenbaum, and Chuang Gan. 2023. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510

    2023 arXiv

  56. [64]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223

    2023 arXiv

  57. [65]

    Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy. 2022. https://arxiv.org/abs/2206.08474 Xlcost: A benchmark dataset for cross-lingual code intelligence . Preprint, arXiv:2206.08474

    2022 arXiv

  58. [66]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. https://arxiv.org/abs/2307.15043 Universal and transferable adversarial attacks on aligned language models . Preprint, arXiv:2307.15043

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.