Pith. sign in

REVIEW 5 major objections 5 minor 40 references

Adversarial Attack Classification and Robustness Testing for Large Language Models for Code

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Word-level perturbations are the most damaging attacks on code LLMs.

desk verdict Useful robustness benchmark for code LLMs, but the granularity ordering claim has load-bearing classification flaws; deserves peer review with major revision. read the letter →

arxiv 2506.07942 v1 pith:QKSDY7HB submitted 2025-06-09 cs.SE

Yang Liu , Armstrong Foundjem , Foutse Khomh , Heng Li This is my paper
classification cs.SE
keywords adversarialperturbationscodegenerationrobustnessLLM4Codeperturbationgranularitymetricsnaturallanguagepromptscommentstaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that the robustness of large language models for code is not uniform: it is systematically ordered by the granularity of the input perturbation. Word-level edits—synonym substitutions, variable and identifier renaming, word swaps in task descriptions—are the most damaging, while sentence-level rewrites such as paraphrase, back-translation, and tense changes are the easiest to survive, and character-level edits sit in between with mixed effects. The paper argues this ordering appears both when the code itself is perturbed and when natural-language task descriptions are perturbed, and that comments are a fragile third surface where even sentence-level insertions hurt, especially random ones. A two-dimensional taxonomy (content type crossed with granularity) plus dedicated robustness metrics is offered as a reusable testing recipe, and the experiments also report that single-language models tended to outlast multilingual ones under attack.

What carries the argument

The load-bearing mechanism is a two-dimensional classification of adversarial attacks: content type (code, natural-language task description, or comment) crossed with granularity (character, word, sentence/statement). Each perturbation method from two existing toolkits is assigned to exactly one cell, and then model outputs are scored with three robustness metrics: robust pass rate (fraction of perturbed prompts still solved), robust drop (relative performance loss versus unperturbed prompts), and robust relatives (fraction of predictions that change). The taxonomy supplies the paper's ordering: the sentence-level bin looks safest and the word-level bin most dangerous. A second mechanism is the re-use of previously released perturbed datasets extended with six natural-language attacks and with new model families added to the evaluation.

What would settle it

Recompute the robustness metrics after reclassifying every perturbation by the smallest linguistic unit it actually changes—for example, counting inflectional function-name changes as word-level and newline insertions as character-level—and compare the median robust-pass values across bins; if sentence-level no longer ranks best, the central claim is a taxonomy artifact. A complementary check holds perturbation intensity fixed across granularities by measuring edit distance per token.

Watch

Extended reading notes

Core claim

The central claim is that LLM4Code robustness is granularity-dependent in a consistent way. Across six models and two Python code-completion benchmarks, the paper finds that word-level perturbations—swapping a function name for a random token, renaming variables, replacing words in docstrings—produce the largest drops in functional correctness, whereas sentence-level changes such as back-translation or tense shifts cause the smallest degradation, and character-level changes are inconsistent. The same ordering is observed for code and task descriptions; comments, tested only at sentence level, degrade models substantially, with randomly inserted comments worse than converting a docstring into comments. The paper also claims that a general-purpose model family can be more robust than a code-specialized one in some scenarios, and that monolingual models generally withstand perturbations better than multilingual models. If this is right, adversarial evaluation of code models should be organized around semantic word-level attack surfaces first.

Load-bearing premise

The ordering of robustness by granularity depends on the paper's own assignment of each perturbation method to exactly one granularity bin; several methods labeled sentence/statement-level, such as inflectional changes to function names and newline insertions, actually alter identifiers or whitespace, and re-labeling them could erase the finding that sentence-level attacks are least harmful.

Editorial extensions

If this is right

  • Defenders and model developers should concentrate robustness training on semantic word-level edits—synonym substitution, identifier renaming, word swaps in prompts—because these cause the largest functional failures.
  • Robustness testing for code models should routinely perturb natural-language prompts and comments, not just code tokens, since the same vulnerability ordering appears on all three surfaces.
  • New models can be triaged cheaply by testing sentence-level and word-level bins first; the metrics give a pass/drop/change summary for each perturbation class.
  • Specialized single-language models may be preferable to multilingual generalists in safety-critical coding pipelines, because the paper observes them surviving perturbations more often.
  • Random comment insertion is a stronger attack than converting a docstring into comments, so evaluation suites that omit comments will underestimate real-world risk.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same granularity ordering, if it reflects surface-token reliance, should generalize to other programming languages, but Python-specific tokenization and syntax mean the exact drop magnitudes may not transfer.
  • Editorial inference: the paper's mono-versus-multi robustness gap points to a testable trade-off—fine-tuning a multilingual code model on a single language may buy robustness at the cost of cross-language capability; the paper reports the correlation but does not run that experiment.
  • Editorial inference: a cheap defense suggested by the results but not tested by the authors is input canonicalization, normalizing synonyms and identifier paraphrases before generation, which could neutralize the word-level class.
  • Editorial inference: the character-level mixed result may split further into typo-like noise, which models tolerate, and systematic Unicode or character substitutions, which break them; the current grouping could hide two different failure modes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a two-dimensional taxonomy for adversarial perturbations of LLM4Code inputs, classifying attacks by content type (code, task description, comments) and granularity (character, word, sentence/statement). The authors apply perturbations from ReCode and OpenAttack, plus a custom comment-insertion perturbation, to the HumanEval and MBPP benchmarks and evaluate six models (CodeGen-2B-mono, CodeGen-2B-multi, GPT-J-6B, InCoder-1B, CodeLlama-7b-hf, Meta-Llama-3-8B-Instruct). Robustness is quantified with RPs@k, RDs@k, and RRs@k. The headline claim is that word-level perturbations are the most harmful, sentence-level perturbations are the least harmful, and character-level perturbations are mixed, with mono-language models generally more robust than multi-language models.

Significance. The paper provides a useful empirical benchmark artifact: it uses externally defined perturbation generators, standard datasets, and a public replication package, and it extends model coverage to Llama-3 and CodeLlama. The per-method result tables (Tables 5-21) are a valuable resource for future robustness studies. However, the headline granularity ordering is not robustly established. The assignment of perturbations to granularity levels conflates linguistic granularity with semantic impact, and several reported model-dataset cells already contradict the claimed ordering. The main contribution at this stage is the taxonomy and per-perturbation robustness data rather than the claimed cross-model regularity.

major comments (5)
  1. [Table 4; Section 5.3 (RQ1)] Table 4 assigns FuncRenameInflectionalVariation (S1), newlines (S5), and newlineaftercode (S6) to Statement-Level, but these perturbations change a single function-name token or insert whitespace/newlines rather than altering sentence or statement structure. This is not a mere labeling issue: S5 and S6 are among the least damaging perturbations for most models (e.g., Table 5, codegen-2B-mono S5 RP=0.415 and S6 RP=0.427 versus S2 RP=0.262), so the statement-level median is inflated by construction. The granularity ordering must be recomputed under a defensible classification, or the claim should be restricted to per-method results.
  2. [Tables 12-15; Section 5.3 (RQ2)] The claimed word-level-most-harmful ordering is contradicted in the paper's own tables. In RQ2 MBPP (Table 15), codegen-2B-mono has a word-level median RP of 0.32 versus a character-level median of 0.21, so word-level perturbations are less harmful than character-level in this cell. In RQ2 HumanEval (Table 12), CodeLlama-7b-hf has word-level median RP 0.04 and RD 22.22% versus character-level median RP 0.02 and RD 55.56%. The paper should either report the proportion of model-dataset cells that follow the claimed ordering or explain why these exceptions do not undermine the aggregate claim.
  3. [Table 4; Sections 4.6 and 5.3] The aggregation of heterogeneous perturbations into level medians mixes granularity with semantic impact. Within Table 4's word level, W1 (camel-case renaming) and W6 (random renaming) have very different semantic consequences; within the statement level, DeadCodeInserter and newline insertions are pooled together. The level medians therefore reflect method composition as much as perturbation granularity. Please report per-method results as the primary evidence, treat level medians only as descriptive, and include a sensitivity analysis over alternative groupings.
  4. [Tables 9 and 16; Section 6] The Wilcoxon tables are not interpretable as reported: the compared conditions are unlabeled, several raw p-values exceed 0.05 (e.g., Table 9 rows with p=0.596, 0.445, 0.463, and 0.860; Table 16 rows with p=0.677 and 0.328), and the text's claim that the majority of p-values are below 0.05 with 87% large effect sizes does not establish that the granularity ordering holds consistently. Please provide labeled pairwise comparisons, the number of tests, and the Bonferroni-corrected significance decisions for each comparison.
  5. [RQ3; Table 17] RQ3 asks how comment perturbations at character, word, and sentence levels affect accuracy, but Table 17 contains only two perturbations, both classified as sentence-level (doc2comment and randominsertcomments). No character- or word-level comment perturbations are evaluated, so the RQ3 results cannot support the stated scope. Please revise the research question or add the missing perturbation levels.
minor comments (5)
  1. [Section 4.8, Tables 1-2] The sample counts sum to 380 (298 + 82) rather than the stated 381; please correct the arithmetic or the sampling description.
  2. [Section 5.2 and Tables 5-21] Model names are inconsistent: 'Meta-Llama-3-8B-Instruct' and 'Llama-3-8B-Instruct' are used interchangeably, and Section 5.2 says five model families while six models are evaluated.
  3. [Listing 14] Listing 14 shows syntactically invalid Python (an unindented 'returnresult' followed by 'else:'), which distracts from the intended illustration of comment insertion.
  4. [Section 2.2] Section 2.2 contains a near-verbatim duplicate of the threat-model paragraph; one copy should be removed.
  5. [Table 11] Table 11 uses inconsistent capitalization for the classification column ('word-Level' alongside 'Word-Level'); please standardize the formatting.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation is forced by construction; only a mild self-attribution of the granularity taxonomy, which itself is inherited from OpenAttack.

  1. renaming known result [Section 4.6, 'Classification of Attacks for Large Language models for Code'; also Section 3.3]
    "We get idea from OpenAttack [10] which provided the classification of perturbations in different levels. We introduce this classification method to code-specific perturbations because the similarity of code and natural language which we discussed before. Based on it, we propose a novel two-dimensional classification system."

    The paper labels its two-dimensional character/word/sentence taxonomy as 'novel' in Contribution 1, while attributing the granularity dimension to OpenAttack and, in Section 3.3, to the authors' own prior work [27] ('our study [27] introduces a novel classification system'). This makes the taxonomy/novelty claim partly self-referential, but the central robustness ordering is not derived from the taxonomy by construction: the RP/RD/RR values come from externally defined ReCode/OpenAttack perturbations and HumanEval/MBPP unit-test execution. The taxonomy is an input grouping, not a fitted prediction, so this is a mild attribution issue rather than a circular derivation.

full rationale

The study is an empirical benchmark, not a derivation. No parameter is fitted and then renamed as a prediction; the robustness metrics RPs@k, RDs@k, RRs@k and the perturbation generators are taken from prior external work (ReCode, OpenAttack), and HumanEval/MBPP are external benchmarks with unit-test-based correctness. The reported ordering (sentence-level most robust, word-level least) is a set of measured pass-rate medians across these externally generated perturbations, so the conclusion is not equivalent to the inputs by construction. The only circularity-adjacent feature is that the two-dimensional granularity taxonomy is presented as novel while being traced to the authors' own earlier paper [27] and to OpenAttack; this affects the novelty framing and the construct validity of the granularity buckets (e.g., function-name inflection labeled statement-level), but it does not force any particular empirical result. Accordingly, no significant circularity is present; a mild score of 2 reflects the self-referential taxonomy attribution, not a reduction of the findings to the inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted parameters or invented entities appear. The paper is an empirical evaluation using existing perturbation tools, metrics, and benchmarks; its load-bearing assumptions are the representativeness of ReCode and OpenAttack perturbations and the correctness of the paper's granularity labels.

assumptions (4)
  • domain assumption Functional correctness measured by pass@1 unit tests on HumanEval and MBPP is a valid proxy for code-generation robustness.
    The paper's metrics and conclusions all depend on this; Section 5.1 uses test-case pass rates, and Section 8.1 acknowledges it misses security, quality, and intent defects.
  • domain assumption ReCode and OpenAttack perturbation sets are representative of real-world adversarial attacks on code completion.
    All conclusions are drawn from these toolkits; Section 4.7 selects them by availability and compatibility, not by demonstrated coverage of a real-world attack distribution.
  • ad hoc to paper Each perturbation method can be assigned to exactly one granularity level (character, word, or statement), and that assignment is correct.
    Table 4 contains questionable labels, such as FuncRenameInflectionalVariation and newline insertions as statement-level, and the main ranking of levels depends on these labels.
  • domain assumption Results are stable without repeated runs for most models; identical outcomes on two models justify not repeating the rest.
    Section 5.1 states experiments were repeated only for two models and no variance is reported for the remaining four.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial Attack Classification and Robustness Testing for Large Language Models for Code." pith.science (2026). https://pith.science/paper/QKSDY7HB

@misc{pith2026250607942,
  author       = {Pith},
  title        = {Pith review of: Adversarial Attack Classification and Robustness Testing for Large Language Models for Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKSDY7HB}},
  note         = {Machine review of arXiv:2506.07942}
}
read the original abstract

Large Language Models (LLMs) have become vital tools in software development tasks such as code generation, completion, and analysis. As their integration into workflows deepens, ensuring robustness against vulnerabilities especially those triggered by diverse or adversarial inputs becomes increasingly important. Such vulnerabilities may lead to incorrect or insecure code generation when models encounter perturbed task descriptions, code, or comments. Prior research often overlooks the role of natural language in guiding code tasks. This study investigates how adversarial perturbations in natural language inputs including prompts, comments, and descriptions affect LLMs for Code (LLM4Code). It examines the effects of perturbations at the character, word, and sentence levels to identify the most impactful vulnerabilities. We analyzed multiple projects (e.g., ReCode, OpenAttack) and datasets (e.g., HumanEval, MBPP), establishing a taxonomy of adversarial attacks. The first dimension classifies the input type code, prompts, or comments while the second dimension focuses on granularity: character, word, or sentence-level changes. We adopted a mixed-methods approach, combining quantitative performance metrics with qualitative vulnerability analysis. LLM4Code models show varying robustness across perturbation types. Sentence-level attacks were least effective, suggesting models are resilient to broader contextual changes. In contrast, word-level perturbations posed serious challenges, exposing semantic vulnerabilities. Character-level effects varied, showing model sensitivity to subtle syntactic deviations.Our study offers a structured framework for testing LLM4Code robustness and emphasizes the critical role of natural language in adversarial evaluation. Improving model resilience to semantic-level disruptions is essential for secure and reliable code-generation systems.

Figures

Figures reproduced from arXiv: 2506.07942 by the authors.

Figure 1
Figure 1. GitHub survey (Fig.1a) indicates 97% of developers adopting AI tools, highlighting the need for trust and clear guidelines to maximize their effectiveness in software development. Meanwhile, a prominent ML practitioner Santiago Valdarrama (Fig.1b) cautions practitioners: ”AI isn’t the threat; those who use AI are.” Prior research on adversarial attacks has generally focused on either natural language processing (NLP… view at source ↗
Figure 2
Figure 2. Our proposed approach in this study to address our RQs [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Task selection labels on the Huggingface portal showing NLP/ text category. The highlighted labels “Text Gener [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 22 canonical work pages

  1. [27]

    38 (2024)

    Adversarial attack classification and robustness testing for code generation models, Empirical Software Engineering p. 38 (2024). DOI https://www.dropbox.com/scl/fo/1zmopkdwx2s4o73j6lr1o/ALiXzdU8vI03OMepNYBVbBg?rlkey =6uax4qc1tqzx5trl0nso41p2c&st=768p2bqo&dl=0

    work page 2024

  2. [1]

    Nijkamp, B

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, C. Xiong, Codegen: An open large language model for code with multi-turn program synthesis, arXiv preprint arXiv:2203.13474 (2022)

    arXiv 2022

  3. [2]

    Fried, A

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.t. Yih, L. Zettlemoyer, M. Lewis, Incoder: A generative model for code infilling and synthesis, arXiv preprint arXiv:2204.05999 (2022)

    arXiv 2022

  4. [3]

    Batouta, R

    Z.I. Batouta, R. Dehbi, M. Talea, O. Hajoui, in2016 4th IEEE International Colloquium on Information Science and Technology (CiSt)(IEEE, 2016), pp. 200–205

    work page 2016

  5. [4]

    Bielik, M

    P. Bielik, M. Vechev, inInternational Conference on Machine Learning(PMLR, 2020), pp. 896–907

    work page 2020

  6. [5]

    Carlini, D

    N. Carlini, D. Wagner, in2017 ieee symposium on security and privacy (sp)(Ieee, 2017), pp. 39–57

    work page 2017

  7. [6]

    Metzen, T

    J.H. Metzen, T. Genewein, V. Fischer, B. Bischoff, On detecting adversarial perturbations, arXiv preprint arXiv:1702.04267 (2017)

    arXiv 2017

  8. [7]

    Ernst, in2nd Summit on Advances in Programming Languages (SNAPL 2017)(Schloss-Dagstuhl-Leibniz Zen- trum f¨ ur Informatik, 2017)

    M.D. Ernst, in2nd Summit on Advances in Programming Languages (SNAPL 2017)(Schloss-Dagstuhl-Leibniz Zen- trum f¨ ur Informatik, 2017)

    work page 2017

Show all 40 references
  1. [8]

    Anand, P

    M. Anand, P. Kayal, M. Singh, On adversarial robustness of synthetic code generation, arXiv preprint arXiv:2106.11629 (2021)

    2021 arXiv

  2. [9]

    S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V. Kumar, S. Tan, B. Ray, P. Bhatia, et al., Recode: Robustness evaluation of code generation models, arXiv preprint arXiv:2212.10264 (2022)

    2022 arXiv

  3. [10]

    G. Zeng, F. Qi, Q. Zhou, T. Zhang, Z. Ma, B. Hou, Y. Zang, Z. Liu, M. Sun, Openattack: An open-source textual adversarial attack toolkit, arXiv preprint arXiv:2009.09191 (2020)

    2020 arXiv

  4. [11]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H.P.d.O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  5. [12]

    R. Jia, P. Liang, Adversarial examples for evaluating reading comprehension systems, arXiv preprint arXiv:1707.07328 (2017) 34 Yang Liu et al

    2017 arXiv

  6. [13]

    Jha, C.K

    A. Jha, C.K. Reddy, inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37 (2023), vol. 37, pp. 14,892–14,900

    2023

  7. [14]

    Rizvi, I

    S. Rizvi, I. Williams, Analyzing transparency and malicious insiders prevention for cloud computing environment, Computers & Security137, 103622 (2024)

    2024

  8. [15]

    Fronza, A

    I. Fronza, A. Hellas, P. Ihantola, T. Mikkonen, inSoftware Quality: Quality Intelligence in Software and Systems Engineering: 12th International Conference, SWQD 2020, Vienna, Austria, January 14–17, 2020, Proceedings 12 (Springer, 2020), pp. 121–133

    2020

  9. [16]

    U. Alon, M. Zilberstein, O. Levy, E. Yahav, code2vec: Learning distributed representations of code, Proceedings of the ACM on Programming Languages3(POPL), 1 (2019)

    2019

  10. [17]

    U. Alon, S. Brody, O. Levy, E. Yahav, code2seq: Generating sequences from structured representations of code, arXiv preprint arXiv:1808.01400 (2018)

    2018 arXiv

  11. [18]

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al., Codebert: A pre-trained model for programming and natural languages, arXiv preprint arXiv:2002.08155 (2020)

    2020 arXiv

  12. [19]

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, et al., Graphcodebert: Pre-training code representations with data flow, arXiv preprint arXiv:2009.08366 (2020)

    2020 arXiv

  13. [20]

    De la Rosa, A

    J. De la Rosa, A. Fern´ andez, inIberLEF@ SEPLN(2022)

    2022

  14. [21]

    Roziere, J

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X.E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al., Code llama: Open foundation models for code, arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  15. [22]

    Huang, X

    W. Huang, X. Ma, H. Qin, X. Zheng, C. Lv, H. Chen, J. Luo, X. Qi, X. Liu, M. Magno, How good are low-bit quantized llama3 models? an empirical study, arXiv preprint arXiv:2404.14047 (2024)

    2024 arXiv

  16. [23]

    X. Yuan, P. He, Q. Zhu, X. Li, Adversarial examples: Attacks and defenses for deep learning, IEEE transactions on neural networks and learning systems30(9), 2805 (2019)

    2019

  17. [24]

    S. Garg, G. Ramakrishnan, Bae: Bert-based adversarial examples for text classification, arXiv preprint arXiv:2004.01970 (2020)

    2020 arXiv

  18. [25]

    T. Le, J. Lee, K. Yen, Y. Hu, D. Lee, Perturbations in the wild: Leveraging human-written text perturbations for realistic adversarial attack and defense, arXiv preprint arXiv:2203.10346 (2022)

    2022 arXiv

  19. [26]

    Szegedy, W

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks, arXiv preprint arXiv:1312.6199 (2013)

    2013 arXiv

  20. [28]

    Mastropaolo, L

    A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, G. Bavota, in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)(IEEE, 2023), pp. 2149–2160

    2023

  21. [29]

    Improta, P

    C. Improta, P. Liguori, R. Natella, B. Cukic, D. Cotroneo, Enhancing robustness of ai offensive code generators via data augmentation, Empirical Software Engineering30(1), 7 (2025)

    2025

  22. [30]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H¨ ost, M.C. Ohlsson, B. Regnell, A. Wessl´ en, et al.,Experimentation in software engineering, vol. 236 (Springer, 2012)

    2012

  23. [31]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al., Program synthesis with large language models, arXiv preprint arXiv:2108.07732 (2021)

    2021 arXiv

  24. [32]

    J. Cao, Z. Chen, J. Wu, S.c. Cheung, C. Xu, Can ai beat undergraduates in entry-level java assignments? benchmarking large language models on javabench, arXiv e-prints pp. arXiv–2406 (2024)

    2024

  25. [33]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, et al., Measuring coding challenge competence with apps (2021), arXiv preprint arXiv:2105.09938 (2021)

    2021 arXiv

  26. [34]

    Z. Yang, J. Shi, J. He, D. Lo, inProceedings of the 44th International Conference on Software Engineering(2022), pp. 1482–1493

    2022

  27. [35]

    J. Peng, L. Cui, K. Huang, J. Yang, B. Ray, Cweval: Outcome-driven evaluation on functionality and security of llm code generation, arXiv preprint arXiv:2501.08200 (2025)

    2025 arXiv

  28. [36]

    Valeev, R

    A. Valeev, R. Garaev, V. Lomshakov, I. Piontkovskaya, V. Ivanov, I. Adewuyi, Yabloco: Yet another benchmark for long context code generation, arXiv preprint arXiv:2505.04406 (2025)

    2025

  29. [37]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P.J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, The Journal of Machine Learning Research21(1), 5485 (2020)

    2020

  30. [38]

    Y. Lin, Z. Ren, N. Touzi, J. Yang, Random horizon principal-agent problems, SIAM Journal on Control and Optimiza- tion60(1), 355 (2022)

    2022

  31. [39]

    S Kumar, M

    S. S Kumar, M. Adam Lones, M. Maarek, H. Zantout, inProceedings of the 1st International Workshop on Large Language Models for Code(2024), pp. 88–93

    2024

  32. [40]

    Juristo, A.M

    N. Juristo, A.M. Moreno,Basics of software engineering experimentation(Springer Science & Business Media, 2013) Author Biography Y ang Liuis a Ph.D. candidate in the Department of Computer and Software Engineering at Polytechnique Montr´ eal. He received his M.Sc. degree from ...

    2013

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.