Pith. sign in

REVIEW 4 major objections 4 minor 109 references

English prompts are not systematically better for code generation: Chinese prompts yield higher test-pass rates for Python across three LLMs, and no language dominates for Java.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 00:58 UTC pith:XOIOFEAT

load-bearing objection Solid, useful multilingual code-gen study; English-not-best holds, but Chinese-on-Python advantage needs translation-controlled follow-up before I'd trust it. the 4 major comments →

arxiv 2607.14816 v1 pith:XOIOFEAT submitted 2026-07-16 cs.SE cs.AI

Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

Saima Afrin , Alessandro Midolo , Camilo Escobar-Vel\'asquez , Mario Linares-V\'asquez , Weiyuan Ding , Bowen Xu , Massimiliano Di Penta , Antonio Mastropaolo This is my paper
classification cs.SE cs.AI
keywords language biasmultilingual promptscode generationcode qualitylarge language modelsstatic analysisempirical study
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper studies whether the language of a programming prompt changes the code an LLM produces, and finds that non-English prompts are not systematically worse than English—and can be better. Using 460 real-world coding tasks translated into Chinese, Hindi, Spanish, and Italian, it shows that English prompts do not consistently give the highest test-pass rates; Chinese prompts give higher Python correctness across all three tested LLMs, while for Java no language dominates. The language of the prompt changes code structurally—complexity, comment density, static-analysis warnings—but mostly in severity-minor style differences, not correctness-critical faults. It also finds that generated code frequently mixes English with the prompt language in comments and string literals. The implication is that developers can work in their own language without a quality penalty, but should expect language-mixed code and review it.

Core claim

The paper's central claim is that source code generated with prompts in languages other than English is not necessarily less correct, and that the effect of prompt language depends on both the programming language and the model. Concretely, prompting in Chinese consistently outperforms English on Python code generation across all three LLMs tested, while no language dominates for Java, where some combinations (e.g., Hindi with one model) substantially reduce compilation success. The same broad finding—non-English prompts do not systematically hurt correctness—holds on a class-level Python benchmark, though the Chinese advantage does not replicate there. Quality metrics and static-analysis wa

What carries the argument

The load-bearing instrument is a curated multilingual benchmark: 460 coding tasks (230 Python, 230 Java) from an existing real-world benchmark, translated into Chinese, Hindi, Spanish, and Italian by native-speaker co-authors with a GPT-4 draft as initial support, then manually refined to preserve technical meaning. The study couples this with an evaluation pipeline that measures functional correctness via test-pass rates, structural quality via cyclomatic complexity (branch-path count), non-comment lines of code, cognitive complexity, and comment density, and defect indicators from several static-analysis tools (pylint, flake8, PMD, SonarCloud). A manual annotation step classifies the langu

Load-bearing premise

The results assume the manually curated translations preserve the technical meaning and style of the original English prompts without introducing a language-dependent bias; if, for example, Chinese prompts were translated in a systematically more precise or directive style, the observed Chinese performance gain could be a translation artifact rather than an effect of the language itself.

What would settle it

Have independent native speakers translate the Chinese prompts back to English without seeing the originals, then run the same generation: if the back-translated English prompts produce the same high pass rates as the Chinese ones, the advantage is in the wording, not the language; conversely, if the advantage vanishes when the prompts are translated back, language-specific processing is at work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Developers can write prompts in their native language without a systematic correctness penalty, provided they validate the model-language combination on their tasks.
  • Chinese prompts are a viable option for Python code generation; the paper observes a consistent pass-rate advantage over English for all tested models.
  • For Java, no prompt language dominates; some language-model combinations (especially Hindi with Claude) sharply reduce compilation and pass rates, so caution is warranted.
  • Teams that want consistent native-language documentation must actively enforce it: identifiers default to English, and comments/literals mix languages; formatting tools can clean up most style warnings.
  • Code generated from non-English prompts is structurally different but not worse on the studied metrics; severity-aware analysis shows extra warnings are mostly minor.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Chinese advantage on Python is likely tied to training-data distribution, but since model training data is undisclosed this remains a hypothesis; it is testable by evaluating the same translations on additional state-of-the-art models and observing whether the gap tracks the model's language-support quality.
  • The mixed-lexicon finding suggests a lightweight remedy not tested in the paper: instructing the model to output all comments and string literals in a single target language, possibly combined with signature translation, may yield near-consistent localization for most models.
  • Because prompt language changes the algorithmic strategy the model selects (e.g., set-based vs. list-based deduplication), multilingual prompting could be used as a cheap way to generate diverse alternative implementations for the same task—useful for test-generation and code-review exercises.
  • The structural differences in generated code imply that teams adopting non-English prompts should add an automated formatting pass (e.g., a code formatter) as a standard post-generation step; this would likely remove most of the extra style/convention warnings observed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents a curated multilingual benchmark for code generation: 460 CoderEval tasks (230 Python, 230 Java) with English prompts translated and manually curated into Chinese, Hindi, Spanish, and Italian, plus ClassEval (100 Python classes) as a replication. Three LLMs (GPT-4o mini, DeepSeek-V3, Claude 3.5 Sonnet) generate code at temperature zero (ten iterations for CoderEval; ten for Claude on ClassEval, single for GPT/DeepSeek). The authors evaluate functional correctness (test pass rates), structural quality (Lizard, SonarCloud, flake8, pylint, PMD), and lexicon (identifiers/comments/literals). Headline findings: English prompts are not consistently best; Chinese prompts yield higher pass rates on CoderEval Python for all three LLMs; effects depend on programming language and model; generated code often mixes English with the prompt language in comments/literals.

Significance. The study is relevant and timely. Its strengths include manual curation by native-speaker domain experts, a statistical protocol with paired tests, BH adjustment, effect sizes, ten iterations per task, and a replication on ClassEval. The controlled Italian-signature experiment for RQ3 is a nice addition. The dataset and replication package are valuable assets. The main claims are externally measured against established benchmarks and tools, with no circularity. However, the interpretive weight placed on the Chinese-prompts advantage and the ClassEval generalization depends on translation equivalence and on a single-iteration design, which are the main risks.

major comments (4)
  1. [Section 3.2, Section 6, Table 3] The headline Chinese-prompt advantage on CoderEval Python rests on translations produced by a single native-speaker author per language from GPT-4 drafts, with no inter-translator reliability check. Section 6 concedes that 'having multiple independent translators per language would further strengthen translation reliability.' Because the Chinese advantage is consistent across all three LLMs (g=0.28/0.19/0.25), even small systematic differences in explicitness or style between the Chinese and English prompt sets could shift pass rates. Fig. 1 illustrates different algorithmic strategies, but the paper does not rule out that the Chinese translation was more specific (e.g., mentioning deduplication). Please add a translation-equivalence check (back-translation, second translator on a sample, or per-prompt specificity analysis) or soften the claim.
  2. [Section 3.1, Section 4.1, Table 3] The paper does not report the language of the repositories from which CoderEval Python tasks are drawn. If many originate from Chinese-language projects, the English docstring may itself be a translation, making the Chinese prompt closer to the original specification. This would confound the 'language bias' interpretation with task-origin effects. The external framings in Section 4.1 do not address this. Please analyze the task provenance or discuss this explicitly as a threat to the Chinese-advantage claim.
  3. [Section 3.1, Table 3] ClassEval pass rates for GPT and DeepSeek are based on a single iteration (n=100 classes per language), justified by Friedman tests in Section 3.5 that were run on CoderEval metrics and static-analysis warnings, not on pass rates or on ClassEval. Class-level pass rates are highly variable (Claude ranges 8.3–12.8%), so single-iteration estimates may not be representative. The ClassEval generalization claims (Section 4.1) should be supported by ten iterations for all models, or by demonstrated stability of pass rates across iterations for ClassEval.
  4. [Section 3.6, Table 3] McNemar's test is applied to per-iteration outcomes across all ten iterations per task. Since multiple iterations from the same task are correlated (same prompt, same model, only sampling noise), the pairs are not independent, which can inflate the chi-square statistic and produce false positives. The paper does not account for clustering by task. Consider per-task aggregated outcomes (e.g., majority pass), a mixed-effects logistic model, or a bootstrap at the task level for the comparisons in Table 3.
minor comments (4)
  1. [Section 3.5] The text says 'Cohen’s k' but should be 'Cohen's kappa'. Also, the thresholds for 'almost perfect' and 'very strong' should be stated explicitly (e.g., kappa ranges).
  2. [Table 9] In the Claude Python row, the value '9.26' appears without a clear decimal convention; other entries have two decimals inconsistently. Please harmonize decimal places and check for typos.
  3. [Abstract / Section 3.1] The abstract says '460 coding tasks spanning Python (230) and Java (230)', but the study also uses ClassEval (100 classes). Clarify that the 460 refers to CoderEval only, or include ClassEval in the count.
  4. [Section 4.1] The phrase 'pass@10 values' is used, but Table 3 column header says 'pass'. Since the table aggregates over ten iterations, please make the header consistent and define what the percentage represents (e.g., proportion of task–iteration pairs passing).

Circularity Check

0 steps flagged

No significant circularity: all findings are external measurements against CoderEval/ClassEval with no fitted parameters or self-derived definitions.

full rationale

The paper makes no derivation claim. It reports an empirical measurement study: prompts are translated and manually curated (Section 3.2), code is generated by three third-party LLMs (Section 3.3), and correctness is evaluated with the external CoderEval and ClassEval test suites (Section 3.4). The central claim that non-English prompts are not systematically worse, and the language-specific differences in Table 3, are observed outcomes of executing held-out tests, not consequences of a definition or of a fitted parameter renamed as a prediction. The acknowledged limitation that translations were produced by single native-speaker authors (Section 6: 'having multiple independent translators per language would further strengthen translation reliability') is a genuine construct-validity threat: differences could partly reflect translation style rather than language per se. But that is an experimental confound, not circularity, because the pass rates are not defined in terms of the translations, and the benchmark tests are independent of the prompt construction. Self-citations are limited and non-load-bearing: the replication package [5] and the CheckStyle-exclusion rationale citing the authors' prior work [4], where the exclusion is also justified by configuration-dependence (Section 3.4). No uniqueness theorem, imported ansatz, or self-validating benchmark is used to force a conclusion. Thus the paper is self-contained as an empirical study; score reflects no circularity found.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The central claims are empirical; no free parameters are fitted and no new entities are postulated. The findings rest on benchmark validity, translation fidelity, quality-proxy validity, and sampling-representativeness assumptions listed above.

axioms (5)
  • domain assumption CoderEval test pass/fail is a valid operationalization of functional correctness for a code generation task.
    Used to address RQ1; the paper notes tests may not be fully adequate but argues that a relative assessment across languages mitigates this (Section 6, construct validity).
  • domain assumption Code metrics (NLOC, cyclomatic/cognitive complexity, comment counts) and static-analysis warnings from flake8, pylint, PMD, and SonarCloud are reasonable proxies for code quality.
    Central to RQ2; the paper acknowledges that other tools or metrics could reveal different patterns (Section 6).
  • domain assumption Native-speaker co-author translations preserve the technical meaning and naturalness of the original English prompts.
    Load-bearing for all RQs; described in Section 3.2, with the limitation of no multiple independent translators acknowledged in Section 6.
  • domain assumption Temperature 0, fixed seed, and ten iterations (for Claude) produce representative samples, so a single iteration is sufficient for parts of the analysis.
    Section 3.3 and 3.5; Friedman tests on CoderEval are used to justify single-iteration ClassEval results for GPT and DeepSeek, an extrapolation.
  • domain assumption Keeping benchmark signatures in English while translating docstrings does not systematically distort the prompt-language comparison.
    Necessary for test execution and integration; the paper acknowledges this limitation and runs a partial Italian-signature ablation only for Python (Sections 3.5 and 6).

pith-pipeline@v1.3.0-alltime-deepseek · 36172 in / 12180 out tokens · 99450 ms · 2026-08-02T00:58:02.129334+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude. Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments. Our results show that (i) English prompts do not consistently produce the best functional correctness or code quality, (ii) the impact of prompt language depends on both the programming language and the LLM, and (iii) generated code frequently mixes English with the prompt language in comments and string literals. These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.

Figures

Figures reproduced from arXiv: 2607.14816 by Alessandro Midolo, Antonio Mastropaolo, Bowen Xu, Camilo Escobar-Vel\'asquez, Mario Linares-V\'asquez, Massimiliano Di Penta, Saima Afrin, Weiyuan Ding.

Figure 1
Figure 1. Figure 1: Comparison of code generated with Chinese(left) and English(right) prompts. Only the Chinese code passes all benchmark [PITH_FULL_IMAGE:figures/full_fig_p014_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Generated code from different language prompts: Chinese (left), and English (right). The Chinese code includes more comments. [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Generated code from Chinese (left), and English (right). The two codes look completely different despite the task being the [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Generated code from Italian (left), and English (right). The two codes exhibit differences in terms of [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Generated code from Hindi (left), and English (right). The two codes exhibit differences in terms of [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Generated code from Hindi (left), and English (right). The English code has a higher cyclomatic complexity than Hindi. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Generated code from Spanish (left), and English (right). The two codes exhibits differences in terms of [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

109 extracted references · 3 canonical work pages · 2 internal anchors

  1. [1]

    [n. d.]. Amazon CodeWhisperer. https://aws.amazon.com/codewhisperer/. Accessed: 2024-03-10

    2024

  2. [2]

    [n. d.]. GitHub Copilot – Your AI pair programmer. https://github.com/features/copilot/. Accessed: 2024-03-10

    2024

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report

    2023

  4. [4]

    Saima Afrin, Antonio Mastropaolo, Bowen Xu, and Massimiliano Di Penta. 2025. Is Quantization a Deal-breaker? Empirical Insights from Large Code Models.arXiv preprint arXiv:2507.09665(2025)

    Pith/arXiv arXiv 2025

  5. [5]

    Large Language Models for Multilingual Code Generation: A Benchmark and a Study on Code Quality

    Saima Afrin, Alessandro Midolo, Camilo Escobar-Velásquez, Mario Linares-Vasquez, Ding Weiyuan, Bowen Xu, Massimiliano Di Penta, and Antonio Mastropaolo. 2025.Replication Package of the paper "Large Language Models for Multilingual Code Generation: A Benchmark and a Study on Code Quality". https://github.com/saimaafrin/multilingual-prompt-llm-code-generation.git

    2025

  6. [6]

    Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fahmideh, Mst Shamima Aktar, and Tommi Mikkonen. 2023. Towards Human-Bot Collaborative Software Architecting with ChatGPT. InProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. 279–285

    2023

  7. [7]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code summarization). InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  8. [8]

    Alves, José Pombal, Nuno M

    Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An Open Multilingual Large Language Model for Translation-Related Tasks. https://doi.org/10.48550/arXiv.2402.17733 arXiv:2402.177...

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.17733 2024

  9. [9]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. https://doi.org/10.48550/arXiv.2305.10403 arXiv:2305.10403 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.10403 2023

  10. [10]

    Anthropic. n.d.. The Claude 3 Model Family: Opus, Sonnet, Haiku. https://www-cdn .anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf. Accessed: 2025-02-26

    2025

  11. [11]

    Chetan Arora, Ahnaf Ibn Sayeed, Sherlock Licorish, Fanyu Wang, and Christoph Treude. 2024. Optimizing Large Language Model Hyperparameters for Code Generation. arXiv:2408.10577 [cs.SE] 10.48550/arXiv.2408.10577

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.10577 2024

  12. [12]

    Owura Asare, Meiyappan Nagappan, and Nirmal Asokan. 2023. Is github’s copilot as bad as humans at introducing vulnerabilities in code? Empirical Software Engineering28, 6 (2023), 129

    2023

  13. [13]

    Owura Asare, Meiyappan Nagappan, and N. Asokan. 2024. A User-centered Security Evaluation of Copilot. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 158, 11 pages. https://doi.org/10.1145/3597503.3639154 Manuscript submitted to...

    arXiv 2024

  14. [14]

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. InProceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Confere...

    2023

  15. [15]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. Language models are few-shot learners. InProceedings of the 34th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 25 pages

    2020

  16. [16]

    Ann Campbell

    G. Ann Campbell. 2018. Cognitive complexity: an overview and evaluation. InProceedings of the 2018 International Conference on Technical Debt (Gothenburg, Sweden)(TechDebt ’18). Association for Computing Machinery, New York, NY, USA, 57–58. https://doi .org/10.1145/3194164.3194186

    arXiv 2018

  17. [17]

    Jialun Cao, Meiziniu Li, Ming Wen, and Shing-chi Cheung. 2025. A study on prompt design, advantages and limitations of chatgpt for deep learning program repair.Automated Software Engineering32, 1 (2025), 1–29

    2025

  18. [18]

    Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, and Hua Wu. 2023. ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages. InFindings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 1062...

    doi:10.18653/v1/2023.findings-acl.676 2023

  19. [19]

    CheckStyle Team. 2001. CheckStyle. https://checkstyle.org. Accessed: 2025-03-14

    2001

  20. [20]

    Guibin Chen, Chunyang Chen, Zhenchang Xing, and Bowen Xu. 2016. Learning a dual-language vector space for domain-specific cross-lingual question retrieval. InProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering(Singapore, Singapore)(ASE ’16). Association for Computing Machinery, New York, NY, USA, 744–755. https://do...

    arXiv 2016

  21. [21]

    Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, et al. 2024. A survey on evaluating large language models in code generation tasks.arXiv preprint arXiv:2408.16498(2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code

    2021

  23. [23]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems30 (2017)

    2017

  24. [25]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales.Educational and psychological measurement20, 1 (1960), 37–46

    1960

  25. [26]

    1988.Statistical Power Analysis for the Behavioral Sciences(2nd ed.)

    Jacob Cohen. 1988.Statistical Power Analysis for the Behavioral Sciences(2nd ed.). Routledge, New York

    1988

  26. [27]

    2005.PMD applied

    Tom Copeland. 2005.PMD applied. Vol. 10. Centennial Books San Francisco

    2005

  27. [28]

    DeepSeek-AI and Aixin Liu et al. 2025. DeepSeek-V3 Technical Report. https://doi.org/10.48550/arXiv.2412.19437 arXiv:2412.19437 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.19437 2025

  28. [29]

    DeepSeek-AI and Daya Guo et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https: //doi.org/10.48550/arXiv.2501.12948 arXiv:2501.12948 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2025

  29. [30]

    DeepSeek-AI and Xiao Bi et al. 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. https://doi .org/10.48550/ arXiv.2401.02954 arXiv:2401.02954 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.02954 2024

  30. [31]

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. Multilingual Jailbreak Challenges in Large Language Models. https: //doi.org/10.48550/arXiv.2310.06474 arXiv:2310.06474 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.06474 2024

  31. [32]

    Xi Ding, Rui Peng, Xiangping Chen, Yuan Huang, Jing Bian, and Zibin Zheng. 2024. Do code summarization models process too much information? function signature may be all that is needed.ACM Transactions on Software Engineering and Methodology33, 6 (2024), 1–35

    2024

  32. [33]

    Jean-Baptiste Döderlein, Mathieu Acher, Djamel Eddine Khelladi, and Benoit Combemale. 2022. Piloting copilot and codex: Hot temperature, cold prompts, or black magic?arXiv preprint arXiv:2210.14699(2022)

    Pith/arXiv arXiv 2022

  33. [34]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-Collaboration Code Generation via ChatGPT.ACM Trans. Softw. Eng. Methodol.33, 7, Article 189 (Sept. 2024), 38 pages. https://doi.org/10.1145/3672459

    doi:10.1145/3672459 2024

  34. [35]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-Level Code Generation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. IEEE/ACM

    2024

  35. [36]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating Large Language Models in Class-Level Code Generation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computing Machinery, New York...

    arXiv 2024

  36. [37]

    Jean-Baptiste Döderlein, Mathieu Acher, Djamel Eddine Khelladi, and Benoit Combemale. 2023. Piloting Copilot and Codex: Hot Temperature, Cold Prompts, or Black Magic? arXiv:2210.14699 [cs.SE] 10.48550/arXiv.2210.14699

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.14699 2023

  37. [38]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Llm-based test-driven interactive code generation: User study and empirical evaluation.IEEE Transactions on Software Engineering(2024)

    2024

  38. [39]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1469–1481. Manuscript submitted to ACM 32 Saima Afrin, Alessandro Midolo, Camilo Escobar-Velásquez, Mario Linares-Vásquez, Weiyuan Di...

    2023

  39. [40]

    Negar Foroutan, Mohammadreza Banaei, Karl Aberer, and Antoine Bosselut. 2023. Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention. InFindings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9422–...

    work page doi:10.18653/v1/2023.findings-emnlp.632 2023

  40. [41]

    Milton Friedman. 1937. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance.J. Amer. Statist. Assoc.32, 200 (1937), 675–701. https://doi.org/10.2307/2279372

    doi:10.2307/2279372 1937

  41. [42]

    GitHut. 2008. GitHub Language Stats. https://madnight.github.io/githut/#/pull_requests/2024/1. Accessed: 2025-03-03

    2008

  42. [43]

    Grissom and John J

    Robert J. Grissom and John J. Kim. 2005.Effect sizes for research: A broad practical approach(2nd edition ed.). Lawrence Earlbaum Associates

    2005

  43. [44]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting. https://doi.org/10.48550/arXiv.2305.07004 arXiv:2305.07004 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.07004 2023

  44. [45]

    Cordasco Ian and Ziade Tarek. 2011. Flake8:Your tool for style guide enforcement. https://flake8.pycqa.org/en/latest/. Accessed: 2025-03-03

    2011

  45. [46]

    Saki Imai. 2022. Is github copilot a substitute for human pair-programming? an empirical study. InProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 319–321

    2022

  46. [47]

    ISO/IEC. 2011. Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models

    2011

  47. [48]

    JetBrains. 2024. The State of Developer Ecosystem Report 2024. https://www.jetbrains.com/lp/devecosystem-2024/. Accessed: 2025-03-03

    2024

  48. [49]

    JetBrains Inc. 2025. Data Playground: Discover the latest software development trends and insights. https://www .jetbrains.com/lp/devecosystem- data-playground/ Accessed: 2025-03-14

    2025

  49. [50]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. https: //doi.org/10.48550/arXiv.2406.00515 arXiv:2406.00515 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.00515 2024

  50. [51]

    Mohammed Kharma, Soohyeon Choi, Mohammed AlKhanafseh, and David Mohaisen. 2025. Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis. https://doi.org/10.48550/arXiv.2502.01853 arXiv:2502.01853 [cs.CR]

    doi:10.48550/arxiv.2502.01853 2025

  51. [52]

    Avila, Jacob Brunelle, and Baba Mamadou Camara

    Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How Secure is Code Generated by ChatGPT? , 2445- 2451 pages. https://doi.org/10.1109/SMC53992.2023.10394237

    arXiv 2023

  52. [53]

    Kei Koyanagi, Dong Wang, Kotaro Noguchi, Masanari Kondo, Alexander Serebrenik, Yasutaka Kamei, and Naoyasu Ubayashi. 2024. Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot. InProceedings of the 21st International Conference on Mining Software Repositories(Lisbon, Portugal)(MSR ’24). Association for Computing Machi...

    arXiv 2024

  53. [54]

    Valentina Lenarduzzi, Nyyti Saarimäki, and Davide Taibi. 2020. Some SonarQube issues have a significant but small effect on faults and changes. A large-scale empirical study.Journal of Systems and Software170 (2020), 110750. https://doi.org/10.1016/j.jss.2020.110750

    arXiv 2020

  54. [55]

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured Chain-of-Thought Prompting for Code Generation.ACM Trans. Softw. Eng. Methodol.34, 2, Article 37 (Jan. 2025), 23 pages. https://doi.org/10.1145/3690635

    doi:10.1145/3690635 2025

  55. [56]

    Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2024. AceCoder: An Effective Prompting Technique Specialized in Code Generation.ACM Trans. Softw. Eng. Methodol.33, 8, Article 204 (Nov. 2024), 26 pages. https://doi.org/10.1145/3675395

    doi:10.1145/3675395 2024

  56. [57]

    Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haibo Hu, Xiaohong Zhang, and Meng Yan. 2023. Improving ChatGPT Prompt for Code Generation. arXiv:2305.08360 [cs.SE] 10.48550/arXiv.2305.08360

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.08360 2023

  57. [58]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems35 (2022), 1950–1965

    2022

  58. [59]

    Lizhi Liu. 2021. The rise of data politics: digital China and the world.Studies in Comparative International Development56, 1 (2021), 45–67

    2021

  59. [60]

    Le, and David Lo

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo. 2024. Refining ChatGPT- Generated Code: Characterizing and Mitigating Code Quality Issues.ACM Trans. Softw. Eng. Methodol.33, 5, Article 116 (June 2024), 26 pages. https://doi.org/10.1145/3643674

    doi:10.1145/3643674 2024

  60. [61]

    Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023. On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot. In45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2149–2160

    2023

  61. [62]

    Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages.Psychometrika12, 2 (1947), 153–157

    1947

  62. [63]

    Alessandro Midolo and Massimiliano Di Penta. 2025. Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMS. In2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC). 01–11. https://doi .org/10.1109/ICPC66645.2025.00020

    arXiv 2025

  63. [64]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification.Proc. ACM Softw. Eng.1, FSE, Article 103 (July 2024), 23 pages. https://doi.org/10.1145/3660810

    doi:10.1145/3660810 2024

  64. [65]

    Lincoln Murr, Morgan Grainger, and David Gao. 2023. Testing LLMs on Code Generation with Varying Levels of Prompt Specificity. https: //doi.org/10.48550/arXiv.2311.07599 arXiv:2311.07599 [cs.SE]

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2311.07599 2023

  65. [66]

    Sagnik Nath and So Yoon Yoon. 2024. WIP: Beyond Code: Evaluating ChatGPT, Gemini, Claude, and Meta AI as AI Tutors in Computer Science and Engineering Education. In2024 IEEE Frontiers in Education Conference (FIE). 1–5. https://doi.org/10.1109/FIE61694.2024.10893528 Manuscript submitted to ACM Large Language Models for Code Generation from Multilingual Pr...

    arXiv 2024

  66. [67]

    Sabato Nocera, Simone Romano, Rita Francese, and Giuseppe Scanniello. 2024. Training for Security: Results from Using a Static Analysis Tool in the Development Pipeline of Web Apps. InProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training(Lisbon, Portugal)(ICSE-SEET ’24). 253–263. https://doi....

    arXiv 2024

  67. [68]

    Sabato Nocera, Simone Romano, Rita Francese, and Giuseppe Scanniello. 2025. Software engineering education: Results from a training intervention based on SonarCloud when developing web apps.Journal of Systems and Software222 (2025), 112308. https://doi.org/10.1016/j.jss.2024.112308

    arXiv 2025

  68. [69]

    Alberto S Nunez-Varela, Héctor G Pérez-Gonzalez, Francisco E Martínez-Perez, and Carlos Soubervielle-Montalvo. 2017. Source code metrics: A systematic mapping study.Journal of Systems and Software128 (2017), 164–197

    2017

  69. [70]

    Zhang, Mark Harman, and Meng Wang

    Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2025. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Trans. Softw. Eng. Methodol.34, 2, Article 42 (Jan. 2025), 28 pages. https://doi.org/10.1145/3697010

    doi:10.1145/3697010 2025

  70. [71]

    paperswithcode. 2025. Code Generation on HumanEval. https://paperswithcode .com/sota/code-generation-on-humaneval [Online; accessed 27-May-2025]

    2025

  71. [72]

    Timo Pawelka and Elmar Juergens. 2015. Is this code written in English? A study of the natural language of comments and identifiers in practice. In2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). 401–410. https://doi.org/10.1109/ICSM.2015.7332491

    arXiv 2015

  72. [73]

    Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language models. In2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2339–2356

    2023

  73. [74]

    Qiwei Peng, Yekun Chai, and Xuhong Li. 2024. HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Le...

    2024

  74. [75]

    Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. Do Users Write More Insecure Code with AI Assistants?. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security(Copenhagen, Denmark)(CCS ’23). Association for Computing Machinery, New York, NY, USA, 2785–2799. https://doi.org/10.1145/3576915.3623157

    arXiv 2023

  75. [76]

    Rohith Pudari and Neil A Ernst. 2023. From copilot to pilot: Towards AI supported software development.arXiv preprint arXiv:2303.04142(2023)

    Pith/arXiv arXiv 2023

  76. [77]

    Alifia Puspaningrum, Muhammad Anis Al Hilmi, Darsih, Muhamad Mustamiin, and Maulana Ilham Ginanjar. 2023. Vulnerable Source Code Detection using SonarCloud Code Analysis. arXiv:2307.02446 [cs.CY] 10.48550/arXiv.2307.02446

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2307.02446 2023

  77. [78]

    PylintTeam. [n. d.]. Pylint - code analysis for Python. https://www.pylint.org/. Accessed: 2025-03-03

    2025

  78. [79]

    PYPL. [n. d.]. PYPL PopularitY of Programming Language. https://pypl.github.io/PYPL.html. Accessed: 2025-03-03

    2025

  79. [80]

    2023.R: A Language and Environment for Statistical Computing

    R Core Team. 2023.R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/

    2023

  80. [81]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.Journal of Machine Learning Research21, 140 (2020), 1–67. http: //jmlr.org/papers/v21/20-074.html

    2020

Showing first 80 references.