Pith. sign in

REVIEW 4 major objections 4 minor 63 references

A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that neural metrics like BERTScore, CodeBERTScore, and BARTScore cannot reliably tell correct from incorrect multilingual code comments, or real generations from random noise.

desk verdict A solid qualitative study with a valuable dataset and taxonomy, but the headline metric-failure claim needs quantitative support and label reliability checks before it lands. read the letter →

arxiv 2505.15469 v1 pith:2O2ZSRUN submitted 2025-05-21 cs.SE cs.AI

Jonathan Katzy , Yongcheng Huang , Gopal-Raj Panchu , Maksym Ziemlewski , Paris Loizides , Sander Vermeulen , Arie van Deursen , Maliheh Izadi This is my paper
classification cs.SEcs.AI
keywords multilingualcodecommentscommentgenerationopencodinglargelanguagemodelsqualitativeevaluationmetricsneural
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Code models are evaluated almost entirely in English, yet real code bases contain other languages. This paper asks how well five open code LLMs generate comments in Chinese, Dutch, English, Greek, and Polish, and whether standard automatic metrics can judge those comments. Using 12,500 expert-labeled generations from an open-coding study, it builds a 26-category error taxonomy and finds that linguistic errors grow far more than semantic errors in non-English contexts. It then shows that neural metrics such as BERTScore, CodeBERTScore, and BARTScore assign scores that overlap heavily between expert-correct and expert-incorrect comments, and cannot separate real outputs from random noise. The paper's central claim is that these metrics are not trustworthy judges of multilingual comment quality, so automatic scores overstate how well models handle non-English code.

What carries the argument

The argument rests on two linked instruments. First is the open-coding error taxonomy: 26 error types grouped into model-specific, linguistic, semantic, and syntax categories, built through five iterative labeling rounds by six authors and used to label 12,500 model-generated comments. Second is the noise-overlap test: for each generated comment, the authors score a 'uniform noise' sequence (random tokens from the model's tokenizer, same length) and a 'targeted noise' sequence (random tokens sampled from the surrounding code) and compare metric distributions. The taxonomy supplies the qualitative evidence of what goes wrong; the noise-overlap test supplies the quantitative evidence that neural metrics fail to distinguish meaningful completions from meaningless ones. Expert correctness labels (correct/partial/incorrect) provide the reference against which metric separation is judged.

What would settle it

Re-label a random sample of the released 12,500 comments with multiple independent native speakers per language and compute, for each metric, the area under the ROC curve separating expert-correct from expert-incorrect comments; if any neural metric consistently reaches an AUC above roughly 0.9 across languages, the paper's claim that these metrics cannot reliably differentiate correct from incorrect comments would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that modern neural evaluation metrics, despite their wide use, do not measure the correctness of generated code comments in multilingual settings. Across five languages and five 7B–8B code models, the authors show that expert-rated correct and incorrect comments receive substantially overlapping scores from BERTScore, CodeBERTScore, and BARTScore, and that these metrics also assign high scores to pure token noise sampled either uniformly from the tokenizer or from the surrounding code context. Word-level metrics like BLEU and ROUGE-L separate noise from real comments more clearly, though they score everything lower. The authors further find that models are often 'partially correct': they produce comments that are close but contain errors, and that Chinese, despite being well represented in training corpora, is the worst-performing language, while Greek shows the largest jump in linguistic errors (15.1× over English). The paper releases the 12,500 labeled generations to let others verify and build on these findings.

Load-bearing premise

The load-bearing premise is that the correctness labels assigned by a single native-speaker expert per language are reliable ground truth for judging the metrics; if those labels are noisy or biased, the reported overlap between correct and incorrect scores could change materially.

Editorial extensions

If this is right

  • Automatic model rankings for non-English code comment generation are unreliable if based on neural metrics, because the metrics cannot separate correct from incorrect comments.
  • Neural metrics overstate model quality: they score Chinese predictions nearly as high as English ones even though human raters mark Chinese correct 5.5× less often.
  • Practitioners should keep human verification in the loop when generating comments in non-English languages, especially for languages like Greek where linguistic errors jump.
  • Word-level metrics, while harsher, track human judgments across languages better than neural metrics in this study.
  • The released dataset of 12,500 labeled comments gives a common benchmark for testing whether future metrics can separate correct from incorrect and real from noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the noise-overlap test could serve as a cheap, model-agnostic sanity check for any new evaluation metric: before trusting a metric on real outputs, require that its score distribution for genuine predictions be clearly separated from uniform and targeted noise.
  • If neural metrics compress differences between languages as much as the strip plots suggest, then published English-vs-non-English performance gaps for code models may be understated; the true gap could be larger than reported.
  • Because linguistic errors rise far faster than semantic errors in non-English settings, we infer that improving tokenization and adding non-English code to training data may yield larger gains for multilingual code support than improving code reasoning alone.
  • The finding that targeted noise scores below uniform noise for most languages suggests these metrics respond to surface overlap with the surrounding context rather than to meaning; a direct test would be to paraphrase a correct comment and see whether the score drops as much as for noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies multilingual code-comment generation by five open-weight code LLMs (CodeGemma, CodeLlama, CodeQwen1.5, GraniteCode, StarCoder2) across five natural languages (Chinese, Dutch, English, Greek, Polish). Using a Fill-in-the-Middle setup with a three-token ground-truth prefix, the authors collect 12,500 model generations over 2,500 source files, label them through open coding, and derive a taxonomy of 26 error types. They further compare six automatic metrics (BLEU-4, ROUGE-L, METEOR, BERTScore, CodeBERTScore, BARTScore) against expert correctness labels and against two types of random noise. The central claims are that non-English generation introduces substantially more linguistic errors, that neural metrics assign high scores even when human raters judge outputs incorrect, and that neural metrics fail to separate correct from incorrect comments or real generations from noise. The dataset is publicly released.

Significance. If the central claims are supported, the paper provides a useful multilingual benchmark and a sobering result: commonly used neural metrics may give inflated and potentially misleading evaluations of non-English code-comment generation. The strengths of the work include the public release of 12,500 labeled generations, the use of open-weight models for reproducibility, a transparent qualitative methodology, and a taxonomy that can inform future multilingual code-generation research. However, the metric-reliability conclusions currently rest on visual inspection of strip plots and on expert labels whose reliability is not quantified, and the priming design introduces a systematic overlap with the reference. These load-bearing issues need to be addressed before the central negative claims can be accepted.

major comments (4)
  1. [§6.4 and Figure 4 (also §6.3 and Figure 3)] The central claim that neural metrics fail to reliably separate correct from incorrect comments, and real comments from noise, is supported only by visual overlap in strip plots. The abstract states a 'significant score overlap,' but no statistical test, confidence interval, or effect size is reported. I recommend adding quantitative separation measures per metric and per language, for example the area under the ROC curve for distinguishing correct vs. incorrect (or correct+partial vs. incorrect), rank-biserial correlation, or an overlap coefficient, with confidence intervals. The same applies to the noise-vs-real comparison in §6.3, where 'significant overlap' should be quantified rather than asserted from Figure 3.
  2. [§4.2 and §8] The correctness labels are the ground truth against which metric trustworthiness is judged, yet the manuscript reports no inter-rater reliability, no double-labeling proportion, and no adjudication procedure for the final 12,500 labels. Section 8 concedes that labeling bias remains a risk, but the concession does not quantify its effect. If each language was labeled by a single native-level author, label noise or systematic bias could directly create or inflate the observed score overlap in Figure 4, for example by placing near-correct paraphrases in the 'partial' bucket or by mislabeling copied erroneous comments. I request a reliability analysis (e.g., Cohen's kappa or Krippendorff's alpha on a labeled subset) and a sensitivity analysis showing that the metric-overlap conclusions are stable under plausible label noise.
  3. [§4.1 and Figure 1] The inference protocol adds the first three tokens of the ground-truth comment as a priming prefix. Every prediction therefore shares a guaranteed token-level prefix with the reference comment, which inflates all n-gram-based metrics (BLEU, ROUGE, METEOR) and can also affect embedding- and likelihood-based neural metrics. This is especially relevant to the noise comparisons in §6.3, because random noise does not contain this prefix. The manuscript does not control for this artifact or discuss its direction and magnitude. I recommend either ablating the priming prefix from the scored portion of the prediction, reporting scores computed after removing the first three tokens, or explicitly quantifying how much of the metric overlap is attributable to this design choice.
  4. [§4.2 and Table 1 (MS-ME3)] The correctness definition in §4.2 states that a correct prediction must include all of the information in the original comment. However, the error taxonomy's MS-ME3 (Training Data Memorization) explicitly covers cases where 'the prediction is the same as the erroneous ground truth' (e.g., a comment containing a grammatical error or a factual mismatch with the code). Under the §4.2 definition, such a prediction would satisfy the 'include all information in the original comment' criterion and could be labeled correct, even though it is tagged as a memorization error. The manuscript does not specify how this conflict was resolved during labeling. This is not a cosmetic issue: it can directly affect the counts in Table 2 and the score distributions in Figure 4, and therefore the central metric-reliability conclusion.
minor comments (4)
  1. [§4.1] The sentence 'Finally, 500 random files were sampled for each language, resulting in 2,500 generated comments for each language' is imprecise: 500 files × 5 models yields 2,500 predictions per language, not 2,500 comments. Please rephrase to avoid confusing files with generated comments.
  2. [Figure 3] The y-axis label for the BARTScore panel appears garbled ('□12 □10 □8 ...'), and the panels use different effective scales (BLEU-4 appears to be on a 0-100 scale while the other metrics are normalized to 0-1). Please harmonize the axes or state the scaling explicitly for each panel.
  3. [§6.3] The statement that 'the targeted noise performed worse than the uniform noise for all languages except Chinese' is ambiguous: 'worse' could mean lower metric scores or higher overlap with real generations. Please specify the intended meaning and report the corresponding quantitative comparison.
  4. [§6.4] The strip plots are difficult to read because thousands of points overlap. Consider adding summary distributions (e.g., overlaid kernel densities or box plots) and quantitative separation statistics, which would also directly support the paper's central claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: metric-trustworthiness conclusion is grounded in external expert labels and independent noise baselines.

full rationale

The paper's central claim is that neural metrics (BERTScore, CodeBERTScore, BARTScore) fail to separate expert-correct from expert-incorrect multilingual comments and also overlap with random noise. This claim is not derived by fitting or by self-citation: expert correctness labels (Section 4.2) are generated by native-speaker authors and are external to the metric scores; the noise distributions (Section 4.3) are constructed as random tokens independent of both labels and predictions; and the metric scores in Figures 3 and 4 are computed with public off-the-shelf metrics with no calibration against the human labels. The only self-citation with any load-bearing role is [12] as a starting point for the initial error taxonomy: 'An initial set of errors based on previous work [12, 46, 26] was selected and iteratively improved' (Section 4.2). That taxonomy was then revised over five iterations, and the final conclusions about metric reliability do not depend on the prior taxonomy. The acknowledged labeling-bias limitation (Section 8) is a validity threat to the human ground truth, not a circular reduction: noisy labels could in principle change the measured overlap, but this is a measurement-quality concern, not an identity between input and output. No equation is defined in terms of the quantity it predicts, no fitted parameter is renamed as a prediction, and no uniqueness theorem or external constraint is imported from the authors' prior work. The derivation chain is therefore self-contained with respect to its empirical inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several hand-chosen thresholds and domain assumptions. The most consequential is the 3-token priming, which means the evaluation measures constrained continuation, not full comment generation. The reliability of the single-expert labels and the representativeness of the noise distributions directly underpin the metric-failure claims. No fitted parameters are used to derive results; all reported numbers are descriptive statistics over the collected dataset.

free parameters (3)
  • Priming token count = 3
    Hand-chosen number of ground-truth tokens prepended to force the target language (Section 4.1). Without it, models generate English; all reported non-English performance is conditional on this priming.
  • Inference limit threshold = average comment length + 3 standard deviations
    Statistical cutoff used to filter files so all contexts fit within CodeGemma's 4,128-token limit (Section 4.1); affects which files are included in the dataset.
  • Minimum comment length = 10 tokens
    Hand-chosen filter applied across tokenizers to exclude short comments (Section 4.1); affects dataset composition.
assumptions (5)
  • ad hoc to paper The first 3 tokens of the ground-truth comment are given to the model during generation.
    Section 4.1: this changes the task from unassisted comment generation to constrained continuation, and all generation-quality results depend on it.
  • domain assumption OpenSubtitles2016 word-frequency lists are a valid basis for sampling representative code files in each language.
    Section 4.1: files are retrieved by searching GitHub for 100 files per common word; this may bias the corpus toward certain comment styles and topics.
  • domain assumption Single-expert correctness labels are reliable ground truth.
    Sections 4.2 and 8: one native-speaker expert per language labeled correctness; no inter-rater reliability or adjudication is reported.
  • domain assumption Uniform noise and targeted noise are valid proxies for meaningless completions.
    Section 4.3: the central claim that neural metrics cannot separate noise from real generations rests on these two noise distributions being representative of meaningless output.
  • domain assumption Visual score overlap implies metric failure.
    Section 6.4: the conclusion that metrics fail to differentiate correct from incorrect is based on strip-plot overlap, without statistical separation tests.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics." pith.science (2026). https://pith.science/paper/2O2ZSRUN

@misc{pith2026250515469,
  author       = {Pith},
  title        = {Pith review of: A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2O2ZSRUN}},
  note         = {Machine review of arXiv:2505.15469}
}
read the original abstract

Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in non-English contexts, identifying challenges in their adoption and integration into multilingual workflows. We conduct an open-coding study to analyze errors in code comments generated by five state-of-the-art code models, CodeGemma, CodeLlama, CodeQwen1.5, GraniteCode, and StarCoder2 across five natural languages: Chinese, Dutch, English, Greek, and Polish. Our study yields a dataset of 12,500 labeled generations, which we publicly release. We then assess the reliability of standard metrics in capturing comment \textit{correctness} across languages and evaluate their trustworthiness as judgment criteria. Through our open-coding investigation, we identified a taxonomy of 26 distinct error categories in model-generated code comments. They highlight variations in language cohesion, informativeness, and syntax adherence across different natural languages. Our analysis shows that, while these models frequently produce partially correct comments, modern neural metrics fail to reliably differentiate meaningful completions from random noise. Notably, the significant score overlap between expert-rated correct and incorrect comments calls into question the effectiveness of these metrics in assessing generated comments.

Figures

Figures reproduced from arXiv: 2505.15469 by the authors.

Figure 1
Figure 1. Example input used for inference was queried to gather 100 files containing the specific word. The collected files were then de-duplicated. Due to varying architectures, each model has its own maximum token length. To ensure a fair comparison between models, we filter the candidate files by their length after tokenization. We then select only files where all of the context fits within the context of the smallest mod… view at source ↗
Figure 2
Figure 2. Example Evaluation of a Prediction Granite-8B-code-base. Granite-8B-code-base is trained on a col￾lection of programming languages datasets, including GitHub Code Clean4 , StarCoderData5 , and GitHub issues, combined with natural language datasets from StackExchange, Arxiv, and OpenWebMath. English is predominant, with non-English data removed [28]. CodeLlama-7B. CodeLlama-7B builds on Llama2, whose train￾ing corpus… view at source ↗
Figure 3
Figure 3. Metric scores comparing LLM generated comments, to random samples of tokens using two separate distributions. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Strip plot, showing the scores assigned to comment generations by different metrics. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 43 canonical work pages

  1. [1]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel

  2. [2]

    John Airey and Cedric Linder. 2006. Language and the experience of learning university physics in sweden. EUROPEAN JOURNAL OF PHYSICS Eur. J. Phys , 27, (May 2006), 553–560. doi: 10.1088/0143-0807/27/3/009

    work page doi:10.1088/0143-0807/27/3/009 2006

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: an automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 65–72

    work page 2005

  4. [4]

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255

    arXiv 2022

  5. [5]

    Samuel Boguslawski, Rowan Deer, and Mark G Dawson. 2025. Programming education and learner motivation in the age of generative ai: student and educator perspectives. Information and Learning Sciences , 126, 1/2, 91–109

    work page 2025

  6. [6]

    A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, Richard Socher, and Dragomir R. Radev. 2020. Summeval: re-evaluating summarization evaluation. Transac- tions of the Association for Computational Linguistics , 9, 391–409

    work page 2020

  7. [7]

    Zhangyin Feng et al. 2020. CodeBERT: a pre-trained model for programming and natural languages. In Findings of the Association for Computational Lin- guistics: EMNLP 2020. Trevor Cohn, Yulan He, and Yang Liu, (Eds.) Association for Computational Linguistics, Online, (Nov. 2020)

    work page 2020

  8. [8]

    GitHub. 2022. The state of the octoverse: global tech talent report 2022. Ac- cessed: 2024-11-13. (2022). octoverse.github.com/2022/global-tech-talent

    work page 2022

Show all 63 references
  1. [9]

    Hubert Haider and Luka Szucsich. 2022. Theoretical Linguistics, 48, 1-2, 1–39. doi: doi:10.1515/tl-2022-2035

    2022 doi

  2. [10]

    Xing Hu, Xin Xia, David Lo, Zhiyuan Wan, Qiuyuan Chen, and Thomas Zim- mermann. 2022. Practitioners’ expectations on automated code comment gener- ation. In Proceedings of the 44th international conference on software engineering

    2022

  3. [11]

    Maliheh Izadi and et al. 2022. Codefill: multi-token code completion by jointly learning from structure and naming sequences. In Proceedings of the 44th International Conference on Software Engineering , 401–412

    2022

  4. [12]

    Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. 2024. Language models for code comple- tion: a practical evaluation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 1–13. 6https://huggin...

    2024

  5. [13]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A sur- vey on large language models for code generation.arXiv preprint arXiv:2406.00515

    2024 arXiv

  6. [14]

    Jonathan Katzy, Maliheh Izadi, and Arie van Deursen. 2023. On the impact of language selection for training and evaluating programming language models. In 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 271–276

    2023

  7. [15]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT. Vol. 1. Minneapolis, Minnesota, 2

    2019

  8. [16]

    Junaed Younus Khan and Gias Uddin. 2022. Automatic code documentation generation using gpt-3. (2022). eprint: 2209.02235

    2022 arXiv

  9. [17]

    Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large lan- guage models only pass primary school exams in Indonesia: a comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing . Houda Bouamor, J...

    2023 doi

  10. [18]

    M Lewis. 2019. Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    2019

  11. [19]

    Liang, Chenyang Yang, and Brad A

    Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2024. A large-scale survey on the usability of ai programming assistants: successes and challenges. In Pro- ceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24) Article 52. Association for C...

    2024

  12. [20]

    Chin-Yew Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, (July 2004), 74–81. https://aclanthology.org/W04-1013

    2004

  13. [21]

    Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth Inter- national Conference on Language Resources and Evaluation (LREC’16) . Nicoletta Calzolari et al., (Eds.) European Language...

    2016

  14. [22]

    Yinhan Liu. 2019. Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364

    2019 arXiv

  15. [23]

    Irene López-Navarro, Ana Ibáñez Moreno, Miguel Angel Quintanilla, and Jesús Rey-Rocha. 2015. Why do i publish research articles in english instead of my own language? differences in spanish researchers’ motivations across scientific domains. Scientometrics, 103, 939–976

    2015

  16. [24]

    Anton Lozhkov et al. 2024. Starcoder 2 and the stack v2: the next generation. (2024). https://arxiv.org/abs/2402.19173 arXiv: 2402.19173 [cs.SE]

    2024 arXiv

  17. [25]

    Wenhan Lyu, Yimeng Wang, Tingting (Rachel) Chung, Yifan Sun, and Yixuan Zhang. 2024. Evaluating the effectiveness of llms in introductory computer science education: a semester-long field study. In Proceedings of the Eleventh ACM Conference on Learning @ Scale (L@S ’24). Assoc...

    2024

  18. [26]

    Valerie R Mariana. 2014. The Multidimensional Quality Metric (MQM) frame- work: A new framework for translation quality assessment

    2014

  19. [27]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. (2013). arXiv: 1301.3781

    2013 arXiv

  20. [28]

    Mayank Mishra et al. 2024. Granite code models: a family of open foundation models for code intelligence. (2024). arXiv: 2405.04324 [cs.AI]

    2024 arXiv

  21. [29]

    Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2024. Read- ing between the lines: modeling user behavior and costs in ai-assisted program- ming. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24) Article 142. Association ...

    2024

  22. [30]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering , 1–13

    2024

  23. [31]

    Rajathurai Nishanthi. 2020. Understanding of the importance of mother tongue learning. International Journal of Trend in Scientific Research and Development

    2020

  24. [32]

    Jessica Ojo, Kelechi Ogueji, Pontus Stenetorp, and David Ifeoluwa Adelani

  25. [33]

    Yogendra Pal and Sridhar Iyer. 2015. Classroom versus screencast for native language learners: effect of medium of instruction on knowledge of program- ming. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’15). Asso...

    2015 doi

  26. [34]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 311–318

    2002

  27. [35]

    Timo Pawelka and Elmar Juergens. 2015. Is this code written in english? a study of the natural language of comments and identifiers in practice. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME)

    2015

  28. [36]

    Edwin George Pulleyblank. 1995. Outline of classical Chinese grammar

    1995

  29. [37]

    Jipeng Qiang, Yang Li, Chaowei Zhang, Yun Li, Yi Zhu, Yunhao Yuan, and Xindong Wu. 2023. Chinese idiom paraphrasing.Transactions of the Association for Computational Linguistics, 11, (July 2023), 740–754

    2023

  30. [38]

    Baptiste Roziere et al. 2023. Code llama: open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  31. [39]

    Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M

    Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021. Perturbation CheckLists for evaluating NLG evaluation metrics. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Marie-Francine Moens, Xuanjing Hua...

    2021

  32. [40]

    Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik, Dieter Theiler, Armin Haberl, Stefan Thalmann, and Dominik Kowald. 2024. Reproducibility in machine learning-based research: overview, barriers and drivers. CoRR, abs/2406.14325. https://doi.org/10.48550/arXiv.2406.14325

    2024 doi arXiv

  33. [41]

    Florian Tambon, Arghavan Moradi-Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Giuliano Antoniol. 2025. Bugs in large language models generated code: an empirical study. Empirical Software Engineering

    2025

  34. [42]

    CodeGemma Team et al. 2024. Codegemma: open code models based on gemma. (2024). https://arxiv.org/abs/2406.11409 arXiv: 2406.11409 [cs.CL]

    2024 arXiv

  35. [43]

    Gemma Team et al. 2024. Gemma: open models based on gemini research and technology. (2024). arXiv: 2403.08295

    2024 arXiv

  36. [44]

    Qwen Team. 2024. Code with codeqwen1.5. (Apr. 2024). https://qwenlm.github .io/blog/codeqwen1.5/

    2024

  37. [45]

    Qwen Team. 2024. Introducing qwen1.5. (Feb. 2024). https://qwenlm.github.io /blog/qwen1.5/

    2024

  38. [46]

    Tezcan, Arda and Hoste, Veronique and Macken, Lieve. 2017. SCATE taxonomy and corpus of machine translation errors. eng. InTrends in E-tools and resources for translators and interpreters . Approaches to Translation Studies. Vol. 45. Pastor, Gloria Corpas and Durán-Muñoz, Isab...

    2017

  39. [47]

    Anastasios Toumazatos, John Pavlopoulos, Ion Androutsopoulos, and Stavros Vassos. 2024. Still all Greeklish to me: Greeklish to Greek transliteration. InPro- ceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LR...

    2024

  40. [48]

    Hugo Touvron et al. 2023. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  41. [49]

    Dimitra Irini Tzanidaki. 1995. Greek word order: towards a new approach. UCLWorking Paper in Linguistics, 7, 247–277

    1995

  42. [50]

    Lieke Verheijen and Roeland van Hout. 2022. Manifold code-mixing in computer- mediated communication: the use of english in dutch youths’ informal online writing. Ampersand, 9, 100091. doi: https://doi.org/10.1016/j.amper.2022.10009 1

    2022

  43. [51]

    Antonio Vitale, Rocco Oliveto, and Simone Scalabrino. 2024. A catalog of data smells for coding tasks. ACM TOSEM

    2024

  44. [52]

    Tamás Kovács Florian Atzenhofer-Baumgartner Sandy Aoun Anguelos Nico- laou Daniel Luger Franziska Decker Florian Lamminger Georg Vogeler. 2022. Langdetect (revision 0215f72). (2022). doi: 10.57967/hf/0135

    2022 doi

  45. [53]

    Chaozheng Wang, Zongjie Li, Cuiyun Gao, Wenxuan Wang, Ting Peng, Hail- iang Huang, Yuetang Deng, Shuai Wang, and Michael R. Lyu. 2024. Exploring multi-lingual bias of large code models in code generation. (2024)

    2024

  46. [54]

    Witchel, Christopher I

    Harry J. Witchel, Christopher I. Jones, Georgina A. Thompson, Carina E. I. Westling, Juan Romero, Alessia Nicotra, Bruno Maag, and Hugo D. Critchley

  47. [55]

    Ziang Xiao, Susu Zhang, Vivian Lai, and Q Vera Liao. 2023. Evaluating evalu- ation metrics: a framework for analyzing nlg evaluation metrics using mea- surement theory. In 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023. Association for Computat...

    2023

  48. [56]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: evaluating generated text as text generation. NeurIPS, 34, 27263–27277

    2021

  49. [57]

    Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. Don’t trust chatgpt when your question is not in english: a study of multilingual abilities and types of llms. (2023). arXiv: 2305.16339 [cs.CL]

    2023 arXiv

  50. [58]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: evaluating text generation with bert. In International Conference on Learning Representations

    2020

  51. [59]

    Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. Code- BERTScore: evaluating code generation with pretrained models of code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Houda Bouamor, Juan Pino, and Kalika Bali, (Eds...

    2023

  52. [60]

    Albert Ziegler, Eirini Kalliamvakou, Shawn Simister, Ganesh Sittampalam, Al- ice Li, Andrew Rice, Devon Rifkin, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. (2022). arXiv: 2205.06537 [cs.SE]

    2022 arXiv

  53. [2022]

    Frontiers in Psychology, 13, 873844

    Spelling errors in brief computer-mediated texts implicitly lead to linearly additive penalties in trustworthiness. Frontiers in Psychology, 13, 873844

  54. [2024]

    How good are large language models on african languages? (2024)

    2024

  55. [2025]

    Can llms replace manual annotation of software engineering artifacts? In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.