Enhancing Code Generation for Low-Resource Languages: No Silver Bullet

Alberto Martin-Lopez; Alessandro Giagnorio; Gabriele Bavota

REVIEW 2 major objections 4 minor 4 cited by

Enhancing Code Generation for Low-Resource Languages: No Silver Bullet

T0 review · 2 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Across six LLMs, no single technique fixes low-resource code generation; translation-example prompting is the only reliably safe gain.

desk verdict Useful head-to-head comparison of boosters for low-resource code generation, but the test-set epoch picking means the fine-tuning results need a reanalysis before the size-based pattern is taken as fact. read the letter →

arxiv 2501.19085 v1 pith:6KJWE2K6 submitted 2025-01-31 cs.SE

Alessandro Giagnorio , Alberto Martin-Lopez , Gabriele Bavota This is my paper · ORCID

classification cs.SE

keywords codegenerationlow-resourceprogramminglanguagesin-contextlearningfine-tuninglargelanguagemodelspass@1RRacket

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether any single technique can close the gap between LLM code generation in high-resource languages (Python, Java) and low-resource ones (R, Racket). Across six models—five openly released models from 1B to 33B parameters plus Copilot—it compares three in-context-learning prompts with two fine-tuning routes, measuring pass@1 (the fraction of single generations that pass unit tests) on the same 161 translated programming problems. Its central finding is that no technique dominates: fine-tuning gives the smallest model its largest gains, in-context learning becomes the stronger family as models grow, and fine-tuning the largest model drops performance below baseline. The practical conclusion is that prompting with Python-to-target translation examples is a cheap, broadly reliable default, while fine-tuning should be reserved for small models with enough data to update their weights.

What carries the argument

The comparison rests on five treatments applied to the same underlying problem prompts. Three are in-context learning: a prompt with two Python-to-target translation examples, a prompt with hand-written mapping rules from Python syntax to the target, and a standard two-shot prompt with target-language implementation examples. Two are training-based: fine-tuning on the released R and Racket function datasets, and the same fine-tuning preceded by a pre-training phase that asks the model to translate Python functions into the target language. All are evaluated with pass@1 (one generation per problem, 50 repetitions, temperature 0.2) on the 161 translated HumanEval programs, and the central comparison is how the gap between the two families shifts with parameter count.

What would settle it

Re-run the boosting comparison with the fine-tuning training stop point selected on a validation subset split from the training data, and evaluate only that stop point on the 161 test problems. If fine-tuning no longer beats in-context learning for the 1B model, or if the 33B model's fine-tuning degradation disappears, the size-based conclusion is an artifact of test-set peeking.

Watch

Extended reading notes

Core claim

The paper claims that model size, not the technique itself, determines which boosting strategy works for low-resource code generation. For the 1B model, fine-tuning (with or without a preceding translation pre-training phase) is the only family that substantially improves Racket generation, from 7.0% to 18.4% pass@1, and the same model barely moves under any in-context prompt. For 7B and 13B models the picture splits by language: in-context learning is ahead on R, fine-tuning is ahead on Racket. At 33B, fine-tuning degrades both languages relative to the unmodified baseline, while translation-example and few-shot prompts help; Copilot, which cannot be fine-tuned, also gains from these prompts. The paper's summary claim is that translation-example in-context learning is a safe bet across all models that can interpret complex prompts.

Load-bearing premise

The load-bearing premise is that scoring every fine-tuning training stop point on the test benchmark and keeping only the best score gives a fair comparison with prompting; if the stop point had to be chosen before seeing the test set, the apparent benefit of fine-tuning could shrink.

Editorial extensions

If this is right

For a 1B model, fine-tuning on a low-resource language is the only tested route that produces large gains; users of such models should not expect prompting alone to move the metric much.
For 7B–13B models, the best choice depends on the language: in-context learning is more reliable for R, fine-tuning for Racket, so selecting a technique requires a language-specific check.
For models around 33B and above, fine-tuning on small low-resource datasets is risky; it can degrade code generation below the unmodified model, whereas translation-example prompting reliably improves it.
Translation-example prompting is the most reliable cheap option: for models above 1B, it improves over the baseline in every tested language, while the three in-context prompts are not all consistently helpful.
Adding a code-translation pre-training stage before fine-tuning does not consistently beat fine-tuning alone, so the extra training cost is not justified by these results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the size-based ordering suggests a practical scaling rule — the amount of fine-tuning data needed to update a model usefully grows with its parameter count, so for a new low-resource language the first question is model size.
Editorial inference: because the paper chooses the best fine-tuning training stop point by looking at the test benchmark, the reported fine-tuning gains, especially for the 1B model, may be optimistic; a held-out validation split could change the ranking.
Editorial inference: translation-example prompting borrows knowledge from Python, so its success may depend on the target language resembling languages the model already knows; whether it helps a truly unseen language with very different syntax remains open.
Editorial inference: for closed commercial models that cannot be fine-tuned, translation-example prompting is the only tested boosting strategy, making it the obvious candidate for API-based low-resource code generation tools.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

You can trust the paper's main practical message: if you want to improve code generation for a low-resource language without spending on fine-tuning, in-context learning with translation examples is a safe, cheap bet for models of 7B and up. Translation rules are weaker, and few-shot helps less consistently. That conclusion is supported by the data and is worth knowing.

What is actually new here is the systematic comparison. They are the first, as far as I know, to put fine-tuning, few-shot prompting, translation-example prompts, translation-rule prompts, and a code-translation pre-training objective in the same harness across six models spanning 1B to 33B. The evaluation is careful: 50 repetitions, McNemar with Benjamini-Hochberg, and a replication package with per-epoch results. Give them credit for that.

The soft spot is real and it is exactly where the reader put it. In Section IV-C they say they evaluate each fine-tuning epoch on the MultiPL-E benchmark and report only the best. MultiPL-E is the same benchmark on which all final numbers are computed, so this is test-set peeking. It can only favor fine-tuning. The 33B degradation (fine-tuning worse than baseline) is robust because it appears even after oracle epoch selection, but the claimed advantage for the 1B and 7B models rests on comparisons that could shrink once you use a fixed epoch or a validation split. They shipped per-epoch results, so this is checkable without rerunning training; the revision should just do it.

Other limitations are milder and mostly acknowledged: two low-resource languages, two model families, no hyperparameter tuning, three epochs. The translation pre-training did not help, which is a null result but an honest one. I would not call the scope a load-bearing flaw; it is enough for an empirical study.

For a reader, this is a useful paper if you work on code generation for niche languages or on practical LLM adaptation. It deserves a serious peer review, but I would make the revision conditional on the epoch-selection fix. If the fine-tuning numbers hold, the size-based pattern is a decent contribution; if they do not, the ICL-safe-bet message survives anyway.

Referee Report

2 major / 4 minor

Summary. The paper presents an empirical study of techniques for improving LLM code generation in low-resource programming languages. It first measures the performance gap between high-resource languages (Python, Java) and four low-resource languages (Julia, Lua, R, Racket) across six LLMs (RQ1), then compares, on R and Racket, three in-context-learning variants (translation examples, translation rules, few-shot) and two fine-tuning variants (code-generation fine-tuning, and code-translation pre-training followed by fine-tuning) across DeepSeek Coder 1B/7B/33B, Code Llama 7B/13B, and GitHub Copilot (RQ2). The main findings are that fine-tuning helps the 1B model most, in-context learning is a safe and cheap improvement for larger models, and fine-tuning degrades DeepSeek Coder 33B; overall, no single technique is best across all settings.

Significance. The study addresses a timely and practical question, and its design has notable strengths: 50 repetitions per problem, statistical testing with McNemar's test and Benjamini-Hochberg correction, multiple model families and sizes, and a replication package with per-epoch results. If the quantitative conclusions are robust, the paper provides actionable guidance: use fine-tuning for sub-7B models and in-context learning with translation examples for larger models; the negative result for 33B fine-tuning is particularly noteworthy. However, the epoch-selection protocol currently biases the fine-tuning numbers upward, so the headline size-based pattern is not yet established.

major comments (2)

[IV-C, Table III] Section IV-C states: 'For fine-tuned models, we evaluate each epoch on the MultiPL-E benchmark and only report the best model's results.' Since MultiPL-E is also the test set on which all final pass@1 values are reported, selecting the best of three epochs on the test benchmark is a form of test-set peeking: it can only improve the apparent performance of fine-tuning relative to the baseline and to the in-context-learning techniques, which are each evaluated as a single configuration. This is load-bearing for the paper's central size-based conclusion. For example, in Table III the fine-tuning advantage for DeepSeek Coder 1B on R (16.7 vs 13.9 baseline) and for Code Llama 7B on R (14.6 vs 13.9) is small enough that a non-oracle epoch choice could change the qualitative ranking. I request a reanalysis using a fixed epoch (e.g., the final epoch, or an epoch selected on a held-out validation split of MultiPL-E), and that the per-epoch pass@1 values be reported in the paper or supplementary material so the sensitivity of the conclusions to the selection rule can be assessed. The 33B deterioration is less affected because it appears even under oracle selection, but the small- and mid-size comparisons must be rechecked.
[V (Internal validity), IV-B] Beyond epoch selection, the comparison uses unequal configuration protocols: fine-tuning results are the best over three epochs, while baseline and in-context-learning results are single configurations, and hyperparameters (learning rate, batch size, sequence length, number of epochs) are fixed to defaults. Section V acknowledges the lack of hyperparameter tuning, but the abstract and conclusion state the size-based ranking without this caveat. Since low-resource fine-tuning is expensive, I do not require a full hyperparameter search, but the paper should either provide a sensitivity analysis (e.g., a few learning rates or an early-stopping check on a small validation set) or explicitly frame the conclusion as holding under default hyperparameters and with a fair epoch-selection protocol. Without this, the 'no silver bullet' claim is not fully supported.

minor comments (4)

[Tables I and III] The RQ1 baselines in Table I are computed on the 157 programs common to all languages, while the RQ2 baselines in Table III use the full 161 programs; this difference is only mentioned in Section IV-C and not in the table captions, which can confuse readers comparing the two tables.
[IV-A5] The 'pre-training' step is performed on already-pretrained models and is actually a second fine-tuning on a translation objective; the authors acknowledge this in the text, but the terminology in the abstract and in Table III ('Pre-training & Fine-tuning') may still mislead readers, so consider renaming it to something like 'translation fine-tuning + generation fine-tuning'.
[IV-D] The statement that 'all ORs reported in the text are statistically significant' cannot be checked from the paper because all 162 statistical tests are relegated to the replication package; I suggest including a compact table of adjusted p-values for the comparisons cited in the text, or at least for the best-performing technique versus baseline for each model and language in Table III.
[III-A and III-E] The paper initially classifies languages by GitHub repository counts and later states that repository count is not a good discriminator (e.g., Julia vs R); the text should reconcile these statements, for example by noting that repository counts are only a rough proxy and are not the sole determinant of model performance.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study's comparisons are empirical and the benchmark is external to training data; best-epoch selection on MultiPL-E is a test-set bias, not a circular derivation.

full rationale

The paper's claims are empirical comparisons, not derived from assumed equations or from the quantities being predicted. The fine-tuning and pre-training datasets (MultiPL-T, Cassano et al.) are disjoint from the MultiPL-E benchmark problems on which pass@1 is computed, so the benchmark answers are not baked into the training inputs or the prompts. The in-context learning prompts (translation examples, translation rules, few-shot examples) are constructed from external documentation and datasets, and their effect is measured rather than assumed. The cited prior work is by different authors and is used as a data/benchmark source, not as a load-bearing self-citation or a uniqueness argument. One methodological caveat is Section IV-C: 'For fine-tuned models, we evaluate each epoch on the MultiPL-E benchmark and only report the best model's results.' This is test-set peeking and can inflate fine-tuning results relative to the single-configuration baselines and in-context learning; however, it does not make the conclusions equivalent to their inputs by construction, because the per-epoch pass@1 values still depend on model behavior and could all fall below baseline. That is a threat to validity, not circularity of the derivation chain.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The study introduces no new theoretical constructs or mathematically derived quantities. All parameters listed are experimental settings or hyperparameters chosen by hand. The axioms are standard empirical assumptions about benchmark validity and test suite correctness.

free parameters (5)

temperature = 0.2
Sampling temperature for model generations, chosen following Cassano et al. rather than tuned.
number of repetitions = 50
Number of independent generations per problem to estimate pass@1, set above the n=20 stabilization point referenced in the paper.
number of fine-tuning epochs = 3
Chosen because no substantial loss improvement was observed beyond epoch 2, but this may cap fine-tuning performance.
learning rate = 2e-5 (DeepSeek), 5e-5 (Code Llama)
Default values from model documentation, not tuned for this study.
max sequence length = 2048 for fine-tuning, 3072 for in-context learning
Set to accommodate prompt sizes, with different values for the two technique families.

assumptions (3)

domain assumption The MultiPL-E benchmark translations of HumanEval problems are valid measures of code generation performance in R and Racket.
The paper treats pass@1 on these problems as the ground truth for code generation capability, citing Cassano et al. for validity.
domain assumption The fine-tuning datasets from MultiPL-T are correctly translated and high-quality representations of R and Racket code.
The paper adopts Cassano et al.'s datasets without independent verification of translation correctness.
standard math Statistical tests (McNemar and Benjamini-Hochberg) are appropriate for comparing dichotomous pass@1 outcomes across paired models and techniques.
The paper uses these standard tests, assuming independence of paired samples and correct false discovery rate control.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Code Generation for Low-Resource Languages: No Silver Bullet." pith.science (2026). https://pith.science/paper/6KJWE2K6

@misc{pith2026250119085,
  author       = {Pith},
  title        = {Pith review of: Enhancing Code Generation for Low-Resource Languages: No Silver Bullet},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KJWE2K6}},
  note         = {Machine review of arXiv:2501.19085}
}

read the original abstract

The advent of Large Language Models (LLMs) has significantly advanced the field of automated code generation. LLMs rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages. For low-resource languages (i.e., niche programming languages characterized by the scarcity of training data), the limited availability of such data hampers the models' ability to generalize effectively, resulting in poorer code generation performance as compared to high-resource languages. For this reason, there is a quest for techniques able to close this performance gap. We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages, namely: (i) a classic fine-tuning, which is however capped in size by the scarcity of training data; (ii) three variants of in-context learning, with prompts crafted to provide the LLM with additional information about the low-resource language (e.g., few-shot examples showcasing features of the targeted language); and (iii) a pre-training objective teaching the model how to translate between high- and low-resource languages. The context of our study are two low-resource languages (R and Racket) and six LLMs having different architectures and sizes. Our findings reveal that a fine-tuning is usually the best choice for smaller LLMs, possibly due to the fact that even a small dataset is sufficient to train their limited number of parameters. With the increase in size of the models, in-context learning becomes more and more effective, representing a safe and cheap bet (i.e., it always helps, but with different magnitudes). Differently, very large LLMs may deteriorate their performance on low-resource languages when fine-tuning is performed, possibly due to the lack of enough data needed to effectively update their weights.

Figures

Figures reproduced from arXiv: 2501.19085 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Multi-Source and Cross-Scenario Strategy-Guided Code Optimization
cs.SE 2026-07 conditional novelty 7.0 of 10

MoST improves LLM-guided code optimization by clustering optimization strategies from heterogeneous knowledge sources and transferring them across programming languages.
Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation
cs.LG 2026-07 conditional novelty 6.0 of 10

Left-shifting iterative compiler/test refinement into verified SFT data, then GRPO on difficulty-curated IO rewards, lifts Qwen3-8B Julia pass@1 past prior SOTA at 1/3 data and 1/6 cost, and bootstraps Ballerina.
CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer
cs.SE 2025-10 conditional novelty 6.0 of 10

Using Python-generated test oracles and multi-temperature sampling, CodeChemist selects low-resource-language code by execution pass rate and improves Pass@1 up to 69.5% over vanilla sampling.
LLM-based Property-based Test Generation for Guardrailing Cyber-Physical Systems
cs.SE 2025-05 conditional novelty 5.0 of 10

An LLM-based tool, ChekProp, extracts safety properties from CPS code and documentation and generates property-based tests that can serve as design-time checks and proposed runtime guardrails.

Reference graph

Works this paper leans on

37 extracted references · 22 canonical work pages · cited by 4 Pith papers

[1]

GitHub Copilot – Y our AI pair programmer,

“GitHub Copilot – Y our AI pair programmer,” https://github.com/features/copilot/ , accessed: 2024-03-10

work page 2024
[2]

Knowledge transfer from high-resource to low-resource pr ogramming languages for code llms,

F. Cassano, J. Gouwar, F. Lucchetti, C. Schlesinger, A. F reeman, C. J. Anderson, M. Q. Feldman, M. Greenberg, A. Jangda, and A. Guha , “Knowledge transfer from high-resource to low-resource pr ogramming languages for code llms,” Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA2, pp. 677–708, 2024

work page 2024
[3]

On the transferab ility of pre- trained language models for low-resource programming lang uages,

F. Chen, F. Fard, D. Lo, and T. Bryksin, “On the transferab ility of pre- trained language models for low-resource programming lang uages,” in 30th IEEE/ACM International Conference on Program Compreh ension, ICPC, 2022, pp. 401–412

work page 2022
[4]

Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study

T. van Dam, F. van der Heijden, P . de Bekker, B. Nieuwschep en, M. Otten, and M. Izadi, “Investigating the performance of la nguage models for completing code in functional programming langu ages: a haskell case study,” arXiv preprint arXiv:2403.15185 , 2024

work page Pith review arXiv 2024
[5]

Multi-lingual evaluat ion of code generation models,

B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y . Tian, M. Ta n, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V . Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qi an, M. K. Ramanathan, and R. Nallapati, “Multi-lingual evaluat ion of code generation models,” in The Eleventh International Conference on Learning Representatio...

work page 2023
[6]

Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

D. Guo, Q. Zhu, D. Y ang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al. , “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

arXiv 2024
[7]

Code llama: Open foundation models for code,

B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

arXiv 2023
[8]

Measuring the impact of program- ming language distribution,

G. Orlanski, K. Xiao, X. Garcia, J. Hui, J. Howland, J. Mal maud, J. Austin, R. Singh, and M. Catasta, “Measuring the impact of program- ming language distribution,” in International Conference on Machine Learning. PMLR, 2023, pp. 26 619–26 645

work page 2023

Show all 37 references

[9]

Codet5: Identiﬁe r-aware uniﬁed pre-trained encoder-decoder models for code unders tanding and generation,

Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identiﬁe r-aware uniﬁed pre-trained encoder-decoder models for code unders tanding and generation,” arXiv preprint arXiv:2109.00859 , 2021

2021 arXiv
[10]

Codegen: An open large language model for code with multi-turn program synthesis,

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S . Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, Ma y 1-5,

2023
[11]

Palm: Scaling language modeling with pathways,

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P . Barham, H. W. Chung, C. Sutton, S. Gehrmann, P . Schuh, K. Sh i, S. Tsvyashchenko, J. Maynez, A. Rao, P . Barnes, Y . Tay, N. Sha zeer, V . Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bra dbur...

2023
[12]

Codet5+: Open code large language models for code understa nding and generation,

Y . Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. H. H oi, “Codet5+: Open code large language models for code understa nding and generation,” arXiv preprint, 2023

2023
[13]

Starcoder: may the source be with you!

R. Li, L. B. allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. LI, J. Chim, Q. Liu, E. Zheltonozhskii , T. Y . Zhuo, T. Wang, O. Dehaene, J. Lamy-Poirier, J. Monteiro, N. G ontier, M.-H. Y ee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, ...

2023
[14]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L . Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

2023 arXiv
[15]

Wizardcoder: Empowering code la rge language models with evol-instruct,

Z. Luo, C. Xu, P . Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code la rge language models with evol-instruct,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.n...

2024
[16]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Y uan, H. P . d. O. Pinto, J. Ka plan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

2021 arXiv
[17]

Program synthesis with large language models,

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al. , “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

2021 arXiv
[18]

Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneva l-x,

Q. Zheng, X. Xia, X. Zou, Y . Dong, S. Wang, Y . Xue, Z. Wang, L. Shen, A. Wang, Y . Li et al. , “Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneva l-x,” arXiv preprint arXiv:2303.17568, 2023

2023 arXiv
[19]

Multipl-e: a scalable and polyglot approach to benchmarki ng neural code generation,

F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-C ostin, D. Pinckney, M.-H. Y ee, Y . Zi, C. J. Anderson, M. Q. Feldman et al. , “Multipl-e: a scalable and polyglot approach to benchmarki ng neural code generation,” IEEE Transactions on Software Engineering , 2023

2023
[20]

Is your code gene rated by chatgpt really correct? rigorous evaluation of large langu age models for code generation,

J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code gene rated by chatgpt really correct? rigorous evaluation of large langu age models for code generation,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024
[21]

Multilingual training for s oftware engineering,

T. Ahmed and P . T. Devanbu, “Multilingual training for s oftware engineering,” in 44th IEEE/ACM International Conference on Software Engineering, ICSE , 2022, pp. 1443–1455

2022
[22]

A survey on llm-based co de generation for low-resource and domain-speciﬁc programming language s,

S. Joel, J. J. Wu, and F. H. Fard, “A survey on llm-based co de generation for low-resource and domain-speciﬁc programming language s,” arXiv preprint arXiv:2410.03981, 2024

2024
[23]

Llama 2: Open foundation and ﬁne-tuned chat models,

H. Touvron, L. Martin, K. Stone, P . Albert, A. Almahairi , Y . Babaei, N. Bashlykov, S. Batra, P . Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and ﬁne-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

2023 arXiv
[24]

Codex website,

“Codex website,” https://openai.com/blog/openai-c odex, [n.d.], ac- cessed: 2023-10-08

2023
[25]

Note on the sampling error of the differenc e between correlated proportions or percentages,

Q. McNemar, “Note on the sampling error of the differenc e between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

1947
[26]

Controlling the false discovery r ate: A practical and powerful approach to multiple testing,

B. Y oav and H. Y osef, “Controlling the false discovery r ate: A practical and powerful approach to multiple testing,” Journal of the Royal Statistical Society. Series B (Methodological) , vol. 57, no. 1, pp. 289– 300, 1995

1995
[27]

Replication package,

“Replication package,” https://doi.org/10.5281/ze nodo.13128630, [n.d.]

doi
[28]

The stack: 3 tb of permissively licensed source code,

D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferran dis, Y . Jernite, M. Mitchell, S. Hughes, T. Wolf et al., “The stack: 3 tb of permissively licensed source code,” arXiv preprint arXiv:2211.15533 , 2022

2022 arXiv
[29]

PEGASUS: pre- training with extracted gap-sentences for abstractive summarizati on,

J. Zhang, Y . Zhao, M. Saleh, and P . J. Liu, “PEGASUS: pre- training with extracted gap-sentences for abstractive summarizati on,” in 37th In- ternational Conference on Machine Learning, ICML , 2020, pp. 11 328– 11 339

2020
[30]

Decoupled weight decay re gularization,

I. Loshchilov and F. Hutter, “Decoupled weight decay re gularization,” in 7th International Conference on Learning Representations , ICLR , 2019

2019
[31]

D programming language,

“D programming language,” https://dlang.org/, [n.d. ], accessed: 2024- 07-24

2024
[32]

Haskell language,

“Haskell language,” https://www.haskell.org/, [n.d .], accessed: 2024-07- 24

2024
[33]

Phi- 3 technical report: A highly capable language model locally on your phone,

M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah , H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl et al. , “Phi- 3 technical report: A highly capable language model locally on your phone,” arXiv preprint arXiv:2404.14219 , 2024

2024 arXiv
[34]

Codegemma: Open code models based on gemma,

C. Team, “Codegemma: Open code models based on gemma,” arXiv preprint arXiv:2406.11409, 2024

2024 arXiv
[35]

Retrieval-augmented generation for large langua ge models: A survey,

Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Su n, and H. Wang, “Retrieval-augmented generation for large langua ge models: A survey,” arXiv preprint arXiv:2312.10997 , 2023

2023 arXiv
[36]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q . V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

2022
[2023]

OpenReview.net, 2023

2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

GitHub Copilot – Y our AI pair programmer,

“GitHub Copilot – Y our AI pair programmer,” https://github.com/features/copilot/ , accessed: 2024-03-10

work page 2024

[2] [2]

Knowledge transfer from high-resource to low-resource pr ogramming languages for code llms,

F. Cassano, J. Gouwar, F. Lucchetti, C. Schlesinger, A. F reeman, C. J. Anderson, M. Q. Feldman, M. Greenberg, A. Jangda, and A. Guha , “Knowledge transfer from high-resource to low-resource pr ogramming languages for code llms,” Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA2, pp. 677–708, 2024

work page 2024

[3] [3]

On the transferab ility of pre- trained language models for low-resource programming lang uages,

F. Chen, F. Fard, D. Lo, and T. Bryksin, “On the transferab ility of pre- trained language models for low-resource programming lang uages,” in 30th IEEE/ACM International Conference on Program Compreh ension, ICPC, 2022, pp. 401–412

work page 2022

[4] [4]

Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study

T. van Dam, F. van der Heijden, P . de Bekker, B. Nieuwschep en, M. Otten, and M. Izadi, “Investigating the performance of la nguage models for completing code in functional programming langu ages: a haskell case study,” arXiv preprint arXiv:2403.15185 , 2024

work page Pith review arXiv 2024

[5] [5]

Multi-lingual evaluat ion of code generation models,

B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y . Tian, M. Ta n, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V . Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qi an, M. K. Ramanathan, and R. Nallapati, “Multi-lingual evaluat ion of code generation models,” in The Eleventh International Conference on Learning Representatio...

work page 2023

[6] [6]

Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

D. Guo, Q. Zhu, D. Y ang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al. , “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

arXiv 2024

[7] [7]

Code llama: Open foundation models for code,

B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

arXiv 2023

[8] [8]

Measuring the impact of program- ming language distribution,

G. Orlanski, K. Xiao, X. Garcia, J. Hui, J. Howland, J. Mal maud, J. Austin, R. Singh, and M. Catasta, “Measuring the impact of program- ming language distribution,” in International Conference on Machine Learning. PMLR, 2023, pp. 26 619–26 645

work page 2023