REVIEW 3 major objections 5 minor 24 references
A Preliminary Study of Multilingual Code Language Models for Code Generation Task Using Translated Benchmarks
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Poly-Coder's language ranking flips between two translated HumanEval benchmarks
desk verdict Useful new pass@1 measurements, but the abstract's validation claim is contradicted by the paper's own Java finding—send to review, expect major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is Poly-Coder, an open multilingual code model trained on twelve languages with known per-language perplexity, evaluated through the pass@1 metric at temperature 0.2. The two translated benchmarks are the load-bearing comparators: MultiPL-E, which automatically transpiles HumanEval problems into 18 languages with a few incomplete translations, and HumanEvalSynthesize, a manual, complete translation of all 164 problems into five languages shipped with OctoPack. The evaluation harness used for HumanEvalSynthesize does the mediating work, and the paper's central doubt is whether that harness faithfully reproduces the original benchmark.
What would settle it
Run the same 164 HumanEval problems and the same Poly-Coder checkpoints through the original OctoPack evaluation scripts and through the general harness the authors used, keeping temperature at 0.2. If Java stays above Rust in both, the paper's reversal is an artifact or a translation difference; if the two runs disagree, the HumanEvalSynthesize numbers in Table 3 are a property of the harness, not the benchmark.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that translated HumanEval benchmarks give a usable but unstable picture of multilingual code-model performance. Using Poly-Coder's 160M, 0.4B, and 2.7B variants, pass@1 generally rises with model size in both benchmarks, and the overall pattern is close enough to the perplexity rankings reported when Poly-Coder was released that the authors take this as provisional validation of the translated-benchmark approach. The qualification is visible in the data itself: in MultiPL-E, Java outperforms Python at the two larger sizes, which 'does not align with the perplexity scores'; in HumanEvalSynthesize, Rust beats Java and JavaScript for the 2.7B model, nearly reversing the order. The authors attribute such reversals to translation methodology and to implementing HumanEvalSynthesize through a general evaluation framework that may differ from the original OctoPack setup, so they stop short of endorsing any single translated benchmark as authoritative.
Load-bearing premise
The HumanEvalSynthesize half of the study is only meaningful if the evaluation harness the authors used is a faithful reproduction of the original OctoPack setup; Section 6 says the implementation 'may differ from the original setup,' and if that is true the table does not measure HumanEvalSynthesize.
Editorial extensions
If this is right
- If translated benchmarks do track perplexity, they can serve as a proxy for multilingual capability when perplexity is unavailable or untrusted.
- If benchmark choice changes which language appears strongest, then single-benchmark multilingual comparisons are not reliable evidence; a result should be confirmed on a second translated benchmark.
- The consistently low pass@1 across all languages, even for the largest Poly-Coder model, warns that small absolute differences between languages can be magnified into misleading conclusions.
- The paper's validation recipe—checking a new translated benchmark against an already-known model's performance—becomes a required step before a translated benchmark is adopted.
- If the harness mismatch is real, the HumanEvalSynthesize column should be read as a reimplementation result, not as a result for OctoPack's published benchmark.
Reading between the lines
- A per-language correlation test between perplexity and pass@1 would settle the paper's validation claim more directly than the overall pattern, since the Java and Rust reversals show that an aggregate agreement can hide per-language failures.
- The same two benchmark families could be compared on a stronger model, such as an instruction-tuned code model, to see whether the ranking reversals are a Poly-Coder quirk or a property of the benchmarks themselves.
- If the harness limitation is confirmed, then published translated benchmarks should start reporting which evaluation script they used, with version hashes, because the paper shows the script can change the result.
- A natural extension is to translate the same benchmark with both methodologies for the same five languages and compute agreement on a per-problem basis, which would isolate translation quality from evaluation-harness effects.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a preliminary empirical study of PolyCoder (160M, 0.4B, and 2.7B) on two translated versions of HumanEval, MultiPL-E and HumanEvalSynthesize, across Python, JavaScript, Java, and Rust, using pass@1 at temperature 0.2. It presents pass rates in Tables 2 and 3, observes several inconsistencies between benchmarks and with model-size trends, and discusses reproducibility challenges. The abstract claims the results align well with perplexity-based metrics, thereby validating translated benchmarks; this claim is the load-bearing conclusion the paper must support.
Significance. The research question is relevant: translated benchmarks are increasingly used to evaluate multilingual code generation, and independent empirical checks are valuable. The paper's strengths are that it uses publicly available models and benchmark pipelines, reports raw pass@1 scores in readable tables, and explicitly acknowledges limitations in Section 6. If the alignment-with-perplexity claim were supported, the result would be a useful validity check for MultiPL-E and HumanEvalSynthesize. However, the paper provides no statistical analysis, and its own data contradict the claim; the contribution is therefore only a small, partially interpretable data point, not a validation of translated benchmarks.
major comments (3)
- [Abstract and Section 4 (Tables 2-3)] The central validation claim in the abstract is contradicted by the paper's own results. Table 2 shows that in MultiPL-E, Java is the best-performing language for the 0.4B and 2.7B PolyCoder models (3.83% and 5.63%), and Section 4 explicitly states that this 'does not align with the perplexity scores' from Xu et al. Table 3 shows that in HumanEvalSynthesize, Rust (a low-resource language) is the second-best language for the 2.7B model at 3.05%, which also contradicts the expected perplexity ordering. No correlation coefficient, confidence interval, or significance test is provided. The abstract's statement that outcomes 'align well with evaluation metrics used during the training phase, such as perplexity' is therefore unsupported and, by the paper's own account, false; the paper's main conclusion must be reframed or the claim must be withdrawn.
- [Section 3.2 and Section 6] The HumanEvalSynthesize measurements were obtained with the bigcode-evaluation-harness rather than the OctoPack authors' original setup. Section 6 concedes that this implementation 'may differ from the original setup described in the OctoPack study.' If the harness is not equivalent, Table 3 does not measure HumanEvalSynthesize, and the cross-benchmark comparisons in Section 4 (e.g., performance being 'almost reversed' between benchmarks) are not valid. The authors should either demonstrate equivalence to the original OctoPack evaluation or explicitly treat Table 3 as results on the BigCode re-implementation, not on HumanEvalSynthesize itself.
- [Section 4, Tables 2 and 3] The language-level conclusions are drawn from very small absolute differences without uncertainty quantification. For instance, in Table 2 the 2.7B model scores 5.63% on Java, 5.56% on Python, and 5.16% on JavaScript, while in Table 3 the 2.7B model scores 6.10% on Python but 3.05%, 2.44%, and 1.22% on Rust, JavaScript, and Java. Given roughly 156-164 problems and a single pass@1 estimate per cell, these differences are plausibly within sampling noise. Confidence intervals, multiple seeds, or a statistical test are needed to support any ranking claim and to support the 'inconsistency' findings that motivate the paper.
minor comments (5)
- [Section 4] The sentence comparing Rust performance across benchmarks appears to swap the benchmark names: as written, it says Rust's 3.05% score is 'higher than in the MultiPL-E ... compared to HumanEvalSynthesize,' but Table 3 shows 3.05% in HumanEvalSynthesize and Table 2 shows 2.30% in MultiPL-E.
- [Section 2.1] There is a typographical error in 'HumanEval-XL [19]., which translates' with an extra period after the bracket.
- [Section 2.3] The text refers to reference [10] as 'OctoCoder,' but the cited paper is 'OctoPack'; this may confuse readers.
- [Section 3.2] The capitalization of the benchmark name is inconsistent: 'HumanEvalSyntheSize' in several places versus 'HumanEvalSynthesize' in the benchmark's own name; please standardize.
- [Section 4] The claimed consistency of the Python pass@1 scores with Xu et al.'s reported values (2.13%, 2.96%, 5.59%) should be quantified; the observed values differ by up to 0.6 percentage points, so the paper should state whether this is within expected sampling error rather than asserting consistency as validation.
Circularity Check
No circularity: the paper runs an external model on published benchmarks and compares to externally reported perplexity, with no fitted parameters or self-referential derivation.
full rationale
This paper is an empirical evaluation study, not a derivation. It takes a pre-trained model (PolyCoder), runs it against two published translated benchmarks (MultiPL-E and HumanEvalSynthesize), and reports pass@1 results. There are no equations that define one result in terms of another, no fitted parameters renamed as predictions, and no load-bearing reliance on the authors' own prior results. The only comparison to training-phase metrics is the reference to perplexity scores reported in the original PolyCoder paper (Xu et al., [23]), which is external evidence used as a point of contrast, not as an input to the computation. The paper's abstract claim that benchmark outcomes 'align well' with perplexity is empirically contradicted by the paper's own observation that Java's MultiPL-E performance 'does not align with the perplexity scores,' but an unsupported or internally inconsistent claim is a correctness threat, not circular reasoning. The study is self-contained as an evaluation: its conclusions could be falsified by re-running the same benchmarks, and none of the reported pass@1 values are forced by construction or by self-citation. Accordingly, no circular step is present and the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Pass@1 at temperature 0.2 is a reliable enough measure of code generation performance for comparing languages and benchmarks.
- domain assumption The translated problems in MultiPL-E and HumanEvalSynthesize are functionally equivalent to the original HumanEval problems and to each other in difficulty and semantics.
- ad hoc to paper The bigcode-evaluation-harness implementation of HumanEvalSynthesize adequately represents the OctoPack benchmark.
- domain assumption The Python pass@1 results in Xu et al. [23] are a valid baseline for the model's training-phase performance.
Cite this review
Pith. "Pith review of A Preliminary Study of Multilingual Code Language Models for Code Generation Task Using Translated Benchmarks." pith.science (2026). https://pith.science/paper/4F2O3ORT
@misc{pith2026241115470,
author = {Pith},
title = {Pith review of: A Preliminary Study of Multilingual Code Language Models for Code Generation Task Using Translated Benchmarks},
year = {2026},
howpublished = {\url{https://pith.science/paper/4F2O3ORT}},
note = {Machine review of arXiv:2411.15470}
}
read the original abstract
Evaluating the performance of Code Language Models (CLMs) for software engineering tasks, especially in multilingual and low-resource programming language settings, poses significant challenges. These challenges are primarily due to the lack of high-quality benchmarks across various programming languages and the imbalanced nature of the CLMs training corpus. Although recent advances in one of the common downstream tasks, code generation, have shown promise by introducing translated benchmarks using different methodologies, there is a current lack of empirical evidence assessing these benchmarks. To address this gap, we conducted a preliminary study to evaluate the performance of Poly-Coder, a pioneering open-source, multilingual CLM built for code generation. We utilized two existing state-of-the-art translations of the popular code generation benchmark, HumanEval, facilitated by the OctoPack and MultiPL-E studies. Our results suggest that the outcomes observed in these translated benchmarks align well with evaluation metrics used during the training phase, such as perplexity, thereby validating their effectiveness in estimating the performance of CLMs. However, we identified several inconsistencies in the CLMs' performance across the translated benchmarks and encountered challenges in replicating the results. These initial insights highlight the need for more comprehensive empirical studies to fully understand translated benchmarks' methodological approaches, limitations, and reproducibility. Such studies are essential to ensure their reliability before they are widely adopted.
Figures
Reference graph
Works this paper leans on
-
[2]
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, 4https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf 98 and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732
arXiv 2021
-
[3]
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. 2022. A framework for the evaluation of code generation models. https://github.com/bigcode-project/bigcode-evaluation-harness
2022
-
[4]
Antonio Mastropaolo et al. 2023. On the Robustness of Code Generation Tech- niques: An Empirical Study on GitHub Copilot. arXiv:2302.00438
arXiv 2023
-
[5]
Ben Athiwaratkun et al. 2023. Multi-lingual Evaluation of Code Generation Models. arXiv:2210.14868
arXiv 2023
-
[6]
Baptiste Rozière et al. 2024. Code Llama: Open Foundation Models for Code. arXiv:2308.12950
arXiv 2024
-
[7]
Dan Hendrycks et al. 2021. Measuring Coding Challenge Competence With APPS. arXiv:2105.09938
arXiv 2021
-
[8]
Federico Cassano et al. 2022. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. arXiv:2208.08227 [cs.LG]
arXiv 2022
-
[9]
Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG]
arXiv 2021
Show all 24 references
-
[10]
Niklas Muennighoff et al. 2024. OctoPack: Instruction Tuning Code Large Lan- guage Models. arXiv:2308.07124
2024 arXiv
-
[11]
Qinkai Zheng et al. 2023. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X. arXiv:2303.17568
2023 arXiv
-
[12]
Raymond Li et al. 2023. StarCoder: may the source be with you! arXiv:2305.06161
2023 arXiv
-
[13]
Rongao Li et al. 2023. TACO: Topics in Algorithmic COde generation dataset. arXiv preprint arXiv:2312.14852 (2023)
2023 arXiv
-
[14]
Shuai Lu et al. 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. CoRR abs/2102.04664 (2021)
2021 arXiv
-
[15]
Xueying Du et al. 2023. ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation. arXiv:2308.01861
2023 arXiv
-
[16]
Xinyi Hou et al. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620
2024 arXiv
-
[17]
Mohamad Khajezade, Jie JW Wu, Fatemeh Hendijani Fard, Gema Rodriguez-Perez, and Mohamed Sami Shehata. 2024. Investigating the Efficacy of Large Language Models for Code Clone Detection. In Proceedings of the 32nd IEEE/ACM Interna- tional Conference on Program Comprehension (<c...
2024 doi
-
[18]
Rangeet et al. Pan. 2024. Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). ACM. https: //doi.org/10.1145/3597503.3639226
2024
-
[19]
Qiwei Peng, Yekun Chai, and Xuhong Li. 2024. HumanEval-XL: A Multilin- gual Code Generation Benchmark for Cross-lingual Natural Language Gener- alization. In Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Evaluatio...
2024
-
[20]
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297
2020 arXiv
-
[21]
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Under- standing and Generation. arXiv:2109.00859
2021 arXiv
-
[22]
Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2024. Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models. arXiv:2308.10462
2024 arXiv
-
[23]
Xu, Uri Alon, Graham Neubig, and Vincent J
Frank F. Xu, Uri Alon, Graham Neubig, and Vincent J. Hellendoorn. 2022. A Sys- tematic Evaluation of Large Language Models of Code. arXiv:2202.13169 [cs.PL]
2022 arXiv
-
[24]
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig
-
[2018]
InInternational Conference on Mining Software Repositories (MSR)
Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow. InInternational Conference on Mining Software Repositories (MSR). ACM, 476–486. https://doi.org/10.1145/3196398.3196408 99
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.